Architecture · ops.tradeit.gg

Infrastructure

flowchart TD
    Users[Users / Clients]

    subgraph Cloudflare
        CF[Cloudflare DNS + CDN + WAF]
       
    end

    subgraph AWS["AWS Account"]

        subgraph PROD["Production Environment"]
            EC2P[EC2 Instance]
            DockerP[Docker Containers]
            Redis[Redis]
        end

        subgraph STAGE["Staging Environment"]
            RedisS[Redis]
            EC2S[EC2 Instance]
            DockerS[Docker Containers]
        end

        RDS[(RDS Database)]
        DD[Datadog Agent]
    end

    Users --> CF
    CF --> EC2P
    CF --> EC2S

     

    EC2P --> DockerP
    EC2S --> DockerS

    DockerP <--> Redis
    DockerP --> RDS
    DockerS <--> RedisS
    DockerS --> RDS

    DockerP --> DD
    DockerS --> DD

flowchart LR


    subgraph AWS["AWS eu-west-1"]
      subgraph VPC["Default VPC 172.31.0.0/16"]
        IGW[Internet Gateway]
        RT[Main Route Table]

        subgraph AZA["eu-west-1a"]
          SA[Subnet subnet-5af85e3d]
        end

        subgraph AZB["eu-west-1b"]
          SB[Subnet subnet-f98f23b0]
        end

        subgraph AZC["eu-west-1c"]
          SC[Subnet subnet-5f977504]
        end
      end
    end

    IGW --> RT
    RT --> SA
    RT --> SB
    RT --> SC

flowchart LR
  EC2A[EC2 - AZ A]
  EC2B[EC2 - AZ B]
  RDSC[RDS - AZ C]
  CacheC[ElastiCache - AZ C]

  EC2A -->|Cross-AZ $| RDSC
  EC2B -->|Cross-AZ $| RDSC
  EC2A -->|Cross-AZ $| CacheC
  EC2B -->|Cross-AZ $| CacheC

1. Document Info

• Document name: Infrastructure & Operations
• Scope: Production + Staging infrastructure (AWS)
• Audience: Engineering, Operations, Management
• Last updated: _(auto)_
• Owner: Infrastructure / DevOps
• Sources of truth:
• AWS CLI inventory dumps (JSON)
• Datadog (Cloud Network Map, Host Map, Monitors, Logs)

Purpose

This document describes the current and target infrastructure architecture of Zengaming (TradeIT). It is intended for:

• Operational understanding
• Cost optimization planning
• Incident response
• Security reviews and audits

2. Executive Summary

Zengaming (TradeIT) runs primarily in AWS eu-west-1 (Ireland) on the Default VPC model (public subnets). Workloads are deployed as Docker containers on EC2 using SSH + docker run (no ECS).

Key observations (validated via AWS + Datadog):

• Cross–Availability Zone (AZ) traffic is a major cost driver when heavy dependencies are in different AZs
• Per-customer Elastic IP allocation is not scalable (cost/quota/ops risk)
• Hard-pinned OpenSearch node IPs can force cross-AZ traffic and prevent easy scaling/replacement
• Security is enforced primarily via AWS Security Groups
• Compute distribution across AZs is uneven (Datadog Host Map: 1a: 5, 1b: 14, 1c: 18)
• RDS pressure appears driven by connection anomalies / storage utilization, not CPU

A target architecture is defined to reduce cost, improve scalability, and minimize public exposure.

3. Accounts & Organization

• Single AWS account
• Environment separation is logical (not account-based)
• Shared VPC and shared networking components

Recommendation: In the future, split prod / non-prod into separate accounts for blast-radius and governance.

4. Regions & Availability

Notes:

• Load balancers are Multi-AZ (when used)
• Datadog Host Map indicates uneven host distribution (1a: 5, 1b: 14, 1c: 18)

5. Network Topology (VPC)

VPC Configuration

Subnets

Internet & NAT

Implications:

• All instances are internet-routable at the network layer
• Security relies heavily on Security Groups
• NAT costs are avoided at the expense of exposure risk
• Datadog Cloud Network Map shows significant east–west traffic (service-to-service + data stores) and outbound to third-party APIs

6. Compute Layer (Docker on EC2 – Audited)

Overview

The platform runs application workloads directly on EC2 instances using Docker, without a container orchestrator (ECS/EKS).

• Orchestration: None (manual host-based scheduling)
• Deployment method: SSH-based (deploy.sh)
• Runtime: Docker + PM2
• Service isolation: Per-instance / per-container
• Scaling model: Manual or ASG (limited, workload-specific)

This design places EC2 as a primary operational and cost factor, requiring explicit documentation of placement, exposure, and ownership.

EC2 Inventory Summary (eu-west-1)

Key observation:

There is no meaningful workload distribution strategy across AZs.

Most compute (especially bots and inventory workloads) resides in eu-west-1c, creating:

- AZ blast-radius risk

- Cross-AZ data transfer costs when dependencies live elsewhere

Availability Zone Distribution

eu-west-1a

• Login (prod)
• Socket (prod)
• Login – staging
• Socket – staging
• Old staging site

eu-west-1c (dominant)

• Inventory service (c5.4xlarge)
• TradeIT backend (stg)
• OpenSearch-related workloads
• Legacy production site
• 545 bot instances (t3.micro)

Risk:

Inventory, bots, and backend traffic are heavily concentrated in 1c, while some dependencies (historically OpenSearch / RDS endpoints) span other AZs → cross-AZ traffic amplification.

Instance Role Classification

Networking & Exposure Model

• All instances reside in public subnets
• Most instances have public IPv4 addresses
• Inbound protection relies entirely on Security Groups
• No private subnets
• No NAT Gateways
• No internal-only compute tier

Implications

• Every misconfigured SG = internet exposure
• EIP usage grows linearly with instance count
• IPv4 cost and quota pressure
• Harder to enforce zero-trust boundaries

Docker Runtime Model

• Containers launched manually via:
• docker pull
• docker rm
• docker run
• No scheduler awareness (CPU / memory contention possible)
• No placement constraints
• No bin-packing efficiency
• No automatic rescheduling on failure

Container networking patterns

• Standard bridge mode:
• -p <hostPort>:3000
• Some services use:
• --network host (notably inventory)

Risk:

--network host bypasses container isolation and couples service behavior directly to the host’s network stack.

Process Management

• PM2 runs inside containers
• Container restarts handled via:
• --restart unless-stopped
• No health-based orchestration
• No rolling deployments
• No automated rollback

IAM & Identity (Observed)

• Most instances have NO IAM instance profile attached
• Access often relies on:
• SSH
• Static credentials
• Network-level trust

Security concern:

Lack of IAM roles prevents least-privilege access to AWS services and complicates auditability.

Operational Risks (Current State)

• Blast radius: Single-AZ concentration
• Cost: Cross-AZ traffic + public IPv4
• Security: Public compute + SG sprawl
• Scalability: Manual, host-bound
• Recovery: Instance failure = service outage unless manually handled
• Governance: No strong ownership tagging or lifecycle policy

Immediate Improvement Opportunities (Non-Disruptive)

1. Tag all EC2 instances

• Service
• Environment
• Owner
• Criticality

1. AZ-aware placement

• Co-locate backend + OpenSearch + Redis
• Reduce cross-AZ chatter

1. Reduce public exposure

• Introduce internal-only instances
• Front with NLB / ALB

1. IAM roles

• Attach instance profiles
• Remove static credentials

1. Prepare migration path

• ECS (EC2 launch type) as a first step
• Preserve instance-level control while gaining scheduling and health checks

Strategic Direction (Target State)

This section intentionally documents reality, not aspiration.

Migration paths are defined elsewhere in the document.

7. Load Balancing & Edge

7.1 Actual Ingress Architecture (Verified)

The platform does not use AWS ALB or NLB.

All ingress traffic is terminated on NGINX running directly on EC2 instances, which act as:

• TLS termination point
• L7 reverse proxy
• Rate limiter
• Path-based router
• Backend load balancer

Cloudflare functions strictly as an external CDN + WAF, not as an internal AWS load balancer.

7.2 End-to-End Request Path (Unambiguous)

Production & Staging (Current)

Client ↓ Cloudflare (DNS, CDN, WAF) ↓ HTTPS (443) Elastic IP (EC2 instance) ↓ NGINX (EC2) ↓ Upstream services (Docker containers / other EC2 hosts)

7.3 Load Balancing Responsibility

NGINX performs:

• least_conn upstream balancing
• Path-based routing (/api, /socket.io, /blog, /news, etc.)
• Header normalization
• Rate limiting (limit_req)
• Health checks
• Failover logic

7.4 TLS Termination

Source of truth: TLS is terminated inside AWS on NGINX, using local certificate files.

7.5 Exposed Ports

Public exposure is controlled by Security Groups, not subnet isolation.

7.6 Elastic IP Usage Model

Important implications:

• Every EIP = internet-exposed instance
• Scaling requires manual EIP management
• Failover requires EIP reassociation

7.7 NGINX as the Real Load Balancer (Evidence)

From nginx.conf:

• upstream backend { least_conn; ... }
• Multiple backend EC2 IPs defined
• Rate limiting via limit_req_zone
• TLS certificates configured locally
• Path-based routing across multiple services
• Health checks handled via /healthcheck

This confirms:

NGINX is the actual L7 load balancer

7.8 Diagram Correction Notes (Important)

❌ Incorrect (previous diagrams):

• ALB shown inside Cloudflare
• ALB shown as ingress
• Cloudflare acting as AWS L7 LB

✅ Correct representation: [ Internet ]

[ Cloudflare ]

[ Elastic IP ]

[ EC2 + NGINX ]

[ Backend EC2 / Containers ]

ALB must not appear in diagrams unless it actually exists.

7.9 Risks of Current Design

7.10 Target Direction (Not Yet Implemented)

This would:

• Reduce EIP count drastically
• Improve failover
• Enable autoscaling
• Remove public EC2 exposure

7.11 Summary (One Sentence)

Cloudflare is the edge; NGINX on EC2 is the real load balancer; AWS ALB is not used.

8. Cloudflare (Edge, DNS & Security Layer)

Purpose & Role

Cloudflare is the authoritative edge and DNS layer for all public-facing traffic for the tradeit.gg platform.

It provides:

• Authoritative DNS
• Global CDN and HTTP proxy
• Edge TLS termination
• L7 security controls (WAF, rate limiting, bot protection)
• Origin IP masking

Cloudflare is external to AWS and always sits in front of AWS infrastructure.

Cloudflare Account & Zones

Edge Traffic Model (Authoritative)

Client
  ↓ HTTPS (443)
Cloudflare Edge
  - TLS termination
  - WAF / rate limits
  - Bot protection
  - CDN
  ↓ HTTPS
AWS Origin (ALB or EC2)

Cloudflare is the only internet-facing entrypoint. AWS origins are never exposed directly unless explicitly documented.

DNS & Routing Strategy

Cloudflare DNS is the single source of truth for traffic routing.

Routing patterns in use:

• CNAME → nginx ALB (preferred)
• A / CNAME → EC2 (legacy, proxied)
• CNAME → Cloudflare Tunnel (restricted access)
• DNS-only records for email, verification, and third-party SaaS

Production Hostnames

Hostname: tradeit.gg Target: ntginx alb Proxy: Proxied Notes: Primary production entry

Hostname: www.tradeit.gg Target: tradeit.gg Proxy: DNS only Notes: Alias

Hostname: api.tradeit.gg Target: EC2 hostname Proxy: Proxied Notes: Legacy direct-to-instance

Hostname: inventory.tradeit.gg Target: EC2 IPv4 Proxy: Proxied Notes: Service-specific endpoint

Staging Hostnames

Hostname: stg-lb.tradeit.gg Target: nginx alb Proxy: Proxied Notes: Main staging

Hostname: stg-banana.tradeit.gg Target: nginx alb Proxy: Proxied Notes: Legacy / parallel

Hostname: stg-tunneled.tradeit.gg Target: Cloudflare Tunnel Proxy: Proxied Notes: No public origin

Proxy Status Policy

Public web / API: Always proxied ALB origins: Always proxied Email (MX, DKIM, SPF): DNS only Verification records: DNS only SaaS integrations: Case-by-case

Any A or CNAME record pointing to AWS infrastructure must be proxied unless explicitly documented.

TLS & Certificates

TLS terminates at Cloudflare Edge.

Client ↔ Cloudflare:

• HTTPS
• Cloudflare-managed certificates

Cloudflare ↔ Origin:

• HTTPS to ALB (ACM certificates)
• HTTPS to EC2/NGINX (legacy)

TLS mode:

• Full / Full (Strict)

No plaintext HTTP traffic is intentionally exposed.

Security Controls

Cloudflare enforces:

• Web Application Firewall (WAF)
• Global and per-host rate limiting
• Bot and abuse mitigation
• Origin IP masking

High-value domains are fully proxied by default.

Email & Non-HTTP DNS Records

Cloudflare DNS hosts records for:

• Google Workspace (MX)
• SendGrid, Mandrill, Intercom, Sendinblue (DKIM / CNAME / TXT)
• DMARC, SPF, MTA-STS, TLS-RPT

These records are DNS-only by design.

8. Data Stores

Database – Aurora MySQL (RDS)

Notes:

• A “public endpoint” exists, but inbound should be limited to approved IPs / SGs
• Datadog indicates connection anomalies/storage alerts can be bigger drivers than CPU

Cache – Redis (ElastiCache)

9. Search Layer (OpenSearch on EC2)

Current implementation

• OpenSearch is running on EC2 instances (self-managed), not AWS OpenSearch Service.
• Backend containers reference OpenSearch nodes via hard-coded IP mappings (--add-host ...).

Verified node placement (AWS CLI)

Key finding (cost + latency):

• Backend in 1c talking to the coordinating node in 1b can generate cross-AZ traffic during search requests / coordination flows.
• Hard-pinning IPs prevents easy failover and makes scaling/replacement risky.

10. Messaging & Async Processing

• No dedicated managed messaging layer (SQS/Kafka/EventBridge)
• Some domains disable queue at runtime (DISABLE_QUEUE=1)
• Section reserved for future adoption

11. Identity, Access & Secrets

• Access controlled via AWS IAM
• EC2 instances assume IAM roles (where configured)
• Secrets/parameters stored in:
• AWS Secrets Manager (to validate via secrets_list.json)
• SSM Parameter Store (to validate via ssm_parameters_list.json)
• Service-to-service access controlled via Security Groups

12. Security Controls

Security Groups

Inbound Rules Summary

Hard rules:

• 0.0.0.0/0 must never be allowed unintentionally
• SSH restricted to trusted IPs only

13. Observability (Monitoring, Logging, Tracing)

• Metrics and logs exported to Datadog
• Host Map + Network Map used to validate AZ distribution and top talkers
• Alerting includes host metrics + RDS anomaly monitors + Redis monitors

Cost note:

• If CloudWatch Logs are also ingesting the same logs as Datadog, this can create double-cost.
• Prefer a single “source of truth” for log storage/retention where possible.

14. Backup, Disaster Recovery & Business Continuity

• Aurora automated backups enabled (verify retention window)
• Redis persistence configuration to be confirmed
• DR currently depends on regional availability and manual recovery steps

RTO / RPO to be defined.

15. Storage Layer – EC2 EBS Volumes & Snapshots

Overview

The platform relies heavily on EC2-attached EBS volumes, primarily used as root disks for Docker-based services running directly on EC2.

There is no centralized backup or lifecycle policy currently enforced.

EBS Volume Characteristics

Observations:

• Volumes are small and uniform
• Mostly root disks
• Indicates many small, long-lived EC2 instances
• No data volumes separated from compute

Snapshot & Backup Status

Important:

• Existing snapshots are incidental, not operational backups
• No guaranteed recovery point (RPO)
• No defined recovery time objective (RTO)

Risks

• Data loss risk: Root volume corruption = total service loss
• Compliance risk: No retention policy
• Operational risk: Manual recovery only
• Cost risk: Orphaned snapshots accumulate silently

Recommendations (Phase 1 – Low Risk)

1. Enable EBS encryption by default
2. Introduce DLM snapshot policies:

• Daily snapshots
• 7–14 day retention

1. Identify:

• Unused volumes
• Volumes attached to stopped instances

1. Tag volumes with:

• Service
• Environment
• Owner

Recommendations (Phase 2 – Structural)

• Separate data volumes from root disks
• Move stateful services to:
• RDS / ElastiCache / OpenSearch managed
• Reduce EC2 footprint
• Prepare migration path to ECS / EKS

This account contains a large number of Amazon EBS volumes, mostly small (gp2, ~8 GiB) root volumes attached to EC2 instances running Docker-based services.

Without active management, EBS volumes and snapshots can become a significant hidden cost.

Current Observations

• Volume type: gp2
• Typical size: 8 GiB
• Volume count: ~600 volumes
• Snapshot coverage: 0 / 598 volumes
• No lifecycle or retention policy enabled
• Some volumes are unattached or attached to stopped instances

Cost Optimization Opportunities

1. Convert gp2 → gp3 (Immediate Savings)

gp3 is the recommended default EBS volume type.

Benefits

• ~33% cost reduction
• No downtime
• No data loss
• Can be done on running instances

17. Cost Management & Optimization

Current Cost Drivers

Cross-AZ Traffic

• Exists when compute-heavy services and data stores / OpenSearch nodes are in different AZs.
• Verified example: Backend (1c) ↔ OpenSearch coordinating node (1b).

Elastic IP Sprawl

• High EIP count is a cost/quota/ops risk and not scalable long-term.

RDS Cost Amplification

• Connection storms + latency + oversized instances can amplify costs.

Target Architecture (direction)

Expected outcomes:

• Less cross-AZ traffic
• Reduced EIP count
• Lower VPC + DB costs
• Improved stability and scalability

18. CI/CD & Release Process

This environment uses CircleCI for build and (in some cases) deployment. Deployments are still Docker-on-EC2 (not ECS), executed via SSH to target hosts (often via a bastion).

CI/CD Platform

• CI provider: CircleCI
• Pipelines location: .circleci/config.yml
• Docker registry: Docker Hub ($CIRCLE_PROJECT_USERNAME/<image>)
• Notification: Slack (CircleCI Slack orb)

Build Pipeline (Docker build & push)

Job: build Runtime: cimg/base:2022.03 + setup_remote_docker Output: Docker image pushed to Docker Hub

Build behavior:

• Image tag format:
• Branch builds: branch-name transformed with / → --

Example: feature/new-ui → feature--new-ui

• Tag builds: CIRCLE_TAG (used as-is)
• Build uses many --build-arg values to inject runtime configuration at build-time.

Risk note (important):

Using --build-arg for secrets (tokens/keys) may bake secrets into image layers if passed via ARG → ENV inside Dockerfile.

Prefer runtime secrets via SSM/Secrets Manager or Docker runtime env vars.

Build steps summary: 1) Docker Hub login (DOCKER_LOGIN/DOCKER_PASSWORD) 2) docker build with many build args 3) docker push

Slack:

• Notifies tradeit-builds on success/failure with metadata (branch, author, commit, job link)

Deploy Pipeline (CircleCI → Bastion → Target EC2)

Job: deploy Runtime: CircleCI machine executor (Ubuntu)

Deployment method:

• Uses a bastion jump host with a tunnel:
• dmz/open_tunnel orb opens a local port to the target
• SSH connects to 0.0.0.0:$BASTION_PORT as a proxy to the target EC2 instance

Deploy steps (current): 1) Add SSH keys (CircleCI-managed fingerprint) 2) Pre-approve bastion host key via ssh-keyscan 3) Open tunnel to target via bastion 4) On target host:

• docker image prune -fa
• docker pull <image>:<tag>
• docker tag ... new-tradeit
• docker rm -f new-tradeit
• docker run --name new-tradeit -p 3000:3000 -dit --restart unless-stopped new-tradeit
• Print container logs

Slack:

• Sends fail notifications to tradeit-builds
• Sends success deploy notifications (template: success_tagged_deploy_1)

Branching / Release Policy

Workflow: build-docker-image-and-push

• Staging build triggers
• branches: feature/*, hotfix/*, bugfix/*, release/*
• tags: staging-*, dev-*
• Production build triggers
• branches: master, likeprod
• tags: strict semver pattern vX.Y.Z...

Note: Current deploy job appears primarily used for staging-style deployment via bastion.

Production deployments may still be performed via manual scripts or separate pipelines depending on service.

Known Gaps / Improvement Ideas

• No health-based rollout (stop/start single container)
• No automated rollback
• No immutable infra (still SSH imperative)
• Secrets passed at build-time (risk of secret leakage)

Planned improvements:

• Move secrets to runtime (SSM/Secrets Manager)
• Add a post-deploy health check and rollback logic
• Consider migrating long-term to ECS (EC2 launch type) for safer rollouts

# Docker on EC2 — Operations Documentation

---

## 1. Service Info

- **Service name:** tradeit / tradeit-backend / tradeit-inventory-server
- **Environment:** prod / stg / likeprod
- **Repository:** https://github.com/zengamingx/<repo>
- **Docker image:** zengamingx/<image>
- **Runtime:** Docker + PM2
- **Deployment type:** SSH-based deployment
- **Owner:** DevOps / Platform
- **Slack channel:** #tradeit-dev

---

## 2. Hosts & Placement

| Env | Instance Name | Instance ID | AZ | Type | Public | Notes |
|---|---|---|---|---|---|---|
| prod | Login | i-0388bbadd9fb77b30 | eu-west-1a | c7a.large | Yes | User entrypoint |
| prod | Socket | i-07c992a075c2a5aeb | eu-west-1a | c7a.large | Yes | WebSockets |
| stg | TI Backend Stg | i-0089b6d6d03867c2d | eu-west-1c | t3.large | No | Backend |

---

## 3. Networking

- **Inbound path:**  
  Cloudflare → EC2 (direct EIP / Public IP)

- **Host ports:**  
  3000, 4001–4005 (domain-based routing)

- **Container ports:**  
  3000

- **Docker network mode:**  
  bridge (inventory uses host)

- **Security groups:**  
  - `sg-00219c4fb3212cea4` (base)
  - `sg-051880495a8dddc9a` (Cloudflare HTTPS)

⚠️ **Important**
- Any instance with a public IP is internet-reachable
- 0.0.0.0/0 must never be allowed unintentionally
- SSH restricted to trusted IPs only

---

## 4. Dependencies

| Dependency | Endpoint | Port | AZ Sensitive | Notes |
|---|---|---:|---|---|
| Backend API | 172.31.45.74 | 3000 | Yes | Hardcoded |
| Aurora MySQL | cluster endpoint | 3306 | Yes | Public endpoint, SG-restricted |
| Redis | ElastiCache | 6379 | Yes | Same AZ preferred |
| OpenSearch | 172.31.25.126 / 172.31.36.183 | 9200 | Yes | Hard-pinned IPs |
| External APIs | Stripe, Intercom | 443 | No | Internet |

---

## 5. Build

- **Dockerfile:**  
  `/Dockerfile`


- **Secrets used at build time:**  
  - NODE_AUTH_TOKEN  
  - DOTENV_KEY  
  - FONT_AWESOME_TOKEN  

⚠️ **Risk**
- Secrets passed via `ARG → ENV` are baked into image layers
- Migration to runtime secrets recommended

---

## 6. Deploy Procedure (Current)

### Summary
- SSH-based deploy via `deploy.sh`
- Manual approval for prod
- No health checks
- No rollback automation

Service-Level Infrastructure Documentation (TradeIT)

1. Service Role

This NGINX instance acts as a public-facing edge proxy responsible for:

• TLS termination
• HTTP → HTTPS enforcement
• Rate limiting & abuse protection
• Reverse proxy routing
• WebSocket handling
• Maintenance and block pages
• Internal service aggregation

This service is internet-exposed and participates in the Direct-to-Instance ingress architecture (no AWS ALB in front).

2. Ingress Architecture Position

Client ↓ Cloudflare (CDN, WAF, optional TLS) ↓ NGINX (this service, public IP / EIP) ↓ Internal services (Docker / EC2 private IPs)

Notes

• Cloudflare is the primary edge
• NGINX provides application-aware routing
• Security relies on Cloudflare + Security Groups + NGINX rules

3. Security Model

Network

• Instance runs in a public subnet
• Access restricted by:
• Cloudflare IP allowlisting (SG level)
• NGINX deny rules
• Rate limits

TLS

• TLS termination happens at NGINX
• Certificates are locally managed
• HSTS enabled

Hardening

• server_tokens off
• XSS, Frame, Content-Type protection headers enabled

4. Rate Limiting Strategy

Global (Per Host)

• 100 requests/sec
• Burst: 30
• Enforced on HTTPS listener

Per-IP

• 85 requests/sec
• Connection limiting enabled

Whitelisted IPs

• localhost
• RFC1918 ranges

5. Backend Routing Model

• Static upstreams
• No service discovery
• AZ placement matters (cross-AZ traffic risk)
• Scaling is manual

6. Ports & Exposure

7. Routed Services Overview

8. Error Handling

• 403 / 5xx → maintenance page
• 429 → rate-limited page
• 405 rewritten to 200 for static compatibility

9. Logging & Observability

• Extended access log format
• Includes:
• Upstream timing
• Rate-limit status
• Cloudflare country header
• Logs shipped to Datadog

10. Operational Risks

• Hard-coded private IPs
• No upstream health checks
• Manual TLS lifecycle
• Single-instance scaling unit
• Public IP exposure

11. Improvement Direction

• Replace with ALB / NLB where possible
• Reduce EIP usage
• Move rate limiting to Cloudflare
• Introduce service discovery
• Centralize TLS

12. NGINX Configuration (Source of Truth)

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
worker_rlimit_nofile 8192;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 8192;
    multi_accept on;
    use epoll;
}

http {
    limit_req_zone $host zone=global_rate_limit:10m rate=100r/s;

    log_format rate_limit_log '$remote_addr - $host [$time_local] "$request" '
                              '$status $body_bytes_sent "$http_referer" '
                              '"$http_user_agent" "$request_time" "$upstream_response_time" '
                              '"$pipe" "$limit_req_status"';

    limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;
    limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=85r/s;

    resolver 8.8.8.8 ipv6=off;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65s;
    client_max_body_size 20M;
    server_tokens off;

    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    ssl_session_cache shared:SSL:20m;
    ssl_session_timeout 1d;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for" '
                        '"$host" reqtime=$request_time '
                        'upstream="$upstream_addr" '
                        'ratelimit=$limit_req_status '
                        'country=$http_cf_ipcountry';

    access_log /var/log/nginx/access.log main_ext;

    map $remote_addr $whitelisted {
        default 0;
        127.0.0.1 1;
        ::1 1;
        10.0.0.0/8 1;
        192.168.0.0/16 1;
    }

    upstream backend {
        least_conn;
        server ip-172-31-42-95.eu-west-1.compute.internal:3000;
        server ip-172-31-44-148.eu-west-1.compute.internal:3000;
    }

    server {
        listen 443 ssl http2 default_server;
        server_name tradeit.gg www.tradeit.gg;

        ssl_certificate /etc/nginx/tradeit.crt;
        ssl_certificate_key /etc/nginx/tradeit.key;

        limit_req zone=global_rate_limit burst=30 nodelay;
        limit_req_status 429;

        location /api/v2/ {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location / {
            proxy_pass http://localhost:3000;
        }
    }
}

ops.tradeit.gg — Internal Engineering Docs