Guide · Operations · Jun 3, 2026

Tradebot Fleet Operations

Deploy, access, and operate the co-located bot fleet. Since OPS-235/229 every bot runs co-located (~3 per shared server), credential-free at runtime — env is fetched at deploy from KMS-encrypted SSM Parameter Store. No run.sh, no plaintext, no auto-scaling groups, no public SSH.

---

1. One-time workstation setup

1. AWS CLI v2 with your IAM credentials, region eu-west-1.
2. Session Manager plugin — SSH rides SSM, so no public SSH and no IP lists:

   brew install --cask session-manager-plugin

3. steamtest.pem → ~/.ssh/steamtest.pem, chmod 600. It's in authorized_keys fleet-wide — get it from Ehud (do not commit or share it).
4. Bot resolver:

   cp tradebot/deploy/remote/bot-resolve.sh ~/.ssh/bot-resolve.sh && chmod +x ~/.ssh/bot-resolve.sh

5. ~/.ssh/config blocks:

   Host bot*
     ProxyCommand sh -c "aws ssm start-session --target $(~/.ssh/bot-resolve.sh %h) --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region eu-west-1"
     User ec2-user
     IdentityFile ~/.ssh/steamtest.pem
     StrictHostKeyChecking accept-new
   
   Host i-* mi-*
     ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region eu-west-1"
     User ec2-user
     IdentityFile ~/.ssh/steamtest.pem

2. SSH into a bot's host

ssh bot42            # resolves bot 42 -> its co-lo host (bot-ids tag), tunnels over SSM
ssh i-0123abcd...    # any SSM-managed instance by id

On the host (each runs ~3 bot<ID> containers):

docker ps                                  # bot40 bot41 bot42 ...
docker logs --since 1h bot42 | tail -50
docker logs bot42 2>&1 | grep -i "logged into steam"

3. Deploy to production

Always deploy a digest, never :master — what runs is then deterministic and rollback is exact.

Whole fleet, one command — confirms (type the host count) and resolves the digest for you:

remote/deploy-all.sh                      # :master to every prod host (~3 bots each), phased
remote/deploy-all.sh <tag|@sha256:digest>  # explicit image

One bot: remote/deploy-bot.sh <id> (see Common operations). Raw form of the fleet deploy:

# 1. resolve the digest to ship
docker buildx imagetools inspect zengamingx/tradeit-bot:master

# 2. fleet deploy (phased, auto-halts on failures)
aws ssm send-command --document-name tradebot-deploy \
  --targets Key=tag:deploy-group,Values=tradebot \
  --parameters 'imageRef=zengamingx/tradeit-bot@sha256:<digest>,bots=all' \
  --max-concurrency 10% --max-errors 5%

# 3. check (per instance)
aws ssm get-command-invocation --command-id <id> --instance-id <i-...> \
  --query '{Status:Status,Out:StandardOutputContent}' --output json

• bots=all = every existing bot<ID> container on each host. Single host: --targets Key=instanceids,Values=i-...; single bot: bots=42.
• Rollback = re-run with the previous digest.
• deploy.sh on each host fetches env per bot from Parameter Store and health-checks logins; banned/retired bots are skipped via the STATUS registry.
• Watch in Datadog: service:tradeit-tradebot-server @context.botId:<N> — per-bot identity is the context.botId log attribute.

4. Deploy to staging

Staging bots (898/899/900) are co-located on one t3.small, tagged deploy-group=tradebot-staging so a production deploy never touches them, and run the staging docker tag — a distinct build from prod's master. One command, no instance ids:

remote/deploy-staging.sh            # deploy :staging (default) to the staging host
remote/deploy-staging.sh master     # or any floating tag (resolved to a digest here)
remote/deploy-staging.sh zengamingx/tradeit-bot@sha256:<digest>   # or an exact digest

It targets the staging tag, runs the same tradebot-deploy document as prod, prints per-host health + rc, and refuses to run if zero hosts carry the tag. Each staging bot's per-bot DOTENV_KEY automatically wins over the fleet-shared key at deploy. Default is :staging — staging historically runs that tag, not master.

5. Common operations

# deploy ONE bot (resolves its host; defaults to :master; confirms PROD) — preferred
remote/deploy-bot.sh 42                # or: remote/deploy-bot.sh 42 <tag|@sha256:digest>

# restart one bot in place (keeps current image + env)
ssh bot42 'docker restart bot42'      # Steam re-login is automatic (refresh token / TOTP)

# equivalent raw form (target the HOST, never the deploy-group tag)
aws ssm send-command --document-name tradebot-deploy \
  --targets Key=instanceids,Values=$(~/.ssh/bot-resolve.sh 42) \
  --parameters 'imageRef=<current digest>,bots=42'

# bot category registry (exception-only; absence = active)
aws ssm put-parameter --name /tradebot/<id>/STATUS --type String --value banned --overwrite
aws ssm delete-parameter --name /tradebot/<id>/STATUS    # back to active

Adding a new bot / moving a bot between hosts → see colocate-trio.sh + the adding-a-tradebot playbook; per-bot data dir is /home/root/.local/share-<ID> on the host (Steam session persists there).

6. Access notes

Generated Jun 3, 2026 · tradeit.gg engineering · OPS-235 / OPS-229