CDN Image Update Guide

Pipeline Overview

Download

steam-img-downloader
Python · Steam CDN

Crop & Resize

steam-img-cropper
Node.js · Sharp → WebP

Push to S3

tradeit-prepared-skins
S3 → CloudFront

Prerequisites

Ensure you have Python 3, Node.js 22, and AWS CLI configured with the correct credentials before starting. The virtual environment lives inside the repo root (bin/, lib/, include/).

Step-by-Step

1

Navigate to the downloader repo

cd ~/Projects/steam-img-downloader

2

Activate the virtual environment

The venv is bootstrapped in the repo root. Activate it so python3 resolves to the isolated environment with all dependencies.

python3 -m venv . # only needed on first run or after a clean checkout
source bin/activate

If dependencies are already installed and the venv exists, skip the python3 -m venv . line and just source the activation script.

3

Download CS2 skin images

Fetches skin metadata from the ByMykel CSGO API and downloads any missing images into all_cs2_skin_images/. Already-downloaded files are skipped (skip-if-exists). Failed downloads are logged to a timestamped failed_skins_YYYYMMDD_HHMMSS.log.

python3 index.py

Runtime: This can take 15–20+ minutes on a full run with ~6,000+ skins. To re-download a specific file, delete it from all_cs2_skin_images/ first.

4

Navigate to the cropper repo

cd ~/Projects/steam-img-cropper

5

Sync raw images into the cropper's input directory

The cropper reads from tradeit-skins/csgo/ — not directly from the downloader's output. This rsync copies new/changed files only (-a = archive mode, preserves timestamps and permissions).

rsync -a ~/Projects/steam-img-downloader/all_cs2_skin_images/ tradeit-skins/csgo

6

Run the image cropper

Processes all images in tradeit-skins/csgo/ through 35 dimension variants and outputs WebP files to tradeit-cropped-skins/csgo/. Existing outputs are skipped unless you force a rebuild.

npm run start

# Force rebuild all (e.g. after dimension config changes):
npm run crop-force

Processing is CPU-parallel via Sharp (concurrency = core count). A full run with 6,000+ skins across 35 dimensions is compute-heavy — expect 10–30 min depending on machine. A skip log is written to skip-log-{timestamp}.txt.

7

Sync processed images to S3

Uploads the cropped WebP images to the tradeit-prepared-skins S3 bucket under the csgo/ prefix. CloudFront serves directly from this bucket. Only changed/new files are transferred.

aws s3 sync tradeit-cropped-skins/csgo s3://tradeit-prepared-skins/csgo

CDN propagation: CloudFront caches aggressively. If existing images were replaced (not just added), you may need to create an invalidation: aws cloudfront create-invalidation --distribution-id <DIST_ID> --paths "/csgo/*"

Troubleshooting

Never run python index.py without activating the venv first — system Python may lack requests / schedule and silently fail.

Symptom	Likely cause & fix
High failure count in log	Steam CDN rate-limiting. Increase `DELAY` env var (default 0.2s) and re-run — skip-if-exists means only failed files are retried.
`ModuleNotFoundError: requests`	Venv not activated. Run `source bin/activate` from the downloader root, then `pip install -r requirements.txt` if the module is still missing.
rsync copies nothing / wrong path	Trailing slash on source matters: `all_cs2_skin_images/` (with slash) copies contents into destination. Without slash it creates a nested directory.
Cropper produces no output	Check `tradeit-skins/csgo/` is non-empty after rsync. If `INPUT_DIR` is overridden in `.env`, verify it points to the correct path.
S3 sync: `NoCredentialsError`	AWS credentials not configured. Run `aws configure` or export `AWS_PROFILE` / `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`.
Old images still showing on site	CloudFront cache. Create an invalidation for `/csgo/*` (or specific paths) via the AWS console or CLI as shown above.

Full Run — Quick Copy

# 1. Download new skins
cd ~/Projects/steam-img-downloader
source bin/activate
python3 index.py

# 2. Crop & resize
cd ~/Projects/steam-img-cropper
rsync -a ~/Projects/steam-img-downloader/all_cs2_skin_images/ tradeit-skins/csgo
npm run start

# 3. Push to CDN
aws s3 sync tradeit-cropped-skins/csgo s3://tradeit-prepared-skins/csgo