tradeit.gg

Service Deep-Dive · Backend · May 15, 2026

Pricing Pipeline

End-to-end: how item prices flow from market data through the pricing engine into MySQL

Read on Wiki: /engineering/backend/pricing-pipeline (id 56)  ·  Repo: zengamingx/pricing-manager
~25k
Items per run
12
Pipeline steps
1.75×
Trade multiplier
3600s
Redis lock TTL
5
Price types written

Contents


1. At-a-glance

pricing-manager is a standalone NestJS service that recalculates buy/sell prices for all ~25,000 CS2 skin items in a single synchronous pass. It is the only writer of the 5 canonical price fields (bot_trade_price, player_trade_price, store_price, instant_sell_price, stable_price) in steamtrade.item_prices_raw. Triggered externally via HTTP; Redis-locked to prevent concurrent runs.

2. Purpose & boundaries

Owns

  • All price calculation logic (ECDF, Multiplier tiers, spread ratio, deficit adjustment)
  • Final write of all 5 price types to item_prices_raw
  • Price-protection rails (doppler, spike, market-share guards)
  • Adjustment percentages written to price_adjuster
  • GET /snapshots/take — point-in-time price snapshots

Does NOT own

  • Steam market scraping (steam-pricing-scraper)
  • Buff163/Doppler scraping (tradeit-buff-scrapper)
  • CSGOSkins.gg scraping — only fetches from their API at run time
  • Stock aggregation (tradeit-stock-manager writes item_stocks)
  • Reading prices for trade evaluation (tradeit-backend, new-tradeit)
  • OpenSearch indexing — no OS interaction

3. Process / runtime model

4. Architecture (data flow)

4a. System-level data flow

graph LR
    subgraph "Upstream scrapers (separate services)"
        SPS["steam-pricing-scraper"]
        BUFF["tradeit-buff-scrapper"]
        CSGOS["csgoskins-pricing"]
    end

    subgraph "External API (called at run time)"
        CSAPI["csgoskins.gg\n/prices"]
    end

    subgraph "MySQL steamtrade (reads)"
        BPS[("buff_prices_scraper")]
        SNAP[("item_prices_snapshot")]
        TIS[("trade_item_stats")]
        SIS[("store_item_stats")]
        EOD[("item_eod")]
        PG[("pricing_groups")]
        CONF[("configurations")]
    end

    subgraph "MySQL tradeitConnection (reads)"
        ISTK[("item_stocks")]
    end

    subgraph "pricing-manager"
        PM["AppService\n.updatePrices()"]
    end

    subgraph "MySQL steamtrade (writes)"
        IPR[("item_prices_raw")]
        PA[("price_adjuster")]
    end

    subgraph "Consumers"
        BE["tradeit-backend\ntrade evaluation"]
        NT["new-tradeit\nmarketplace display"]
        ADM["tradeit-admin\npricing overrides"]
    end

    SPS --> BPS
    BUFF --> BPS
    CSAPI --> PM
    BPS --> PM
    SNAP --> PM
    TIS --> PM
    SIS --> PM
    EOD --> PM
    PG --> PM
    CONF --> PM
    ISTK --> PM
    PM --> IPR
    PM --> PA
    IPR --> BE
    IPR --> NT
    IPR --> ADM

4b. Pipeline steps (sequential within one run)

graph TD
    S1["1. READ buff_prices_scraper\n(CS2 appId=730 items only)"]
    S2["2. JOIN item_stocks\n(bot/container/reserved/listing stock)"]
    S3["3. FETCH csgoskins.gg /prices API\n(market prices by market_hash_name)"]
    S4["4. STABLE PRICE\nECDF + quantile (mathjs) + mean of last 30 snapshots"]
    S5["5. ASSIGN MARGINS\npricingGroups lookup by stable_price tier (cached 2 min)"]
    S6["6. MERGE TRADE STATS\ntrade_item_stats + store_item_stats (if config flag enabled)"]
    S7["7. BASE PRICES\nMultiplier enum tiers: Base=1.75, Low=1.725, High=1.775..."]
    S8["8. LOAD PREVIOUS PRICES\nitem_prices_raw (prev bot/player/stable)"]
    S9["9. LOAD EOD PRICES\nitem_eod avg/median for 7d and 30d max stock"]
    S10["10. READ RUNTIME CONFIG\nweeklyInOutRatioWeight + wantedMaxStockLowCapacityRatio"]
    S11["11. CALCULATE PER ITEM\ncurrentStock, wantedMaxStock, neededStock, deficit"]
    S12["12. ADJUST and WRITE\nspread ratio, protection rails, markup flags\n→ UPSERT item_prices_raw + price_adjuster"]

    S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 --> S9 --> S10 --> S11 --> S12

    classDef read fill:#0e2a1e,stroke:#10b981,stroke-width:2px,color:#a7f3d0
    classDef calc fill:#1e2438,stroke:#a855f7,stroke-width:2px,color:#e9d5ff
    classDef write fill:#2a1218,stroke:#f43f5e,stroke-width:2px,color:#fecdd3
    classDef config fill:#2a1f0e,stroke:#f59e0b,stroke-width:2px,color:#fde68a

    class S1,S2,S3,S8,S9 read
    class S4,S5,S6,S7,S11 calc
    class S12 write
    class S10 config

5. Inputs & outputs

DirectionSurfaceDescription
INGET /update-pricesHTTP trigger — starts the full pipeline. Returns {"message":"Prices updated successfully"} after completion. Redis-locked for 3600 s.
INGET /snapshots/takeManually triggers a price snapshot. Reads item_prices_raw.live_price → batch-writes to item_prices_snapshot. Also Redis-locked (3600 s).
INGET /healthNestJS Terminus: pings both DBs, checks disk <75%, heap <150 MB.
INGET /Health-check hello — returns HELLO_MESSAGE env var.
INMySQL steamtradeReads: buff_prices_scraper, item_prices_snapshot, trade_item_stats, store_item_stats, item_eod, pricing_groups, configurations, item_prices_raw (prev prices).
INMySQL tradeitConnectionReads: item_stocks (bot stock, container stock, reserved, locked, listing).
INcsgoskins.gg /pricesExternal HTTP API — market prices per market_hash_name across multiple markets (csfloat, buff, skinport etc.).
OUTMySQL steamtrade.item_prices_rawUPSERT of 5 price types (bot, player, store, instant-sell, stable) + stock fields + tags per item.
OUTMySQL steamtrade.price_adjusterWrites adjustment percentages (item_price_adjust_percent) and protection diff columns.
OUTMySQL steamtrade.item_prices_snapshotBatch-written by GET /snapshots/take — one row per item per snapshot.

6. Redis surface

Redis is used only for distributed locking and in-process cache. No pub/sub, no persistent key storage for prices. Redis client uses plain SET key 'locked' NX EX ttl.

KeyTTLWritten byRead byBehavior on lock held
updatePrices 3600 s app.service.ts:73 (@LockWithRedis decorator) Same decorator on entry Silent no-op — logs "Another instance is already running" and returns immediately
takeSnapshot 3600 s item-prices-snapshot.service.ts:23 Same Silent no-op
pricingGroups in-process 120 s pricing-groups.service.ts:33 (@CacheReturnValue) Same service, subsequent calls within TTL Cache miss → re-reads DB. Not a Redis key — stored in a JS Map.

Connection: redis://${TI_REDIS_HOST}:${TI_REDIS_PORT} — no auth in env schema (shared ElastiCache). Lock key naming is plain string — no prefix/namespace. Lock is released in finally block (redis.decorator.ts:34) but a crashed process leaves the key until TTL expiry.

7. MySQL surface

steamtrade DB (pricing DB — MYSQL_* env vars)

TableOpsKey columns / indexesNotes
buff_prices_scraper SELECT item_id, buff_buy, buff_sell, buff_sell_amount, buff_buy_amount, is_doppler Filtered to app_id=730 (CS2 only). Written by tradeit-buff-scrapper.
item_prices_snapshot SELECT  INSERT item_id, live_price, created_at — indexes: item_id_created_at_IDX Reads last 30 snapshots per item to compute stable price mean. Written by /snapshots/take.
item_prices_raw SELECT  UPSERT item_id (UNQ), bot_trade_price, player_trade_price, store_price, instant_sell_price, stable_price, live_price, wanted_max_stock, needed_stock, current_stock, final_deficit, popularity_score, ... The primary price table. Consumers (tradeit-backend, new-tradeit) read from here. Indexes: UNQ_item_id, IDX_updated_at.
price_adjuster UPSERT item_price_adjust_percent, protection diff columns (wanted_max_stock_spike, wanted_max_stock_in_out_ratio) Stores per-item adjustment breakdown. Written by price-adjuster.service.ts:adjustFinalPrices.
pricing_groups SELECT min_price, group, margin Price tier table — maps stable_price ranges to margin groups. Cached in-process for 2 min.
trade_item_stats SELECT item_id, monthly_in_amount, monthly_out_amount, weekly_in_amount, weekly_out_amount, daily_in_amount, monthly_in_unique_amount, monthly_out_unique_amount, ... Bot/trade volume stats used for wanted_max_stock calc. Written by tradeit-backend. Merged with store_item_stats via mergeWith(add).
store_item_stats SELECT Same schema as trade_item_stats Store-channel volume stats. Conditionally included via pricingIncludeStoreItemStats config flag.
item_eod SELECT item_id, stable_price, bot_trade_price, store_price, instant_sell_price, wanted_max_stock End-of-day historical prices. Used to compute 7d/30d median max stock for wanted_max_stock calculation.
configurations SELECT key, value (varchar), type Runtime config flags — see §10. Values always returned as strings; booleans stored as '1'/'0'.

tradeitConnection DB (main platform DB — TI_MYSQL_* env vars)

TableOpsKey columnsNotes
item_stocks SELECT item_id, bot_stock, container_bot_stock, tradable_container_bot_stock, locked_reserved_items_stock, temp_reserved_items_stock, user_listing_stock Written by tradeit-stock-manager. Effective current stock = botStock + containerBotStock + tradableContainerBotStock − lockedReservedItemsStock − userListingStock.

8. OpenSearch surface

N/A — pricing-manager has no OpenSearch dependency. Prices are read from MySQL by the services that feed OpenSearch (tradeit-backend, tradeit-inventory-parser).

9. Async continuations

pricing-manager has no Bull queues, no pub/sub, no cron jobs. It is purely request-driven — both endpoints are triggered by external callers.

EndpointCalled byCadenceNotes
GET /update-prices [OPEN: who calls this?] ~every 10 min (inferred from pricing cycle in cron catalog) Externally orchestrated. The Redis lock prevents overlap — concurrent calls no-op.
GET /snapshots/take [OPEN: who calls this?] Unknown cadence Point-in-time snapshot. Must be called regularly to keep the last-30-snapshots mean meaningful for stable price.

10. Configuration flags & guards

All flags read from steamtrade.configurations via AdminConfigService.getByKey(). Values are varchars; boolean flags use '1'/'0' and are compared with === '1'.

KeyTypeRead atEffect
weeklyInOutRatioWeightfloatapp.service.ts:106Weight for weekly in/out trend in wanted_max_stock calculation.
wantedMaxStockLowCapacityRatiofloatapp.service.ts:107Cuts max stock proportionally when bot capacity is low.
blockDepositByPrevMonthPricebool ('1'/'0')price-adjuster.service.ts:584When '1', sets deposit price to 0 for items where price dropped vs prev-month average.
adjustByLiveToStablePriceRatioboolprice-adjuster.service.ts:585Enables live-to-stable price ratio adjustment pass.
blockDepositByAvg7DStablePriceboolprice-adjuster.service.ts:586Blocks deposits when live price dips below 7-day stable average threshold.
liveToStableModifierfloatprice-adjuster.service.ts:587Scaling factor for the live-to-stable price ratio adjustment.
avg7DStableBotPriceModifierfloatprice-adjuster.service.ts:588Scaling factor when adjusting bot price using 7-day average stable price.
tradePriceMarkupfloatprice-adjuster.service.ts:740Final markup added to bot trade price.
storePriceMarkupfloatprice-adjuster.service.ts:741Markup added to store price.
instantSellPriceAdjustPercentfloat (percent)price-adjuster.service.ts:742Percentage adjustment applied to instant-sell price. Stored as whole number (e.g. 5 = 5%). Divided by 100 at read time.
decreaseDepositPriceForCheapItemsboolprice-adjuster.service.ts:743Reduces deposit price for items below the cheap-item threshold.
pricingIncludeStoreItemStatsboolstore-item-stats.service.ts:17When enabled, merges store-channel volume stats with bot-channel stats for wanted_max_stock calculation.

Hardcoded constants

ConstantValueFile:linePurpose
TRADE_MULTIPLIER1.75price-adjuster.service.ts:27Core ratio between tradeit price and weighted market price.
Multiplier.Base1.75base-price-calculator.enum.ts:2Default Multiplier tier for base bot/player price calculation.
Multiplier.Low1.725base-price-calculator.enum.ts:3Used for low-margin-group items.
Multiplier.High1.775base-price-calculator.enum.ts:4
Multiplier.VeryHigh1.81base-price-calculator.enum.ts:5
Multiplier.Extreme1.84base-price-calculator.enum.ts:6
OVERSTOCK_DEFICIT-1price-adjuster.service.ts:26Deficit value assigned when item is overstocked (>30 stock & daily incoming ≥ daily max).
LAST_SNAPSHOT_PRICES_NUM30stable-price-calculator.service.ts:17Number of historical snapshots used for stable price mean.
TOP_PERCENTILE0.70stable-price-calculator.service.ts:18ECDF quantile upper bound — filters out top 30% of prices.
UNSURE_PRICE_MULTIPLIER1.75price-adjuster.service.protections.ts:9Fallback multiplier applied to stablePrice when finalBot is zero (Doppler protection).
TRADEIT_MARKET_SHARE_MAX0.50item-prices-raw.service.protections.ts:93Cap: tradeit cannot target holding > 50% of total market supply for an item.
TRADEIT_MARKET_SHARE_NORMAL0.30item-prices-raw.service.protections.ts:94For high-value items (>250 USD stable), cap at 30% market share.
TRADEIT_HIGH_STABLE_PRICE25000 (¢)item-prices-raw.service.protections.ts:95$250 stable price — threshold for stricter market-share cap.
PERCENTAGE_CHANGE_THRESHOLD0.03price-adjuster.service.ts:28Maximum allowed percentage change for monthly price averages.
MarginGroupThreshold.VeryHigh0.95base-price-calculator.enum.ts:10Margin group tier thresholds.

11. Failure modes

Redis lock not released on crash

Trigger: Node process killed/OOM during a run before the finally block in redis.decorator.ts executes.
Blast radius: All subsequent GET /update-prices calls silently no-op for up to 1 hour (lock TTL). No error surfaced to caller — returns 200 immediately.
Recovery: redis-cli DEL updatePrices on the ElastiCache Redis, or wait for TTL expiry.

csgoskins.gg API failure

Trigger: HTTP error from csgoskins.gg /prices endpoint (rate limit, downtime).
Blast radius: Pipeline aborts mid-run (axios catchError throws). item_prices_raw not updated for this cycle. Redis lock is released in finally so next trigger retries normally.
Recovery: Automatic — next scheduled trigger retries. No circuit breaker implemented.

MySQL connection failure (either DB)

Trigger: Network interruption or Aurora failover during run.
Blast radius: Pipeline throws, prices stale until next successful run. Docker restart policy (--restart unless-stopped) restarts the container.
Recovery: Container auto-restarts. Health endpoint (/health) pings both connections — useful to identify which DB is down.

Stale pricing (slow run / high item count)

Trigger: Run takes longer than the trigger interval. Since the Redis lock is held for the duration and up to 3600 s, overlapping requests are silently dropped.
Blast radius: Prices may lag behind market for items processed late in the pipeline.
Recovery: Monitor run duration via @timer() logs on AppService.updatePrices.

Doppler fetch ban

Trigger: Using axios instead of fetch for Doppler items in tradeit-buff-scrapper. This is an upstream scraper issue but worth noting.
Blast radius: Doppler items get no buff data → PriceAdjusterServiceProtections.protectDopplers applies the UNSURE_PRICE_MULTIPLIER (1.75) fallback on stablePrice.
Recovery: Fix is in tradeit-buff-scrapper — Doppler requests MUST use native fetch.

12. Observability

13. Code map (file:line)

ComponentFile:lineNotes
HTTP entry — GET /update-pricessrc/app.controller.ts:12Calls appService.updatePrices()
HTTP entry — GET /snapshots/takesrc/item-prices-snapshot/item-prices-snapshot.controller.ts:11Calls snapshotService.takeSnapshot()
HTTP entry — GET /healthsrc/health/health.controller.ts:25NestJS Terminus: 2 DB pings + disk + heap
Pipeline orchestratorsrc/app.service.ts:75AppService.updatePrices() — 12-step pipeline
Redis lock decoratorsrc/app.service.ts:73@LockWithRedis('updatePrices', 60 * 60)
Lock implementationsrc/common/redis/redis.decorator.ts:7SET key 'locked' NX EX ttl via LockService
Redis client setupsrc/common/redis/redis.module.ts:16Creates client using TI_REDIS_HOST/TI_REDIS_PORT
Step 1 — Buff data fetchsrc/buff-prices-scraper/buff-prices-scraper.service.ts:16Reads buff_prices_scraper WHERE app_id=730
Step 2 — Stock joinsrc/item-stocks/item-stocks.service.ts:18Reads item_stocks (tradeitConnection)
Step 3 — csgoskins fetchsrc/csgoskins-prices-fetcher/csgoskins-prices-fetcher.service.ts:21HTTP GET /prices via HttpService (axios)
Step 4 — Stable pricesrc/stable-price-calculator/stable-price-calculator.service.ts:27ECDF + quantileSeq (mathjs) + mean of last 30 snapshots
Snapshot mean lookupsrc/item-prices-snapshot/item-prices-snapshot.service.ts:23Reads item_prices_snapshot, locked with @LockWithRedis('takeSnapshot', 3600)
Step 5 — Margin groupssrc/pricing-groups/pricing-groups.service.ts:22Finds tier by findLast(lte(minPrice, price)); cached 2 min
Step 6 — Item stats mergesrc/app.service.ts:211Parallel read of trade_item_stats + store_item_stats; merged via mergeWith(add)
Step 7 — Base pricessrc/base-price-calculator/base-price-calculator.service.tsApplies Multiplier enum tiers based on marginGroup
Multiplier enumsrc/base-price-calculator/base-price-calculator.enum.ts:1Base=1.75, Low=1.725, High=1.775, VeryHigh=1.81, Extreme=1.84
Step 8 — Previous pricessrc/item-prices-raw/item-prices-raw.service.tsReads previous item_prices_raw row per item
Step 9 — EOD pricessrc/item-eod/item-eod.service.ts:49getDayAvg and getDayMedian for 7d/30d columns
Step 11 — Deficit calcsrc/price-adjuster/price-adjuster.service.ts:48calcDeficit() — inventory balance algorithm. OVERSTOCK_DEFICIT=-1
Step 12 — Final adjust + writesrc/price-adjuster/price-adjuster.service.ts:735adjustFinalPrices() — spread ratio, markup flags, store price, instant-sell
Doppler protectionsrc/price-adjuster/price-adjuster.service.protections.ts:18Fallback: stablePrice * UNSURE_PRICE_MULTIPLIER (1.75) when finalBot = 0
Market-share protectionsrc/item-prices-raw/item-prices-raw.service.protections.ts:93Caps wanted_max_stock at 50% (30% for high-value items) of total market supply
Admin config readersrc/admin-config/admin-config.service.ts:12Reads from configurations table; returns null when key absent
Module wiringsrc/app.module.ts:43Two TypeORM connections: default (MYSQL_*) + tradeitConnection (TI_MYSQL_*)
Bootstrapsrc/main.tsListens on port 3000

14. Open questions

OPEN-1: Who triggers GET /update-prices?

The service has no internal cron. Something external calls GET /update-prices on schedule. Candidates: tradeit-backend cron, a dedicated scheduler service, or a crontab on the same EC2. Owner to confirm.

OPEN-2: Who triggers GET /snapshots/take?

Same question — the snapshot endpoint is separate from the main pipeline and must be called independently to keep the last-30-snapshots window fresh. Cadence unknown.

OPEN-3: Datadog / Cronitor integration

No Datadog agent config or Cronitor ping found in the repo. Confirm whether production container has a sidecar agent or if this service is currently unmonitored.

OPEN-4: popularity-marker is a placeholder

src/popularity-marker/popularity-marker.service.ts:60markPopularity() has // Placeholder for actual implementation. The in-memory popularity map is populated (shuffleArray logic) but never persisted. Confirm whether this is WIP or intentionally disabled.

OPEN-5: console.log debug statements in production

app.service.ts and price-adjuster.service.ts have bare console.log() calls that log config values on every run. These should be converted to this.verbose() or removed to reduce log noise in production.

Generated 2026-05-15 · tradeit.gg engineering · Owner: pricing team