← Lab
Lab project
CL3 — Crypto Trading Data Service
In productionCrypto trading data2024 — 2025
FastAPI + MongoDB + Celery + Redis market-data stack.
Production-shape crypto data service: Binance ingestion, seasonality pattern detection, Supabase-backed auth, Celery / Redis task pipeline, Docker-compose deployment. Subscription-gated endpoints.
Architecture
How it's built
A FastAPI service over MongoDB 7 with a Celery worker pool for the long Binance pulls and the seasonality math. Redis carries the broker, the task results, and a 300-second cache of Supabase token validations so the auth check never becomes the bottleneck. Every endpoint is subscription-gated at the API layer — workers never see an unauthorised request.
Data flow
- 01Client calls /api/market-data/fetch with a Bearer token — FastAPI checks the 300s Redis cache, falling back to Supabase only on miss.
- 02Subscription tier is validated; unauthorised requests get a 403 before any work is enqueued.
- 03Authorised job is pushed to the Celery broker DB and a task id is returned to the client.
- 04Worker pages through Binance klines (1000 candles per call) with tenacity backoff on rate limits.
- 05Each batch is bulk-inserted into MongoDB; progress percentage is updated in Redis on every page.
- 06Seasonality patterns (SMA crossover, mean reversion) are computed in-process by Pandas and written back to MongoDB.
- 07Client polls /api/task/{id}/progress — FastAPI reads from Redis; the final result is served from MongoDB.
Design decisions
- MongoDB over Postgres — OHLCV is natively timeseries, and the bulk inserts with composite _id = {symbol}_{timestamp} stay duplicate-safe without an extra unique index.
- 300 s Supabase token cache in Redis — the same JWT can be reused for the burst of follow-up reads after a chart loads, cutting Supabase RPS to a trickle.
- tenacity-wrapped Binance client — exponential backoff on 429s keeps the worker pool from getting rate-limit banned.
- Celery progress is tracked in Redis so the client can show a real percentage for a 5-minute klines pull instead of a spinner.
- Flower runs on :5555 in production — task latency and failure rate are visible without bolting on a separate monitoring stack.
External integrations
Supabase (JWT + subscription tier)Binance APIMongoDB 7Redis (broker + token cache)Flower