A real-time cryptocurrency price display project built with Django, Flask, and Kafka.
- Producer: A script connects to Binance’s WebSocket server to receive kline (candlestick) data every second and publishes it to a Kafka topic.
- Inserter: Another script consumes the Kafka stream and batches the data for insertion into a PostgreSQL database every minute. During each insertion, it records a checkpoint: the first and last timestamps for each symbol.
- Django Celery Task: A scheduled task runs every minute to read the checkpoints and aggregate the 1-second data into 1-minute intervals. This step is technically unnecessary since Binance already provides 1-minute klines, but it was implemented as a personal challenge.
- Flask Server: Reads data directly from Kafka and serves it to the frontend via WebSocket every second to display live sidebar prices.
To minimize latency, a few heuristic buffer seconds are added throughout the pipeline.
2m *s- Producer: Binance WebSocket → Kafka Topic
- Frontend: Kafka Topic → Web Client
2m 2s- Inserter: Kafka Topic → PostgreSQL kline_1s table
- Inserts data in the range
[1m 2s, 2m 1s]
2m 5s- Django Celery Task: Aggregates data
- Reads checkpoint table
- Aggregates 1-second data from
[1m 0s, 1m 59s]into 1-minute data in aggregated_kline
2m 7s- Frontend: PostgreSQL aggregated_kline, kline_1s → Frontend
- Updates the last data point for 1m 0s using aggregated_kline to show volume
- Adds a new data point for 2m 0s using kline_1s to show the current open price
- The frontend UI is designed with a cursor-based vibe.
- All containers (except for the frontend and producer) are deployed on the same instance to reduce costs.
- To save storage and memory, the PostgreSQL container deletes old data every 30 minutes via crontab, and the Kafka topic has a data retention policy of 1 minute.
- The frontend and producer run on the same instance to accurately measure data latency, avoiding discrepancies from server time differences.
- Although a more scalable design would separate Kafka topics and database tables by symbol, the current setup handles just two symbols, with no plans to add more, so they are grouped together.