First complete release of the real-time CDC streaming pipeline.
Pipeline: PostgreSQL (OLTP) → Debezium (CDC) → Kafka + Schema Registry (Avro) → Apache Flink → ClickHouse (OLAP) → Grafana. Everything runs locally with a single docker compose up, on 100% synthetic data.
Highlights
- Debezium logical-decoding CDC into Kafka, encoded as Avro with registered schemas
- Four Flink SQL jobs: windowed revenue, top products (sliding window + lookup join), anomaly detection, and revenue by country
- ClickHouse OLAP sink via the native Kafka table engine + materialized views (idempotent ReplacingMergeTree)
- Provisioned Grafana dashboard with five live panels
- GitHub Actions CI (lint + config validation) and no secrets in git