Skip to content

thesahibnanda-max/helios

BranchesTags

Repository files navigation

Helios

Production-grade distributed real-time event streaming + analytics platform. Boots on a laptop via docker compose up. Built end-to-end with OOP/SOLID, full tests, no orchestrator required.

What Helios is (5-line explanation)

Imagine your app generates thousands of events per second — payments, clicks, logins. Helios is the pipeline that:

  1. Catches every event via a fast HTTP/gRPC API.
  2. Stores every raw event durably so nothing is ever lost.
  3. Watches the stream live for fraud, anomalies, and trends.
  4. Saves processed data into 2 databases: one for full history, one for instant dashboards.
  5. Notifies systems (Slack/email/webhooks) when something matters.

Same shape as Stripe Radar + Snowplow + Segment + Datadog Live Monitoring, all open-source, all in one repo.

Why this exists

Most companies bolt 8-10 vendors together (Segment + Snowplow + Datadog + PagerDuty + Looker + Snowflake + Slack webhooks + custom glue). Helios shows the whole pipeline as one coherent system, with a single codebase, single docker-compose, single architectural philosophy. Useful as:

  • Reference implementation for distributed systems engineers
  • Senior backend engineering showcase
  • Drop-in real-time analytics base for a startup

How it works (layman version)

┌──────────────┐
│ Your client  │  curl, JS SDK, mobile app, server
└──────┬───────┘
       │ HTTP POST /v1/events (JSON + JWT)
       ▼
┌──────────────────────────────────────────────┐
│ Gateway (port 8080)                          │  edge: auth + rate-limit
│ - Verifies JWT signature                     │
│ - Checks tenant + idempotency key            │
│ - Translates JSON → gRPC                     │
└──────┬───────────────────────────────────────┘
       │ gRPC
       ▼
┌──────────────────────────────────────────────┐
│ Ingestion (port 8081)                        │  contract gate
│ - Validates JSON against Schema Registry     │
│ - Dedup via Redis idempotency key            │
│ - Serializes to protobuf envelope            │
│ - Writes to Kafka (events.raw.v1)            │
└──────┬───────────────────────────────────────┘
       │
       ▼
┌──────────────────────────────────────────────┐
│ Kafka (port 19092)                           │  durable replayable log
│ Topics: raw, enriched, fraud, anomaly,       │
│   alerts, dlq, cdc.users, cdc.merchants      │
└──────┬───────────────────────────────────────┘
       │
       ├──────────────────────┐
       ▼                      ▼
┌──────────────────┐  ┌───────────────────────────────┐
│ Storage svc      │  │ Flink Jobs (port 8082 UI)     │
│ → Cassandra      │  │  • Enrichment (CDC join)      │
│   raw_events     │  │  • Fraud detection (rules)    │
│   (full history) │  │  • Anomaly (Welford z-score)  │
└──────────────────┘  └────────┬──────────────────────┘
                               │
                  ┌────────────┼──────────────┐
                  ▼            ▼              ▼
        events.enriched   events.fraud   events.anomaly
                  │            │              │
                  ▼            ▼              ▼
        ┌──────────────────────────────────────────┐
        │ Analytics svc → ClickHouse               │  fast dashboards
        │  events_raw, events_1m, 1h, 1d           │
        └──────────────────────────────────────────┘
                  │
                  ▼
        ┌──────────────────────────────────────────┐
        │ Notification svc → webhooks (Slack etc)  │
        │  Dedup via Redis                         │
        └──────────────────────────────────────────┘

Observability sidecar: OTel Collector → Jaeger (traces) + Prometheus (metrics) + Loki (logs) + Grafana (UI)

Plain English summary:

  • Gateway = bouncer at door (checks ID).
  • Ingestion = receptionist (writes name in book).
  • Kafka = giant logbook (never erased, replay anytime).
  • Flink = security cameras watching the lobby live.
  • Cassandra = filing cabinet (every event by tenant/date).
  • ClickHouse = spreadsheet for managers (pre-aggregated).
  • Notification = alarm system.

Quickstart (3 commands)

git clone <repo> && cd helios
make up                  # cold-boot full stack (~3-5 min first time, ~90s warm)
make seed                # push 100 synthetic events end-to-end

Open:

System requirements:

  • 12 GB RAM allocated to Docker (Rancher/Docker Desktop → Preferences → Resources)
  • 6 CPU cores
  • 20 GB disk free
  • Java 21 + Maven 3.9 (only if rebuilding outside Docker)

Send your first event (working curl)

SECRET="helios-local-dev-secret-change-me-32-bytes-minimum"
TENANT="01JHELIOSTENANT00000000000"

# Generate HS256-signed JWT (1 day TTL)
TOKEN=$(python3 -c "
import hmac, hashlib, base64, json, time
hdr=base64.urlsafe_b64encode(json.dumps({'alg':'HS256','typ':'JWT'}).encode()).rstrip(b'=')
pld=base64.urlsafe_b64encode(json.dumps({'sub':'dev','tid':'$TENANT','exp':int(time.time())+86400}).encode()).rstrip(b'=')
sig=base64.urlsafe_b64encode(hmac.new(b'$SECRET', hdr+b'.'+pld, hashlib.sha256).digest()).rstrip(b'=')
print((hdr+b'.'+pld+b'.'+sig).decode())
")

# Build timestamp (millisecond precision RFC 3339)
TS=$(python3 -c "import datetime; print(datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%S.%fZ')[:-4]+'Z')")

curl -sS -X POST http://localhost:8080/v1/events \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Tenant-Id: $TENANT" \
  -H "X-Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d "{
    \"event_type\": \"payment.attempt\",
    \"event_version\": 1,
    \"entity_id\": \"user_1\",
    \"event_time\": \"$TS\",
    \"payload\": {
      \"payment_id\": \"pay_demo_1\",
      \"user_id\": \"user_1\",
      \"merchant_id\": \"mer_1\",
      \"amount\": { \"currency\": \"INR\", \"minor_units\": 5000 },
      \"method\": \"PAYMENT_METHOD_CARD\",
      \"status\": \"STATUS_AUTHORIZED\",
      \"device_id\": \"dev_1\",
      \"ip_address\": \"10.0.0.1\"
    }
  }"

Expect HTTP 202 + {event_id, received_at, deduplicated:false}.

Consume / inspect events

Kafka raw topic (protobuf bytes)

docker exec helios-kafka kafka-console-consumer \
  --bootstrap-server kafka:9092 \
  --topic events.raw.v1 --from-beginning --max-messages 5 --timeout-ms 5000

Other topics: events.enriched.v1, events.fraud.v1, events.anomaly.v1, events.alerts.v1, events.dlq.v1, events.cdc.users.v1, events.cdc.merchants.v1.

Cassandra (full history, replayable)

make cqlsh   # interactive
# or one-shot:
docker exec helios-cassandra cqlsh -e "
USE helios;
SELECT tenant_id, event_id, event_type, entity_id
FROM raw_events
WHERE tenant_id='01JHELIOSTENANT00000000000'
  AND day_bucket='2026-05-15'
LIMIT 10;"

Tables: raw_events, dlq_events, replay_audit.

ClickHouse (aggregated, fast queries)

make clickhouse-cli
# or:
docker exec helios-clickhouse clickhouse-client -u helios --password helios -q "
SELECT count(), event_type FROM helios.events_raw GROUP BY event_type"

Tables: events_raw, events_1m, events_1h, events_1d, alerts.

Analytics query API

curl -sS -X POST http://localhost:8083/v1/analytics/query \
  -H "X-Tenant-Id: 01JHELIOSTENANT00000000000" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT count() FROM events_raw WHERE event_type='\''payment.attempt'\''"}'

Flink job state

curl -sS http://localhost:8082/jobs/overview | python3 -m json.tool

All make targets

make up                          # boot full stack + wait-for-green + print URLs
make down                        # stop services; volumes preserved
make nuke                        # stop + destroy volumes (DESTRUCTIVE)
make ps                          # container status
make logs                        # tail every service
make logs SVC=ingestion-service  # tail single service
make shell SVC=gateway-service   # interactive shell inside a container
make build SVC=gateway-service   # rebuild + restart one service
make build-all                   # rebuild every service image
make proto                       # regenerate protobuf code via buf

# Pipeline operations
make seed                        # push 100 synthetic events through pipeline
make load                        # k6 ingest_steady: 100k ev/s for 5 min
make probe                       # synthetic latency probe gateway → cassandra
make smoke                       # full E2E health check
make replay TOPIC=events.raw.v1 GROUP=storage-writer TS=2026-05-15T06:00:00Z
                                 # reset consumer-group offset to a timestamp
make chaos EXP=kafka-broker-kill # run a chaos experiment

# DB shells
make cqlsh                       # Cassandra REPL
make clickhouse-cli              # ClickHouse REPL
make kafka-console               # Kafka admin shell

# Quality
make test                        # run all unit + integration tests
make lint                        # lint proto + gradle + go

Service ports

Service Host port Purpose
gateway-service 8080 REST ingest API
ingestion-service 8081 / 9190 actuator / gRPC (internal: 9090)
analytics-service 8083 Query API + actuator
storage-service 8084 actuator
notification-service 8086 actuator
flink-jobmanager 8082 Flink UI
kafka (host clients) 19092 external clients (k6, debug tools)
schema-registry 8085 Subjects API
grafana 3000 Dashboards (admin / helios)
jaeger 16686 Traces UI
prometheus 9090 Metrics scrape + UI
loki 3100 Log push API
cassandra 9042 CQL
clickhouse 8123 HTTP API
redis 6379 Idempotency + dedup
minio console 9101 Flink checkpoint browser (helios / heliosheliios)
minio S3 API 9100 S3 endpoint

Stack components (all locked versions)

Layer Component Version Role
Languages Java 21 Services + Flink jobs
Go 1.22 helios-cli operator binary
Framework Spring Boot 3.3.4 gateway, ingestion, analytics, storage, notification
Streaming Kafka 3.7.1 (KRaft) event log
Schema Registry 7.6.1 JSON schema versioning
Flink 1.19 enrichment + fraud + anomaly jobs
Storage Cassandra 4.1 raw event history (TTL=90d)
ClickHouse 24.3 rollup analytics (1m / 1h / 1d MVs)
Redis 7 idempotency + dedup TTL state
MinIO latest S3-compatible Flink checkpoint store
Serialization Protobuf 4.28.2 wire format (envelope + payloads)
gRPC 1.66.0 gateway → ingestion
Observability OpenTelemetry 1.42 / agent 2.x auto-instrumentation
Prometheus 2.54 metrics
Grafana latest dashboards
Loki 3.x log aggregation
Jaeger latest distributed traces
Resilience Resilience4j 2.2.0 circuit breaker (gateway → ingestion)
Caffeine 3.1.8 local schema cache
Auth Nimbus JOSE+JWT 9.41 HS256 JWT verification
Build Gradle 8.10 (services) composite build
Maven 3.9 (flink-jobs) Flink ecosystem standard
Docker Compose v2.37 local orchestration

Repo layout

helios/
├── Makefile               operations (make up, seed, etc)
├── docker-compose.yml     full stack definition (19 containers)
├── .env.example           local env defaults
├── design/                ADRs, runbooks, architecture, dashboards spec
├── proto/                 protobuf source of truth (.proto files)
├── services/              5 Spring Boot services + common-libs (Gradle)
│   ├── platform-bom/        version pinning (BOM)
│   ├── common-libs/
│   │   ├── helios-proto/    generated protobuf classes
│   │   ├── helios-kafka/    Kafka producer/consumer wrapper
│   │   ├── helios-otel/     OTel SDK config + structured logging
│   │   ├── helios-schema/   schema-registry client + validator
│   │   ├── helios-cassandra/ Cassandra driver wrapper
│   │   └── helios-clickhouse/ ClickHouse HTTP client
│   ├── gateway-service/     REST ingest, JWT, rate-limit, circuit breaker
│   ├── ingestion-service/   gRPC, schema validation, idempotency, Kafka produce
│   ├── analytics-service/   ClickHouse writer + query API
│   ├── storage-service/     Cassandra writer + replay infra
│   └── notification-service/ alerts → webhooks (Slack etc)
├── flink-jobs/            3 Flink applications (Maven)
│   ├── common/              shared codec + Kafka source helpers
│   ├── enrichment-job/      CDC join → events.enriched.v1
│   ├── fraud-detection-job/ rule engine + broadcast state
│   ├── anomaly-detection-job/ Welford online stats z-score
│   └── cdc-seeder/          test-fixture CDC producer
├── schemas/
│   ├── cassandra/           CQL DDL migrations
│   ├── clickhouse/          ClickHouse DDL + materialized views
│   └── schema-registry/     JSON schema subjects
├── infra/
│   ├── docker/              per-service Dockerfiles + base-jre + bootstrap
│   ├── bootstrap/           one-shot init (topics, DDL, schemas)
│   ├── grafana/             dashboards JSON + datasource provisioning
│   ├── prometheus/          scrape config + alert rules
│   ├── otel-collector/      pipeline: receivers → exporters
│   ├── loki/                log aggregator config
│   └── promtail/            docker log shipper
├── scripts/
│   ├── dev/                 wait-for-green, seed, smoke, print-urls, probe
│   └── helios-cli/          Go operator binary (replay, DLQ, debug)
├── benchmarks/
│   ├── k6/                  ingest_steady, ingest_burst, query_latency
│   └── chaos/               kafka-broker-kill, flink-tm-kill, redis-partition
└── tests/
    ├── compose-smoke/       boot health + minimal pipeline
    ├── chaos/               failure-mode tests
    └── e2e/                 cross-service integration

How auth works (local dev)

Gateway verifies HS256-signed JWT with the shared secret from JWT_HS256_SECRET env (helios-local-dev-secret-change-me-32-bytes-minimum by default).

Required token claims:

  • tid — tenant id (must match X-Tenant-Id header if both present)
  • exp — expiry epoch seconds

Required headers:

  • Authorization: Bearer <jwt>
  • X-Tenant-Id: <tenant_id>
  • X-Idempotency-Key: <unique-per-request> (any string; repeats return original event_id)

Bypass JWT for local dev: set JWT_HS256_SECRET= (empty) in .env and recreate gateway. Then any non-empty bearer token works.

Production overlay: swap the filter for JWKS-backed verification (see design/adr/0007_auth_strategy.md).

How data flows (event lifecycle)

  1. Client POSTs JSON to gateway with JWT.
  2. Gateway verifies JWT, extracts tenant, forwards via gRPC to ingestion with deadline 2s.
  3. Ingestion fetches the JSON schema for event_type@version from schema-registry (Caffeine-cached), validates body, checks Redis for idempotency key (returns existing event_id on hit), serializes a protobuf EventEnvelope wrapping the typed payload, produces to Kafka events.raw.v1 partition by hash(tenant_id, entity_id).
  4. Kafka persists with retention 7d.
  5. Flink enrichment consumes raw, joins user + merchant CDC broadcast streams, emits to events.enriched.v1.
  6. Flink fraud consumes enriched + CDC broadcast, runs rule engine (AmountThresholdRule, VelocityRule, GeoMismatchRule, UnknownMerchantRule), emits scored events to events.fraud.v1 and alerts to events.alerts.v1.
  7. Flink anomaly consumes enriched, maintains Welford online mean/variance per (tenant, entity, metric), emits z-score anomalies to events.anomaly.v1 + critical alerts to events.alerts.v1.
  8. Storage consumes raw, writes to Cassandra raw_events table partitioned by (tenant_id, day_bucket).
  9. Analytics consumes enriched, writes to ClickHouse events_raw, MVs auto-rollup into events_1m, _1h, _1d.
  10. Notification consumes alerts, dedupes via Redis, fans out webhooks.

Every step is OTel-instrumented — full trace from curl → cassandra visible in Jaeger.

Failure modes (already handled)

  • Idempotency on retry → ingestion dedups via Redis (X-Idempotency-Key), returns original event_id.
  • Schema invalid → 400 + payload to DLQ.
  • Kafka down → gateway circuit breaker opens after 5 consecutive failures, returns 503 with Retry-After: 1.
  • Redis down + IDEMPOTENCY_FAIL_OPEN=true → ingestion processes events without dedup (logs warn), preserves availability.
  • Flink job crash → restarts from last checkpoint (S3/MinIO), exactly-once via Kafka transactional producer.
  • Cassandra slow → storage-service backpressures consumer; offsets only committed on successful flush.
  • Late events past anomaly grace window → DLQ (events.dlq.v1).
  • CDC seeder empty → fraud UnknownMerchantRule becomes louder (graceful degrade).

Tests

make test                       # all unit + integration
make smoke                      # boot + minimal pipeline assertion
cd tests/compose-smoke && go test ./...    # smoke directly
cd tests/chaos && bash ./run.sh kafka-broker-kill
cd tests/e2e && go test -v ./...

Coverage targets: services ≥80%, common-libs ≥90%, Flink jobs ≥75%.

Observability

Signal Tool URL
Metrics Prometheus http://localhost:9090
Dashboards Grafana http://localhost:3000
Logs Loki (via Grafana) http://localhost:3000/explore
Traces Jaeger http://localhost:16686
Flink jobs Flink UI http://localhost:8082

Default dashboard: http://localhost:3000/d/helios-overview

Every service auto-instruments via OTel javaagent (mounted in helios/base-jre image at /opt/otel/javaagent.jar). No code-level instrumentation needed. Spans propagate via W3C traceparent header end-to-end.

Common operations

Tail logs

make logs SVC=gateway-service
make logs SVC=ingestion-service
make logs SVC=storage-service
make logs SVC=analytics-service
make logs SVC=notification-service

Inspect a single trace

# Get a request id from gateway logs, then:
open http://localhost:16686/search?service=gateway-service

Replay events from a timestamp

make replay TOPIC=events.raw.v1 GROUP=storage-writer TS=2026-05-15T06:00:00Z
# audit row written to helios.replay_audit

Rebuild + redeploy a single service after code change

make build SVC=gateway-service

Restart Flink jobs

# Cancel all jobs (returns slots):
for jid in $(curl -fsS http://localhost:8082/jobs/overview | python3 -c "import sys,json;[print(j['jid']) for j in json.load(sys.stdin)['jobs'] if j['state']=='RUNNING']"); do
  curl -sS -X PATCH http://localhost:8082/jobs/$jid
done
# Re-submit:
docker compose up -d --force-recreate flink-job-submitter

Tear it all down

make down      # keeps volumes (data preserved)
make nuke      # wipes everything

Troubleshooting

Symptom Cause Fix
make up hangs at "building services" Rancher memory <8 GB Raise to 12 GB in Rancher Desktop → Resources
Bootstrap exits with code 127 Stale image docker rmi helios/bootstrap:dev && make up
Gateway returns 503 Ingestion gRPC deadline too tight or ingestion unhealthy Check INGESTION_DEADLINE_MS in compose (default 2000)
Gateway returns 401 Bad/expired JWT or wrong secret Regenerate via curl snippet above
Flink job FAILED — s3 plugin Plugin not enabled ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.19.3.jar is set on both jm + tm
Flink job NoResourceAvailable Slot exhaustion Bump taskmanager.numberOfTaskSlots in compose
Loki / otel-collector flips unhealthy Memory pressure on host Raise Docker RAM allocation
Cassandra unhealthy on first boot Long start-up (~60s) Wait; make ps until healthy

Detailed runbooks: design/runbooks/.

Production notes

This repo is the dev/local stack. Production overlay (design/infra/kubernetes_optional.md) swaps:

  • 1 Kafka broker → 3-broker MSK/Confluent cluster
  • 1 Flink JM/TM → HA mode, K8s operator
  • 1 Cassandra → 3-node cluster, RF=3, LOCAL_QUORUM
  • 1 ClickHouse → sharded cluster
  • HS256 JWT → JWKS-backed verifier (Auth0/Okta/Cognito)
  • MinIO → S3 (real bucket per env)
  • Compose health checks → K8s readiness/liveness probes

Spec only — Kubernetes path is optional per ADR-0005.

Contributing

  • Significant changes need an ADR (design/adr/).
  • New failure modes need a runbook (design/runbooks/).
  • Performance claims need a benchmark script + write-up (benchmarks/).
  • All public APIs are protobuf-first — modify .proto and regenerate.
  • Tests: PR must include unit tests; cross-service changes need integration tests.

See CONTRIBUTING.md.

License

Apache-2.0.

About

Helios is a production-grade distributed real-time event streaming and analytics platform built with Java, Kafka, Flink, Cassandra, ClickHouse, Redis, and OpenTelemetry. Fully open-source, Docker-first, and designed to demonstrate Principal-level distributed systems, scalability, resiliency, observability, and event-driven architecture.

Resources

Readme

License

Apache-2.0 license

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages