Problem
The ranking-indexer deployment (live in prod in the knowledge namespace as of #730) has no liveness probe — same as the other Kafka-consumer indexers (kg-indexer, scoring vote/topology indexers). A Kafka consumer can wedge without the process exiting (stuck partition, dropped broker connection, deadlock in the consume loop). With no probe, Kubernetes sees the container as healthy and never restarts it, so a hung consumer fails silently — 1/1 Running, 0 restarts, but processing nothing.
Why it's not trivial
These consumers expose no HTTP port, so a httpGet probe isn't directly usable. Options:
- Heartbeat file +
exec probe: the consume loop touches a file (e.g. /tmp/healthy) on each successful poll/commit; an exec liveness probe fails if the file is older than N seconds.
- Lightweight HTTP health endpoint: add a tiny
/healthz that reports last-progress timestamp; use httpGet. More work, also enables readiness.
- Kafka consumer-lag/assignment check via an exec script.
Scope
- Pick a mechanism (heartbeat-file is the lowest-effort first step).
- Wire it into
ranking-indexer/src/main.rs consume loop and the k8s/{staging,production}/ranking-indexer.yaml manifests.
- Consider generalizing the same pattern to the other consumer indexers (kg-indexer, scoring-service/*) in a follow-up — they share the gap.
Acceptance
- A wedged/hung consumer (no progress for N seconds) gets restarted by k8s.
- Probe doesn't flap under normal idle (no edits to process) — base it on "consumer healthy/assigned," not "edit processed recently," or set N generously.
Context: surfaced during the #730 production deploy review. Related prior art: #344 (kg-indexer observability).
Problem
The
ranking-indexerdeployment (live in prod in theknowledgenamespace as of #730) has no liveness probe — same as the other Kafka-consumer indexers (kg-indexer, scoring vote/topology indexers). A Kafka consumer can wedge without the process exiting (stuck partition, dropped broker connection, deadlock in the consume loop). With no probe, Kubernetes sees the container as healthy and never restarts it, so a hung consumer fails silently —1/1 Running,0 restarts, but processing nothing.Why it's not trivial
These consumers expose no HTTP port, so a
httpGetprobe isn't directly usable. Options:execprobe: the consume loop touches a file (e.g./tmp/healthy) on each successful poll/commit; anexecliveness probe fails if the file is older than N seconds./healthzthat reports last-progress timestamp; usehttpGet. More work, also enables readiness.Scope
ranking-indexer/src/main.rsconsume loop and thek8s/{staging,production}/ranking-indexer.yamlmanifests.Acceptance
Context: surfaced during the #730 production deploy review. Related prior art: #344 (kg-indexer observability).