Skip to content

ranking-indexer: add a liveness mechanism for the Kafka consumer #736

Description

@ohohoreilly

Problem

The ranking-indexer deployment (live in prod in the knowledge namespace as of #730) has no liveness probe — same as the other Kafka-consumer indexers (kg-indexer, scoring vote/topology indexers). A Kafka consumer can wedge without the process exiting (stuck partition, dropped broker connection, deadlock in the consume loop). With no probe, Kubernetes sees the container as healthy and never restarts it, so a hung consumer fails silently1/1 Running, 0 restarts, but processing nothing.

Why it's not trivial

These consumers expose no HTTP port, so a httpGet probe isn't directly usable. Options:

  • Heartbeat file + exec probe: the consume loop touches a file (e.g. /tmp/healthy) on each successful poll/commit; an exec liveness probe fails if the file is older than N seconds.
  • Lightweight HTTP health endpoint: add a tiny /healthz that reports last-progress timestamp; use httpGet. More work, also enables readiness.
  • Kafka consumer-lag/assignment check via an exec script.

Scope

  • Pick a mechanism (heartbeat-file is the lowest-effort first step).
  • Wire it into ranking-indexer/src/main.rs consume loop and the k8s/{staging,production}/ranking-indexer.yaml manifests.
  • Consider generalizing the same pattern to the other consumer indexers (kg-indexer, scoring-service/*) in a follow-up — they share the gap.

Acceptance

  • A wedged/hung consumer (no progress for N seconds) gets restarted by k8s.
  • Probe doesn't flap under normal idle (no edits to process) — base it on "consumer healthy/assigned," not "edit processed recently," or set N generously.

Context: surfaced during the #730 production deploy review. Related prior art: #344 (kg-indexer observability).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions