No supported way to rewind a sink watermark: recovering a drop-but-advance gap is a hand-edit that races the daemon

[bgmcmullen]

Summary

#958 established that a withheld source is dropped and the watermark still advances, so the un-shipped rows stay in the local cache while the sink believes it has moved past them. That issue explicitly defers recovery ("separate ops task, not this issue"). This issue is that deferred half: there is no supported way to perform it.

Recovering a drop-but-advance gap requires hand-editing a private state file that the running daemon overwrites roughly once a minute. There is no verb for it, no --since, no dry-run, and no way to see whether the edit survived.

What's missing

hyp sink has exactly one subcommand:

Subcommands:
  maintain  Run export maintenance (snapshot expiration; data-file compaction with --compact)

Same in deployed 1.22.0 and current master 1.24.0. Nothing resends, rewinds, or resets a watermark. --since exists only on hyp backfill, which imports client history into the cache — the opposite direction.

So the only route is editing:

<stateDir>/plugins/@hypaware/central/sink-instances/central/watermarks/ai_gateway_messages/source=claude.json
{ "v":1, "continuation":{"v":1,"seq":"674822"}, "exportedRowCount":255086, "updatedAt":"..." }

Why the hand-edit races

forwardPartition re-reads the watermark from disk every tick (central/src/sink.js:218), which is good — an edit needs no restart. But the same tick rewrites it at the end (sink.js:346). Measured on the Neutral container:

04:03:28  seq 675930
04:04:37  seq 675948     → 69s apart

That is the entire window. Edit outside it and the next tick silently re-advances past the rewind; nothing reports that the edit was discarded, because from the sink's perspective nothing unusual happened.

Avoiding the race by stopping the daemon is not free: the daemon owns the capture gateway (127.0.0.1:18521), which on this box had live ESTABLISHED connections from running agents. Stopping it to fix an export bookkeeping value interrupts unrelated in-flight capture.

Concrete stakes

On Neutral today: 488,016 rows cached, exportedRowCount frozen at 255,086 since 08-12, ~236,650 rows recoverable. The correct resume point (last seq at/before the last-landed row 08-12T00:50:47Z) is 355254, verified against the complete 488,088-row table and cross-checked across four independent compaction generations, which agree exactly (next seq after the boundary: 355257).

Deleting the watermark instead is not a workaround: with no since, readRowsSince full-scans and includeLegacy defaults true, re-sending all 255k already-delivered rows on chunk boundaries that won't match the idempotency ledger (#958 makes the same point).

Fix directions

• A first-class verb, e.g. hyp sink rewind <instance> --dataset <d> --partition <p> --to-seq <n> (or --since <timestamp>, resolving the seq internally, since operators reason in time, not seq).
• Make it safe against the tick: take the sink instance's lock, or apply the rewind through the driver rather than the file.
• --dry-run reporting how many rows the rewind would re-send.
• Failing a verb: make the watermark file's hand-edit path documented and safe — detect an externally-modified watermark (compare against last-written value) and honor it rather than clobbering it.
• Related: an operator needs to know the seq for a wall-clock instant, but _hyp_ingest_seq is in INTERNAL_FIELDS and stripped from both readRows and the query layer (SELECT _hyp_ingest_seq → "Column not found"), so deriving it today means reading the parquet files directly.

🤖 Generated with Claude Code

[philcunliffe]

Stuck: this is a missing capability, not a bug

What neutral was doing. This issue came up in the fix queue and was read
before dispatch. It was not dispatched, because there is no defect to fix.

Why it cannot proceed as a fix. The issue is explicit that nothing is
misbehaving: forwardPartition re-reading the watermark from disk every tick
(central/src/sink.js:218) is called out as correct, and the gap is that
"there is no supported way to perform it". hyp sink has exactly one
subcommand, maintain; nothing resends, rewinds, or resets a watermark, and
--since exists only on hyp backfill, which moves data the other way.

So closing this means designing and adding a new operator surface: a verb, its
name and scope, whether it takes a timestamp or a sequence, whether it is
per-source or per-dataset, what a dry run shows, and how it interacts with a
daemon that rewrites the file about once a minute. That is a new capability,
and the reconciler's rule is to escalate a missing feature rather than force it
through as a bug fix.

Why no request LLP was minted yet, and the question that comes with it.
Two request LLPs are already open and awaiting a decision from this same
backlog: LLP 0381 in #1368 (boundary-set overflow) and LLP 0382 in
#1374 (doorless listener answers absolute-form on the control path). Both are
green and mergeable and neither has been dispositioned. Minting a third before
any of them is answered would add to a queue rather than move work, so neutral
is asking first instead of assuming.

What it needs from you: one of three answers.

• "Mint the RFC." Neutral will write a request LLP stating the recovery
  problem, the racing hand-edit it replaces, and the candidate shapes for the
  verb, choosing none. It joins LLP 0381 request: an overflowing watermark second re-appends its overflow every tick, and the exit needs a decision #1368 and LLP 0382 request: the control surface answers absolute-form exactly where LLP 0247 said it never would #1374 awaiting decisions.
• "Design it directly." If you already know the shape you want, say so and
  it becomes ordinary implementation work with a normal fix PR.
• "Not now." It stays recorded and neutral stops offering it each tick.

Context worth having for the decision. This is the deferred half of #958,
which established that a withheld source is dropped while the watermark still
advances, leaving un-shipped rows in the local cache and a sink that believes
it has moved past them. Recovery today means hand-editing
<stateDir>/plugins/@hypaware/central/sink-instances/central/watermarks/...
while the daemon overwrites it, with no verb, no dry run, and no way to see
whether the edit survived.

How to unstick. Reply with a comment on this issue; neutral monitors this
thread and will re-engage with your guidance on its next tick.