Skip to content

fix: delete data down to the remaining watermark on reorg - #294

Open
cdsiren wants to merge 5 commits into
indexsupply:mainfrom
cdsiren:fix-reorg-batch-prefix
Open

fix: delete data down to the remaining watermark on reorg#294
cdsiren wants to merge 5 commits into
indexsupply:mainfrom
cdsiren:fix-reorg-batch-prefix

Conversation

@cdsiren

@cdsiren cdsiren commented Aug 2, 2026

Copy link
Copy Markdown

fix: delete data down to the remaining watermark on reorg

Problem

A reorg can permanently wedge a single integration. It stops converging and logs this
forever, ~26 retries/sec, until someone deletes rows by hand:

msg=converge-retry ig=uniswap_v3_swap msg=inserting data: inserting blocks:
ERROR: duplicate key value violates unique constraint "u_evm_uniswap_v3_swap" (SQLSTATE 23505)

Sibling integrations on the same source keep converging, so it reads as a
single-integration failure rather than a reorg. Restarting does not help, because the
conflicting rows are already committed.

Cause

Task.update writes one shovel.task_updates row per converge batch, with num set
to the batch's last block (task.go:443). One update therefore covers the whole range
(previous update, num].

Task.Delete(n) removes updates at num >= n but tells the destination to delete at
block_num >= n. When the batch spans more than one block, the blocks below n in that
same batch keep their data while the watermark rewinds below them — they are the batch's
canonical prefix, not orphaned-chain data, since only n and above were reorged out.

Converge then resumes from the highest remaining update and re-requests those blocks.
dig.Integration.Insert uses CopyFrom, which cannot express on conflict, so the insert
hits u_<table> and rolls back. Every subsequent poll repeats it.

The window is batchSize > 1, which is the normal case: with a 2-block batch it is roughly
a coin flip per reorg. Concretely, from a production instance on mainnet:

watermark (task_updates.num)   25666514
orphan rows in evm_uniswap_v3_swap   2, at blocks 25666516 and 25666517

The update at 25666518 was deleted along with data >= 25666518; blocks 25666516–17 from
that same batch survived below the rewound watermark. Deleting exactly those two rows
unwedged the integration with no restart, and Shovel re-inserted them byte-identically
(md5 of every re-inserted row matched the pre-delete capture), which confirms they were
canonical rather than reorged data.

Fix

Task.Delete now asks where Converge will actually resume — one past the highest
remaining update — and deletes destination data from there, so no block is ever left
without an update covering it. The new resume value is never greater than n (no update
at or above n survives the delete), so this only ever widens the delete, and only within
a batch that was already being discarded.

With no updates left, the task restarts at its configured start, or at the chain head when
there is none; both are above anything already stored, so that case falls back to n
rather than deleting the destination's whole history.

Re-indexing a batch's earlier blocks is the same work the task was already about to do —
it re-requests that range from the node either way.

Tests

  • TestConverge_ReorgBatchPrefix (new): one update covering blocks 1–2, block 2 reorged
    out, asserts the task converges to the canonical chain. Without the task.go change it
    fails with inserting data: inserting blocks: duplicate key: 1 — the same shape as the
    production error above.
  • testDestination was too permissive to catch this and needed two fixes to match
    dig.Integration: Delete now deletes at >= n rather than the single key n, and
    Insert reports a duplicate instead of silently overwriting, since the real destination
    copies into a table with a unique index and no on conflict. The existing
    TestConverge_Reorg passes under both, because it writes one update per block, so
    batch-granularity never comes into play.
  • Full suite green (go test -p 1 ./..., postgres 15 via pqxtest).

Alternatives considered

  • One task_updates row per block. Removes the granularity mismatch at its root, but
    multiplies watermark writes by the batch size.
  • on conflict do nothing on insert. Not expressible through CopyFrom, and it would
    mask genuine duplicates.

cdsiren and others added 5 commits August 2, 2026 15:08
task_updates gets one row per converge batch, with num set to the batch's
last block, so one update covers (previous update, num]. Task.Delete removed
updates at num >= n but only deleted destination data at block_num >= n,
leaving the batch's canonical prefix -- blocks below n that no remaining
update covers.

Converge resumes from the highest remaining update and requests those blocks
again. Integration.Insert uses CopyFrom, which cannot express on conflict, so
the insert violates the destination's unique index and rolls back, and every
subsequent poll repeats it. The integration stops converging until the rows
are deleted by hand; sibling integrations on the same source keep running,
and restarting does not help.

Delete now deletes destination data from one past the highest remaining
update, so no block is left without an update covering it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A converge loop can stop making progress without ever logging an error --
its goroutine parked or exited -- and new-task only happens at process
start, so nothing in-process can revive it. Observed in production
2026-07-15 and again 2026-08-10 (two sibling integrations frozen together
for hours while every other task on the same source kept converging).

-stall-exit <duration> (default 0, disabled) starts a watcher that polls
shovel.task_updates once a minute. A task whose watermark has not advanced
for the given duration, while at least one sibling task on the same source
HAS advanced since the freeze, is declared stalled: the process logs
task-stalled, dumps all goroutine stacks to stderr (the frozen goroutine's
stack is the diagnosis), and exits 70 so the supervisor (docker restart
policy, systemd) replaces it -- the same recovery an operator performs by
hand today, applied in minutes instead of hours and with evidence captured.

Source-wide freezes (an RPC outage) are deliberately not stalls: a restart
would not help and would loop. Tasks that are disabled, gate on
dependencies, or have a configured stop are exempt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat: -stall-exit self-heals silently frozen tasks
The hardcoded 5s cancels legitimate bulk inserts: shovel's startup statement and
busy-box converge passes routinely exceed it. The startup form is fatal - the
process dies on a bare 'ERROR: canceling statement due to statement timeout'
before the structured logger owns the error, and the restart policy loops it
(~5s cadence) until one start gets through. Because an explicit RuntimeParams
entry beats PGOPTIONS, the deployment could not override this from the outside
(observed live 2026-08-24: container env had PGOPTIONS statement_timeout=60000
and still crash-looped at the 5s cadence). 60s still bounds a genuinely hung
statement; stall detection remains the backstop for wedged tasks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(wpg): raise pinned statement_timeout 5s -> 60s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant