fix: delete data down to the remaining watermark on reorg - #294
Open
cdsiren wants to merge 5 commits into
Open
Conversation
task_updates gets one row per converge batch, with num set to the batch's last block, so one update covers (previous update, num]. Task.Delete removed updates at num >= n but only deleted destination data at block_num >= n, leaving the batch's canonical prefix -- blocks below n that no remaining update covers. Converge resumes from the highest remaining update and requests those blocks again. Integration.Insert uses CopyFrom, which cannot express on conflict, so the insert violates the destination's unique index and rolls back, and every subsequent poll repeats it. The integration stops converging until the rows are deleted by hand; sibling integrations on the same source keep running, and restarting does not help. Delete now deletes destination data from one past the highest remaining update, so no block is left without an update covering it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A converge loop can stop making progress without ever logging an error -- its goroutine parked or exited -- and new-task only happens at process start, so nothing in-process can revive it. Observed in production 2026-07-15 and again 2026-08-10 (two sibling integrations frozen together for hours while every other task on the same source kept converging). -stall-exit <duration> (default 0, disabled) starts a watcher that polls shovel.task_updates once a minute. A task whose watermark has not advanced for the given duration, while at least one sibling task on the same source HAS advanced since the freeze, is declared stalled: the process logs task-stalled, dumps all goroutine stacks to stderr (the frozen goroutine's stack is the diagnosis), and exits 70 so the supervisor (docker restart policy, systemd) replaces it -- the same recovery an operator performs by hand today, applied in minutes instead of hours and with evidence captured. Source-wide freezes (an RPC outage) are deliberately not stalls: a restart would not help and would loop. Tasks that are disabled, gate on dependencies, or have a configured stop are exempt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat: -stall-exit self-heals silently frozen tasks
The hardcoded 5s cancels legitimate bulk inserts: shovel's startup statement and busy-box converge passes routinely exceed it. The startup form is fatal - the process dies on a bare 'ERROR: canceling statement due to statement timeout' before the structured logger owns the error, and the restart policy loops it (~5s cadence) until one start gets through. Because an explicit RuntimeParams entry beats PGOPTIONS, the deployment could not override this from the outside (observed live 2026-08-24: container env had PGOPTIONS statement_timeout=60000 and still crash-looped at the 5s cadence). 60s still bounds a genuinely hung statement; stall detection remains the backstop for wedged tasks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(wpg): raise pinned statement_timeout 5s -> 60s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: delete data down to the remaining watermark on reorg
Problem
A reorg can permanently wedge a single integration. It stops converging and logs this
forever, ~26 retries/sec, until someone deletes rows by hand:
Sibling integrations on the same source keep converging, so it reads as a
single-integration failure rather than a reorg. Restarting does not help, because the
conflicting rows are already committed.
Cause
Task.updatewrites oneshovel.task_updatesrow per converge batch, withnumsetto the batch's last block (
task.go:443). One update therefore covers the whole range(previous update, num].Task.Delete(n)removes updates atnum >= nbut tells the destination to delete atblock_num >= n. When the batch spans more than one block, the blocks belownin thatsame batch keep their data while the watermark rewinds below them — they are the batch's
canonical prefix, not orphaned-chain data, since only
nand above were reorged out.Convergethen resumes from the highest remaining update and re-requests those blocks.dig.Integration.InsertusesCopyFrom, which cannot expresson conflict, so the inserthits
u_<table>and rolls back. Every subsequent poll repeats it.The window is
batchSize > 1, which is the normal case: with a 2-block batch it is roughlya coin flip per reorg. Concretely, from a production instance on mainnet:
The update at 25666518 was deleted along with data
>= 25666518; blocks 25666516–17 fromthat same batch survived below the rewound watermark. Deleting exactly those two rows
unwedged the integration with no restart, and Shovel re-inserted them byte-identically
(md5 of every re-inserted row matched the pre-delete capture), which confirms they were
canonical rather than reorged data.
Fix
Task.Deletenow asks whereConvergewill actually resume — one past the highestremaining update — and deletes destination data from there, so no block is ever left
without an update covering it. The new
resumevalue is never greater thann(no updateat or above
nsurvives the delete), so this only ever widens the delete, and only withina batch that was already being discarded.
With no updates left, the task restarts at its configured start, or at the chain head when
there is none; both are above anything already stored, so that case falls back to
nrather than deleting the destination's whole history.
Re-indexing a batch's earlier blocks is the same work the task was already about to do —
it re-requests that range from the node either way.
Tests
TestConverge_ReorgBatchPrefix(new): one update covering blocks 1–2, block 2 reorgedout, asserts the task converges to the canonical chain. Without the
task.gochange itfails with
inserting data: inserting blocks: duplicate key: 1— the same shape as theproduction error above.
testDestinationwas too permissive to catch this and needed two fixes to matchdig.Integration:Deletenow deletes at>= nrather than the single keyn, andInsertreports a duplicate instead of silently overwriting, since the real destinationcopies into a table with a unique index and no
on conflict. The existingTestConverge_Reorgpasses under both, because it writes one update per block, sobatch-granularity never comes into play.
go test -p 1 ./..., postgres 15 via pqxtest).Alternatives considered
task_updatesrow per block. Removes the granularity mismatch at its root, butmultiplies watermark writes by the batch size.
on conflict do nothingon insert. Not expressible throughCopyFrom, and it wouldmask genuine duplicates.