Skip to content

blockchain+netsync: follow-up fixes for headers-first IBD review - #2508

Open
Roasbeef wants to merge 4 commits into
masterfrom
fix-pr-2428-review-items
Open

blockchain+netsync: follow-up fixes for headers-first IBD review#2508
Roasbeef wants to merge 4 commits into
masterfrom
fix-pr-2428-review-items

Conversation

@Roasbeef

Copy link
Copy Markdown
Member

NOTE: This PR is stacked on top of #2428 and should be reviewed/merged
after that PR lands. The diff shown here includes #2428's changes; the new
commits are the last four on this branch.

In this PR, we address a set of issues found during code review of #2428
(the headers-first IBD rework). The changes here are correctness fixes,
thread safety improvements, and performance/cleanup items that sit on top
of the new IBD pipeline. See each commit message for a detailed description
w.r.t the incremental changes.

Correctness Fixes

The new fetchHigherPeers helper dropped the segwit peer filtering that
the old startSync had. Post-segwit activation, this meant btcd could
pick a non-witness peer for sync and fail to fully validate the chain. We
restore the IsDeploymentActive + IsWitnessEnabled check inside
fetchHigherPeers (and the new fetchEqualPeers) so witness-only sync
is enforced whenever segwit is active.

The old startSync also had an equalPeers fallback for peers at the
same height, which is critical for regtest where both nodes start at
genesis (height 0) and neither is strictly higher. We add
fetchEqualPeers and wire it into the block-download path of startSync
to restore this behavior.

We also bring back syncCandidate demotion: peers whose last advertised
block has fallen below our height get syncCandidate = false so they
aren't repeatedly considered in future sync rounds. The demotion uses <
(not <=) to preserve the existing behavior where equal-height peers
remain candidates.

Thread Safety

IsValidHeader, HeaderHashByHeight, and HeaderHeightByHash all read
from bestHeader without holding chainLock, while
maybeAcceptBlockHeader modifies bestHeader under chainLock.Lock().
We acquire chainLock.RLock() in each accessor. The redundant Contains
check in HeaderHashByHeight is also removed since NodeByHeight already
guarantees membership in the chain view.

Performance

buildBlockRequest previously iterated from forkHeight+1 to
bestHeaderHeight on every refill call, i.e. O(n) per invocation. We
introduce a lastBlockRequested cursor that tracks the highest scanned
height, so subsequent calls skip already-requested ranges. The cursor
resets on peer disconnect and IBD completion.

In maybeAcceptBlockHeader, known non-invalid side-chain headers were
falling through to re-run CheckBlockHeaderSanity and
CheckBlockHeaderContext. Since these headers passed validation when first
added, the re-validation is pure overhead. We return early for known
side-chain headers.

checkHeadersList was calling IsValidHeader then HeaderHeightByHash,
each doing their own LookupNode. We add ValidHeaderHeight that
combines both into a single index lookup.

Cleanup

The unused headerNode type (left over from the old headerList-based
sync) is removed. The duplicate nil guard in buildBlockRequest is
removed since the only caller (fetchHeaderBlocks) already checks. A
BIP130 design note is added to handleHeadersMsg explaining why
unsolicited headers are now accepted (peers can proactively push headers
via sendheaders). The crash-recovery behavior of flushToDB's
header-only skip is documented: a restart during header sync loses header
progress, which is an acceptable trade-off given the small size and fast
re-validation of headers.

@coveralls

coveralls commented Mar 24, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 29626832520

Coverage increased (+0.05%) to 52.272%

Details

  • Coverage increased (+0.05%) from the base build.
  • Patch coverage: 31 uncovered changes across 3 files (113 of 144 lines covered, 78.47%).
  • 28 coverage regressions across 4 files.

Uncovered Changes

File Changed Covered %
blockchain/chain.go 37 17 45.95%
netsync/manager.go 88 81 92.05%
blockchain/accept.go 19 15 78.95%

Coverage Regressions

28 previously-covered lines in 4 files lost coverage.

File Lines Losing Coverage Coverage
peer/peer.go 15 72.27%
blockchain/chain.go 8 74.87%
rpcclient/infrastructure.go 3 67.01%
connmgr/connmanager.go 2 84.59%

Coverage Stats

Coverage Status
Relevant Lines: 69792
Covered Lines: 36482
Line Coverage: 52.27%
Coverage Strength: 351387.73 hits per line

💛 - Coveralls

Roasbeef added 4 commits July 17, 2026 21:18
In maybeAcceptBlockHeader, when a header already exists in the block
index and is not invalid, the function previously fell through to
re-run CheckBlockHeaderSanity and CheckBlockHeaderContext for
side-chain headers.  Since these headers were already validated when
first added, this re-validation is pure overhead.

Return early with (false, nil) for known non-invalid side-chain
headers, and simplify the subsequent node creation path which is now
guaranteed to only execute for genuinely new headers.

Also documents the crash-recovery behavior of flushToDB's header-only
skip optimization and fixes a typo in process_test.go.
IsValidHeader, HeaderHashByHeight, and HeaderHeightByHash all read
from bestHeader without holding chainLock, while
maybeAcceptBlockHeader modifies bestHeader under chainLock.Lock().
This creates a potential data race on concurrent access.

Acquire chainLock.RLock() in each accessor before reading
bestHeader.  Also removes the redundant Contains check from
HeaderHashByHeight since NodeByHeight already returns from the
chain view's internal array, guaranteeing membership.

Introduces ValidHeaderHeight as a single-lookup alternative to
calling IsValidHeader + HeaderHeightByHash separately.  This
avoids two redundant LookupNode calls in netsync's
checkHeadersList hot path during IBD.
Several behavioral regressions were introduced when the old
checkpoint-based headers-first mode was replaced with the new IBD
pipeline.  This commit addresses them along with performance and
code quality improvements.

Restore segwit peer filtering in fetchHigherPeers so that post-
activation, only witness-enabled peers are selected for sync.
Without this, btcd could download blocks from a non-witness peer
and fail to fully validate the chain.

Restore syncCandidate demotion for peers whose last advertised block
has fallen behind our height, preventing them from being repeatedly
considered in future sync rounds.

Add fetchEqualPeers and wire it into startSync's block-download
fallback path.  This restores the equalPeers behavior needed for
regtest where both nodes start at height 0 and neither is strictly
higher than the other.

Introduce lastBlockRequested as a cursor in buildBlockRequest to
skip already-scanned height ranges.  The old code re-iterated from
forkHeight+1 on every refill call, which is O(n) per invocation
during IBD with large header chains.  The cursor is reset on peer
disconnect and IBD completion.

Remove the duplicate nil check in buildBlockRequest (the only caller,
fetchHeaderBlocks, already guards against nil).  Remove the unused
headerNode type left over from the old headerList-based sync.
Consolidate checkHeadersList to use the new ValidHeaderHeight for a
single index lookup instead of two.  Add a BIP130 design note to
handleHeadersMsg explaining why unsolicited headers are accepted.
TestFetchHigherPeersDemotesStalePeers verifies that fetchHigherPeers
sets syncCandidate to false for peers whose last block is strictly
below the given height, preventing them from being repeatedly
considered in subsequent sync rounds.

TestStartSyncEqualPeersFallback verifies that when no peer is
strictly higher than our block height but peers at the same height
exist, startSync falls back to selecting one of those equal-height
peers for block download.  This is critical for regtest where both
nodes start at genesis.
@Roasbeef
Roasbeef force-pushed the fix-pr-2428-review-items branch from 421597e to 8310c1e Compare July 18, 2026 02:19
@Roasbeef

Copy link
Copy Markdown
Member Author

Pushed up a new version. First, we've rebased on top of master to pick up the v2 module migration and the recent netsync changes. The rebase surfaced a semantic conflict: on the v2 chaincfg, regtest/simnet mark segwit as always active, so the witness peer filter in fetchHigherPeers/fetchEqualPeers started rejecting every non-witness peer on regtest, which broke local sync (and a pile of tests). We now skip the witness requirement on local test networks via a new requireWitnessPeers helper, mirroring the existing carve out in isSyncCandidate.

Beyond the rebase, this version fixes a few issues found in a correctness review pass, folded into the relevant commits.

The block request cursor (lastBlockRequested) is now invalidated when the best header chain reorgs beneath it. The cursor is only meaningful for the header chain it was built against: we now remember the header hash at the cursor height, and if that hash no longer matches on the next refill, we reset and re-scan from the fork point. Without this, blocks on the new branch below the cursor would never be requested, and IBD could stall until the stall handler fired ~3 minutes later. A notfound for a requested block also rewinds the cursor now, so the block gets re-requested on the next refill rather than waiting out the stall timer. New coverage in TestBuildBlockRequestReorgResetsCursor and TestNotFoundResetsBlockRequestCursor.

Peer demotion has been narrowed to the best block height. Previously the demotion lived inside fetchHigherPeers, which startSync also calls with the best header height. During IBD the header chain races ahead of every peer's advertised block, so any peer even one block behind our header tip would get permanently demoted, eventually draining the whole sync candidate pool. Demotion now lives in a dedicated demoteStalePeers that startSync invokes against the block height only, and fetchHigherPeers is a pure filter again.

maybeAcceptBlockHeader still skips re-validation for known side-chain headers, but no longer returns early: it falls through to the tip promotion logic instead. This matters after a restart, since initChainState resets the best header chain to the best block tip, so a previously known heavier side chain must be able to win back the best header tip when re-announced (otherwise we'd loop on getheaders without making progress until the remote chain grew a new header). TestProcessBlockHeaderRePromoteSideChain covers the restart scenario.

Lastly, TestStartSyncEqualPeersFallback now actually exercises the equal-height fallback: the old version had the peer at height 5 with our block height at 0, so the peer was selected via the ordinary higher-peers path and the assertion passed by accident. The test now pins everything at genesis, matching the two-fresh-regtest-nodes case the fallback exists for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants