node: stop transport errors from poisoning blocks_by_range peers (+ TTL)

[ch4r10t33r]

Problem

Recurring catch-up wedge on devnet (zeam_4: 8995 slots behind, every peer logging cannot serve blocks_by_range and gap=… exceeds the blocks_by_root walk limit (64)). zeam_6 never wedged.

Root cause

The devnet blocks-by-range request … failed lines show the poisoning trigger:

failed (1): IoError
failed (1): Disconnected
failed (1): peer disconnected

These are transport-level failures (mid-stream QUIC teardown / IO error) that surface as RPC error code 1. isBlocksByRangeUnavailable short-circuited on bare code == RPC_ERR_INVALID_REQUEST → misclassified them as "peer doesn't support blocks_by_range" → set the sticky blocks_by_range_unavailable flag.

Self-reinforcing spiral: a node that fell behind issued large blocks_by_range requests → big transfers hit stream teardowns → each poisoned a range-capable peer → eventually no range peers left → gap grew past MAX_BLOCKS_BY_ROOT_CATCHUP_GAP (64) → wedge. Nodes that stayed synced never issued large requests, so they never entered the spiral — that's the zeam_4 vs zeam_6 difference.

Compounding it: the flag was only reset on reconnect (connect() rebuilds PeerInfo). A peer that stayed connected after being wrongly marked was excluded from range sync for the life of the connection — the exact failure markBlocksByRangeUnavailable's own comment warned about ("add a reset/TTL when you do") but never guarded.

Fix

1. Classifier (isBlocksByRangeUnavailable): transport messages (disconnect, ioerror, timeout, reset, closed, cancel, eof, …) are transient and never mark a peer unavailable. Only an explicit "unsupported protocol" reply, or a bare code-1 with a non-transport message, counts.
2. TTL (BLOCKS_BY_RANGE_UNAVAILABLE_TTL_S = 300): the mark is now a timestamp and expires after 5 min even without a reconnect, so any misclassification self-heals in minutes instead of persisting for the connection lifetime.

Tests

• isBlocksByRangeUnavailable does NOT flag transport failures (IoError / Disconnected / peer disconnected / reset / timeout / closed).
• Existing wedge regression extended: mark expires after TTL without reconnect.
• Full zig build test green.

Deploy note

Deploying requires a rebuild + container restart; a restarted zeam_4 starts with fresh (unpoisoned) peer state, and with this fix transport blips no longer re-poison, so it can range-sync out of the 8995-slot gap.

[zclawz]

Approved from my side. I did an adversarial pass over #1050 and found no blocking issues.

What I checked:

• Transport-vs-protocol classification: IoError, Disconnected, peer disconnects, stream resets, timeouts, closed/cancel/eof/broken-pipe messages now stay transient and do not poison the peer as range-incapable.
• Protocol fallback behavior: explicit unsupported/not-available/not-supported/unknown-protocol replies still mark the peer unavailable, and the existing bare RPC_ERR_INVALID_REQUEST fallback is preserved for non-transport code-1 messages.
• TTL semantics: blocks_by_range_unavailable_at is honored only inside the 300s TTL in both direct support checks and range-peer selection, so a misclassification can self-heal without reconnect while genuine unsupported peers are not hammered continuously.
• State migration/call sites: the old bool field is replaced consistently on PeerInfo; node-side marking still only happens on the gated RPC response path, not on local send/transport failures.

Validation I ran locally:

• zig fmt --check pkgs/node/src/blocks_by_range_sync.zig pkgs/node/src/locking.zig pkgs/node/src/network.zig
• git diff --check origin/main...HEAD
• /tmp/zig-0.16.0/zig test pkgs/node/src/blocks_by_range_sync.zig --test-filter isBlocksByRangeUnavailable

I also started zig build test --summary all with the repo Zig 0.16.0 and corrected Rust PATH. It got past the earlier local rustup PATH issue and was compiling Rust glue when I stopped it to avoid holding the Telegram session open on the full local gate. GitHub CI is still the source of truth for the full suite; at the time of review the PR CI run was still in progress.