Skip to content

test(zebrad): cap the live-sync tests at 2 minutes per attempt - #11402

Draft
upbqdn wants to merge 1 commit into
mainfrom
11399-sync-test-timeout
Draft

test(zebrad): cap the live-sync tests at 2 minutes per attempt#11402
upbqdn wants to merge 1 commit into
mainfrom
11399-sync-test-timeout

Conversation

@upbqdn

@upbqdn upbqdn commented Sep 4, 2026

Copy link
Copy Markdown
Member

Motivation

Closes #11399.

Solution

Lower the per-attempt slow-timeout for integration::sync::sync_one_checkpoint_mainnet
and integration::sync::restart_stop_at_height from 10 minutes to 2, keeping their two
retries, and record in .config/nextest.toml and on sync_until why nextest has to be
the bound: with MempoolBehavior::ShouldNotActivate, sync_until waits for zebrad to
exit with wait_with_output, which TestChild::with_timeout does not bound.

Tests

These two tests sync from live peers, so a slow peer stalls them with no limit of their
own. A run on the changed config reproduced a stall and shows the new bound working: the
attempt was terminated at 120s and the retry passed in 7.6s, for 138s total where the old
600s cap would have spent about 610s.

  TRY 1 SLOW [> 60.000s] zebrad::zebrad-tests integration::sync::sync_one_checkpoint_mainnet
TRY 1 TRMNTG [>120.000s] zebrad::zebrad-tests integration::sync::sync_one_checkpoint_mainnet
   TRY 1 TMT [ 120.003s] zebrad::zebrad-tests integration::sync::sync_one_checkpoint_mainnet
  TRY 2 PASS [   7.598s] (2/2) zebrad::zebrad-tests integration::sync::sync_one_checkpoint_mainnet
     Summary [ 138.188s] 2 tests run: 2 passed (1 flaky), 1334 skipped

Both tests pass unstalled in 7.6-13.7s, so 2 minutes leaves roughly nine times the
runtime they need. Making the wait itself bounded, so a stall fails where it happens
instead of relying on the test runner, is left to #11399's follow-up discussion.

AI Disclosure

  • No AI tools were used in this PR
  • AI tools were used: Claude, to measure the stall and write the change.

PR Checklist

  • The PR title follows conventional commits format: type(scope): description
  • The PR follows the contribution guidelines.
  • This change was discussed in an issue or with the team beforehand.
  • The solution is tested.
  • The documentation and changelogs are up to date.

`sync_one_checkpoint_mainnet` and `restart_stop_at_height` normally take
10-30s, but they sync from live peers, and a slow peer stalls them with no
bound: they use `MempoolBehavior::ShouldNotActivate`, so `sync_until`
collects the whole output with `wait_with_output`, which
`TestChild::with_timeout` does not bound. Nextest's `slow-timeout` was the
only limit, and at `period = 5m, terminate-after = 2` a stalled attempt
held the serial live-mainnet group for 10 minutes before the retry passed
in seconds, doubling the unit-test job while still reporting success.

Cap an attempt at 2 minutes, which is about nine times the runtime these
tests need, and record why nextest has to be the limit here.
@upbqdn upbqdn self-assigned this Sep 4, 2026
@v12-auditor

v12-auditor Bot commented Sep 4, 2026

Copy link
Copy Markdown

Warning

No auditable source files found in this PR's diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(zebrad): restart_stop_at_height stalls on live peers and costs 10 minutes before its retry passes

1 participant