Skip to content

Fix bug in F3 stop and use start/stop in catchup tests#1029

Merged
masih merged 1 commit into
mainfrom
masih/test-flaky-pause-resume
Jun 27, 2025
Merged

Fix bug in F3 stop and use start/stop in catchup tests#1029
masih merged 1 commit into
mainfrom
masih/test-flaky-pause-resume

Conversation

@masih

@masih masih commented Jun 24, 2025

Copy link
Copy Markdown
Member

Fix a bug where F3.Stop was never calling stopInternal which actually resets the state and stops the internal components.

Remove the testing API exposed for pause resume in favor of stop start. Because, now that manifest is final, there's no pause/resume and the scenario that was being tested is irrelevant. Instead, stop the nodes completely then start them back up to assert that they do catch up.

The flaky test was repeated for 10 minutes on CI without failing.

Fixes: #1025

@github-project-automation github-project-automation Bot moved this to Todo in F3 Jun 24, 2025
@masih masih self-assigned this Jun 24, 2025
@codecov

codecov Bot commented Jun 24, 2025

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.27%. Comparing base (20d68df) to head (bcc04f9).
Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1029      +/-   ##
==========================================
- Coverage   65.58%   65.27%   -0.31%     
==========================================
  Files          80       80              
  Lines        9703     9703              
==========================================
- Hits         6364     6334      -30     
- Misses       2847     2868      +21     
- Partials      492      501       +9     
Files with missing lines Coverage Δ
f3.go 53.84% <100.00%> (ø)

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@masih masih force-pushed the masih/test-flaky-pause-resume branch 2 times, most recently from 150702e to 865ae86 Compare June 25, 2025 13:59
@masih masih changed the title Investigate TestF3PauseResumeCatchup being flaky Await cert chain to catch up before asserting resumption in tests Jun 25, 2025
@masih masih marked this pull request as ready for review June 25, 2025 14:22
@masih masih requested review from Kubuxu and Copilot June 25, 2025 14:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces an additional wait period in the tests to ensure nodes have caught up with the certification chain before proceeding, reducing flakage on CI.

  • Added a require.Eventually check to verify that node 1 catches up with the certificate chain before proceeding.
  • Added a similar check for node 2, although there appears to be an issue with fetching the correct node’s certificate.

Comment thread f3_test.go Outdated
@masih masih force-pushed the masih/test-flaky-pause-resume branch 2 times, most recently from b30b333 to 1b6f9d5 Compare June 25, 2025 14:35
@masih masih moved this from Todo to In review in F3 Jun 25, 2025
@masih masih removed the request for review from Kubuxu June 25, 2025 14:54
@masih masih marked this pull request as draft June 25, 2025 14:54
@masih masih moved this from In review to In progress in F3 Jun 25, 2025
Comment thread .github/workflows/flake-da-test.yaml Fixed
Fix a bug where F3.Stop was never calling `stopInternal` which actually
resets the state and stops the internal components.

Remove the testing API exposed for pause resume in favor of stop start.
Because, now that manifest is final, there's no pause/resume and the
scenario that was being tested is irrelevant. Instead, stop the nodes
completely then start them back up to assert that they do catch up.

The flaky test was repeated for 10 minutes on CI without failing.

Fixes: #1025
@masih masih force-pushed the masih/test-flaky-pause-resume branch from 2afa056 to bcc04f9 Compare June 27, 2025 09:30
@masih masih changed the title Await cert chain to catch up before asserting resumption in tests Fix bug in F3 stop and use start/stop in catchup tests Jun 27, 2025
@masih masih marked this pull request as ready for review June 27, 2025 10:03
@masih masih requested a review from Kubuxu June 27, 2025 10:03
@github-project-automation github-project-automation Bot moved this from In progress to In review in F3 Jun 27, 2025
@masih masih added this pull request to the merge queue Jun 27, 2025
Merged via the queue into main with commit b0d9ca4 Jun 27, 2025
13 checks passed
@masih masih deleted the masih/test-flaky-pause-resume branch June 27, 2025 12:33
@github-project-automation github-project-automation Bot moved this from In review to Done in F3 Jun 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

TestF3PauseResumeCatchup flaked with failing to look up power table

4 participants