Skip to content

[infra] Run sanity checks in the parallel fixture manager#22311

Open
lolyu wants to merge 3 commits into
sonic-net:masterfrom
lolyu:pfm_sanity_check
Open

[infra] Run sanity checks in the parallel fixture manager#22311
lolyu wants to merge 3 commits into
sonic-net:masterfrom
lolyu:pfm_sanity_check

Conversation

@lolyu

@lolyu lolyu commented Feb 10, 2026

Copy link
Copy Markdown
Collaborator

Description of PR

Summary:
Fixes # (issue)

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • New Test case
    • Skipped for non-supported platforms
  • Test case improvement

Back port request

  • 202205
  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511

Approach

What is the motivation for this PR?

Run the sanity checks in the parallel fixture manager to reduce runtime.
This depends on #21949.

Signed-off-by: Longxiang Lyu lolv@microsoft.com

How did you do it?

As the motivation.

How did you verify/test it?

Run on dualtor-aa testbed:
Before:

         0:01:41.311315 sanity_check                                         1          0:01:41.311315          0:01:41.311315
         0:01:10.551272 sanity_check_full                                    1          0:01:10.551272          0:01:10.551272
  • after:
         0:01:07.309506 sanity_check                                         1          0:01:07.309506          0:01:07.309506
         0:00:35.659110 sanity_check_full                                    1          0:00:35.659110          0:00:35.659110

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu lolyu force-pushed the pfm_sanity_check branch from 9ed06e0 to f35cca3 Compare March 26, 2026 10:53
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu lolyu force-pushed the pfm_sanity_check branch from f35cca3 to 4ea6e58 Compare March 26, 2026 10:57
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@lolyu lolyu requested a review from yxieca March 26, 2026 11:08
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu lolyu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📊 Review Summary — [infra] Run sanity checks in the parallel fixture manager

Nice perf improvement — sanity_check_full cut from ~1m10s to ~35s on dualtor-aa. Clean migration from SafeThreadPoolExecutor to ParallelCoordinator.

✅ Strengths

  • Clear before/after timing data
  • Clean removal of SafeThreadPoolExecutor import
  • Backward-compatible parallel_manager=None default in recover_on_sanity_check_failure

Findings

  • ⚠️ parallel_manager=None not guarded in do_checks — could crash if called without it
  • ⚠️ Variable shadowing: duthost becomes a string in print_logs loop
  • 📝 check_fixture.__name__ mutation could leak as side effect
  • 💡 future.result() exceptions not caught — could propagate unhandled

See inline comments for details and suggested fixes.

⚠️ Approved with suggestions

Comment thread tests/common/plugins/sanity_check/__init__.py
Comment thread tests/common/plugins/sanity_check/__init__.py Outdated
Comment thread tests/common/plugins/sanity_check/__init__.py
Comment thread tests/common/plugins/sanity_check/__init__.py Outdated
@lolyu lolyu force-pushed the pfm_sanity_check branch from 806bf02 to 949ff2e Compare April 3, 2026 07:43
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu

lolyu commented Apr 6, 2026

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu lolyu force-pushed the pfm_sanity_check branch from 949ff2e to f83d41a Compare May 8, 2026 06:29
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

lolyu added 2 commits June 4, 2026 11:41
Signed-off-by: Longxiang Lyu <lolv@microsoft.com>
Signed-off-by: Longxiang Lyu <lolv@microsoft.com>
@lolyu lolyu force-pushed the pfm_sanity_check branch from f83d41a to c433eec Compare June 4, 2026 11:42
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu

lolyu commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu lolyu force-pushed the pfm_sanity_check branch from 18f3293 to e4fe9d5 Compare June 10, 2026 07:20
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@lolyu

lolyu commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Longxiang Lyu <lolv@microsoft.com>
@lolyu lolyu force-pushed the pfm_sanity_check branch from 3b5361a to 46ef9f7 Compare June 12, 2026 04:55
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants