Skip to content

test(consistent): avoid collecting guaranteed backend skips#5425

Open
njzjz-bot wants to merge 1 commit intodeepmodeling:masterfrom
njzjz-bothub:fix/issue-5371-avoid-consistency-skip
Open

test(consistent): avoid collecting guaranteed backend skips#5425
njzjz-bot wants to merge 1 commit intodeepmodeling:masterfrom
njzjz-bothub:fix/issue-5371-avoid-consistency-skip

Conversation

@njzjz-bot
Copy link
Copy Markdown
Contributor

@njzjz-bot njzjz-bot commented Apr 27, 2026

Summary

  • prune inherited backend consistency tests from collection when the generated parameterized class already knows they are unsupported
  • keep the change centralized in source/tests/consistent/common.py instead of editing many individual suites
  • reduce CI skip noise and collection overhead without changing intended backend coverage

Testing

  • python3 -m py_compile source/tests/consistent/common.py

Closes #5371


Authored by OpenClaw (model: gpt-5.4)

Summary by CodeRabbit

Release Notes

  • Tests

    • Improved test collection efficiency by automatically excluding disabled test methods at subclass initialization, preventing unnecessary test runner overhead across different backend configurations.
  • Documentation

    • Updated documentation for test configuration attributes.

Avoid collecting inherited backend consistency tests when the generated
parameterized class already knows that a backend is unsupported for that
parameter combination.

This keeps pytest/unittest from collecting large batches of tests that
would only call skipTest() at runtime, reducing skip noise and collection
overhead in heavy consistency CI jobs.

Authored by OpenClaw (model: gpt-5.4)
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 27, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 3c14ed35-fff6-4d31-9f61-0f2cfe4dadf4

📥 Commits

Reviewing files that changed from the base of the PR and between 9d63816 and bfc5727.

📒 Files selected for processing (1)
  • source/tests/consistent/common.py

📝 Walkthrough

Walkthrough

Adds subclass-time pruning of inherited test methods guaranteed to skip by implementing __init_subclass__ hook that identifies disabled test methods via backend skip flags, CI conditions, and missing dependencies, then shadows those methods to prevent test collection.

Changes

Cohort / File(s) Summary
Test Pruning Mechanism
source/tests/consistent/common.py
Added __init_subclass__ hook, _prune_disabled_test_methods() classmethod to shadow disabled tests by setting them to None, and _disabled_test_methods() classmethod to resolve backend skip flags (skip_tf/skip_dp/skip_pt/skip_pt_expt/skip_jax/skip_pd/skip_array_api_strict), handle missing pt_expt_class, and extend disabled set based on CI + non-CPU execution context. Updated array_api_strict_class docstring.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: implementing collection-time pruning of guaranteed backend skips in consistency tests.
Linked Issues check ✅ Passed The PR implements collection-time filtering to prevent guaranteed backend skips from being collected, directly addressing issue #5371's primary objective of reducing collection overhead and skip noise.
Out of Scope Changes check ✅ Passed All changes are scoped to source/tests/consistent/common.py with focused additions for pruning disabled test methods and a docstring update, all directly supporting the stated objective.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.39%. Comparing base (d42732e) to head (bfc5727).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5425      +/-   ##
==========================================
+ Coverage   80.35%   82.39%   +2.03%     
==========================================
  Files         819      824       +5     
  Lines       85445    87395    +1950     
  Branches     4139     4198      +59     
==========================================
+ Hits        68661    72008    +3347     
+ Misses      15508    14112    -1396     
+ Partials     1276     1275       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests: avoid collecting consistency tests that are guaranteed to skip in CI

1 participant