[ci]: add transient pytest reruns for Modal tests

[macthecadillac]

Summary

• add a shared Modal pytest retry policy using pytest-rerunfailures
• apply transient-only reruns to standard Modal pytest lanes through PYTEST_ADDOPTS
• apply the same rerun arguments to SSIM subprocess pytest commands
• document the CI retry behavior and add static contract coverage

Fixes #1571.

Validation

• Modal L40S focused contract validation:
  • uv run --no-project --with modal python -m modal run /tmp/fastvideo-worktrees/interleavethinker-launcher/fastvideo/tests/modal/launch_l40s_job.py --install-extra none --no-build-kernel --apply-local-patch --patch-paths "pyproject.toml,fastvideo/tests/modal/pr_test.py,fastvideo/tests/modal/ssim_test.py,fastvideo/tests/modal/pytest_retry.py,fastvideo/tests/contract/test_ci_pytest_reruns.py,docs/contributing/ci_architecture.md,docs/contributing/testing.md" --command "source $HOME/.local/bin/env 2>/dev/null || true; source /opt/venv/bin/activate 2>/dev/null || true; uv pip install pytest && pytest fastvideo/tests/contract/test_ci_test_collection.py fastvideo/tests/contract/test_ci_pytest_reruns.py -q"
  • Result: 6 passed in 0.04s
• uv run --no-project --with pre-commit pre-commit run --all-files
  • Result: passed from /tmp/fastvideo_worktrees/issue_1571_flaky_gpu_test_retry
• Bot review follow-up validation after 603c7b0d1:
  • Accepted Gemini feedback for shell exit-code preservation, absolute Modal helper paths, UTF-8 test file reads, and quote-style-neutral dependency assertion.
  • uv run --no-project --with pre-commit pre-commit run --all-files
    • Result: passed.
  • Modal L40S focused contract validation:
    • Result: 6 passed in 0.04s
    • Run: https://modal.com/apps/hao-ai-lab/main/ap-UsHkcTrO6403uBVZQkjIqG
• End-to-end Modal CI lane:
  • env BUILDKITE_REPO=https://github.com/hao-ai-lab/FastVideo.git BUILDKITE_COMMIT=603c7b0d1c662f3d17df749ccec332568297a839 BUILDKITE_PULL_REQUEST=1581 uv run --no-project --with modal python -m modal run fastvideo/tests/modal/pr_test.py::run_unit_test
  • Result: 601 passed, 20 warnings in 13.56s
  • Run: https://modal.com/apps/hao-ai-lab/main/ap-CJI2s0Cm6IIyZhS0oYUya7
• Full Buildkite suite:
  • Triggered via /test full.
  • Buildkite: https://buildkite.com/fastvideo/ci/builds/4285

Review Loop

• review-code sub-agent reviewed macthecadillac/FastVideo branch issue/1571-flaky-gpu-test-retry for issue [ci] [Feature] Flaky GPU Test Retry #1571 and reported no actionable findings.
• No adjudicator/fixer sub-agent was spawned because there were no actionable findings to adjudicate.

GPU Memory Impact

No direct GPU memory impact. The change can extend wall time for transient infrastructure failures by rerunning matching individual pytest failures, but it avoids broad Buildkite lane retries for deterministic failures.

Checklist

• I ran pre-commit run --all-files and fixed all issues
• I added or updated tests for my changes
• I updated documentation if needed
• I considered GPU memory impact of my changes

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=full-suite-passed
• check-success=fastcheck-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[gemini-code-assist]

Code Review

This pull request introduces a shared transient-only pytest rerun policy using pytest-rerunfailures to automatically retry infrastructure-related failures in Modal CI lanes, complete with documentation updates and static contract tests. The reviewer's feedback suggests separating the export statement from command assignment in pr_test.py to prevent masking exit codes, using absolute paths for robustness, and specifying encoding="utf-8" along with relaxing quote checks when reading files in the contract tests.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[macthecadillac]

/test full

[SolitaryThinker]

Rebased onto current main to pick up #1565’s CUDA cleanup and #1561’s FA4-off train-lane policy; no PR content changes (product range-diff clean).

[SolitaryThinker]

/test dreamverse

[SolitaryThinker]

/test encoder

[SolitaryThinker]

/test dreamverse

[Satyam-53]

/test encoder

[SolitaryThinker]

The exact-head CI is now green, but the retry policy still has three blocking auditability/composition issues:

1. fastvideo/tests/modal/pytest_retry.py needs --rerun-show-tracebacks. Without it, a matching first failure that passes on retry makes the lane green while discarding the original traceback. With broad HTTP/CUDA/Modal message matching, we must retain the evidence that explains why a rerun occurred. Please add a behavioral plugin test that proves a matching failure reruns, a plain assertion does not, and the recovered traceback is reported; the current text-presence contract does not exercise runtime behavior.

2. Choose one retry owner for SSIM, LoRA Training, and VSA. These item-level --reruns 2 currently compose with .buildkite/pipeline.yml's existing exit_status: 1, limit: 2 whole-job retries (both direct and full-suite variants), so one persistent matching failure can run three times in each of three jobs—up to nine attempts while repeating already-successful GPU work. Either remove those exit-1 job retries as part of this change or explicitly disable item-level retries for those lanes. CUDA error: operation not supported can also be deterministic, so the current docs' “deterministic failures fail immediately” claim is not true under the composed policy.

3. Narrow the docs' recovery claim. pytest-rerunfailures can rerun an exception that reaches pytest; it cannot recover a killed/preempted Modal worker or container in this non-xdist setup. Those remain Buildkite/Modal retry territory.

No local pytest was run on the maintainer Mac. After these are addressed, please include the focused runtime-policy evidence and let the exact head rerun Fastcheck before we consider arming Full Suite.