Fork refresh blocked: vllm-gpu — Snowball Levanter oracle fails

[loom-oa-dev]

The vllm-gpu fork refresh cannot open its draft Marin pin PR because the descriptor's Snowball parity gate fails in the Levanter reference case on current Marin main. The refreshed vllm-gpu-pp1 and vllm-gpu-pp2 cases pass, but a same-command control against the current GPU release reproduces the Levanter failure with identical metrics, so the gate cannot qualify the refresh.

Reproduce:

1. At Marin 77fb82b478 with current release marin-vllm-gpu-20260827-f0d7cc7f5874, run uv run pytest tests/cluster/vllm/test_snowball_backend_parity.py -m cluster -o addopts= --import-mode=importlib -vv -s.
2. Observe /loom/snowball-parity-levanter-bec78775 fail on tool-atlas-03: maximum probability error 0.09983461487380312 exceeds the configured 0.075 bound. The control pp1 case passes. Its pp2 case exceeds the 30-minute client wait and is killed by fixture cleanup.
3. Pin promoted release marin-vllm-gpu-20260901-199397a5aed7 and run the same command. Observe /loom/snowball-parity-levanter-62cf0ac3 fail with the same case and exact error values, while refreshed pp1 and pp2 both pass (1 failed, 2 passed in 991.35s).

Expected: the descriptor gate passes on Marin's current release and the staged release, or its Levanter oracle is repaired so it can provide a valid blocking comparison.

Refresh evidence:

• Current Marin pin: release marin-vllm-gpu-20260827-f0d7cc7f5874, source f0d7cc7f587482e0ab771e3c9715e726eb914e60.
• Fork stable main: 8536f566fa85665fbe2fec438982bce4612f07b9; staged main-next: 199397a5aed782a26868fe9a0b5425752fa301a6; selected upstream base: 82936c409d17321d5a791796296790508605aaed.
• Candidate run 33456589237 built both architectures after the aarch64 job received swap; release run 33475053108 qualified the exact wheels on H100 and GB200.
• Promoted release records both accelerator validations as passed.
• Stable main remains unchanged. No rollback/date tags or Marin PR were created because the required E2E did not pass.

#7354 recorded an earlier passing Levanter maximum of 0.068273. #7554 concerns a separate vLLM rank-sentinel variance and does not cover this Levanter oracle failure.

[loom-oa-dev]

🤖 The refresh remains held on the current-pin Snowball Levanter baseline. The September 1 control at Marin 77fb82b478 used release marin-vllm-gpu-20260827-f0d7cc7f5874; its levanter-gpu case reported maximum probability error 0.09983461487380312, above the 0.075 limit. The staged release marin-vllm-gpu-20260901-199397a5aed7 produced the same Levanter error while its vllm-gpu-pp1 and vllm-gpu-pp2 cases passed. Candidate run 33456589237 built both architectures, and release run 33475053108 passed H100 and GB200 wheel qualification.

Marin 61e59be455 has no changes since that control to the cluster test, golden configuration, or scoring jobs. No passing current-pin control is linked here.

At 2026-09-07 13:49 UTC, vllm-project/vllm main was 42801b3a6b3bb92397a04117449f4699b09db1df, 1,110 commits past the fork's current overlay base 61874f9842bd084cc5e9e348de4b3b6a8f46cdc8. The existing marin-community/vllm main-next tip 199397a5aed782a26868fe9a0b5425752fa301a6 is 333 upstream commits behind that head.

The vllm-gpu descriptor requires the three-case Snowball parity gate. The refresh workflow cannot qualify a new pin while the current-pin control fails, so this run did not create another staging tip, wheel release, or Marin PR. First determine whether the Levanter result is a model regression or a stale golden and obtain a passing current-pin control. Then rerun the refresh and the full gate:

uv run pytest tests/cluster/vllm/test_snowball_backend_parity.py \
  -m cluster -o addopts= --import-mode=importlib -vv -s

[yonromai]

🤖 The merge-blocking Snowball evidence passes on Marin #9102 at b8f47f799a467f8befb96608ffbedac2783c0277. That revision pins vLLM release marin-vllm-gpu-20260912-177d960f5b43 and MarinSkyRL #544 at 376a3da258fb9e437d507694c7d884e08ef751aa.

The repository pytest entry point could not reach the IAP-fronted Marin hub from the devbox identity and failed before job submission. Four direct cw-us-east-02a controls then invoked the same checked-in scoring functions with the same H100 resources, environment, fixtures, and bounds as their JobRequest definitions:

• /romain/romain.cuda132-snowball-final-177d-checkpoint-v2-01a08d67: succeeded, 1/1 task.
• /romain/romain.cuda132-snowball-final-177d-levanter-01a08d67: succeeded, 1/1 task.
• /romain/romain.cuda132-snowball-final-177d-vllm-pp1-01a08d67: succeeded, 1/1 task.
• /romain/romain.cuda132-snowball-final-177d-vllm-pp2-01a08d67: succeeded, 2/2 tasks in the pipeline-parallel gang.

The checkpoint control compares the pinned native checkpoint with its checkpoint-specific checked-in golden. The Levanter and vLLM controls compare the HF export with the retained cross-backend semantic golden. They use the content-addressed 64-prompt fixture at SHA-256 47863868cbfe336739c8097535f113f4d2dae4954f772eb91511c911433596e8, through 32K context. The 0.075 Levanter and 0.125 vLLM probability-error bounds are unchanged. Levanter uses reference attention through 4096 tokens and FlashAttention 4 for the 16K and 32K cases.

The repair is still on an open draft. I am leaving this issue open for Fixes #8828 to close it when #9102 merges.

[loom-oa-dev]

🤖 A fresh current-pin control at Marin de412954e99488cf004b54104a7eeb4b670d10a2 still fails the descriptor's Levanter case. The exact command completed with 1 failed, 2 passed in 1072.86s: vllm-gpu-pp1 and vllm-gpu-pp2 passed, while /loom/snowball-parity-levanter-7c8eddf5 reproduced tool-atlas-03 maximum probability error 0.09983461487380312 against the unchanged 0.075 bound.

The tested current release remains marin-vllm-gpu-20260827-f0d7cc7f5874, source f0d7cc7f587482e0ab771e3c9715e726eb914e60. The test, checked-in golden, scoring jobs, and GPU descriptor are unchanged from the earlier failing baseline. #9102 contains the passing oracle repair evidence, but it remains open with review required and its repair is absent from Marin main.

This ferry selected upstream vllm-project/vllm 4be3dcf0fc7a9086d978eaea3c5a4d78fb98e44a. The fork's protected main is 9bce8bcfb8676b8507a810522083e0a3bbe0e9f3 on upstream base 9d88ceb02694c6e3df182ec3c87201284f50e2c6; upstream is now 119 commits past that base. main-next remains 15275db540b7fed9670144df4ef7bec2168be36e.

The refresh workflow cannot qualify a new wheel while the current-pin control fails. This run did not move main-next, publish a candidate or release, create tags, or change Marin's pin. Merge the Snowball repair and obtain a passing current-pin control before replaying the 48-commit fork overlay onto the selected upstream head.