Add per-task Opus pass@1 report for the ByteDance sample collections - #1105
Draft
huangkicn wants to merge 1 commit into
Draft
Add per-task Opus pass@1 report for the ByteDance sample collections#1105huangkicn wants to merge 1 commit into
huangkicn wants to merge 1 commit into
Conversation
The four 2026-08-04 ByteDance sample experiments are collection
experiments: they pin a task roster (and a task version per task) but
gather only a subset of the trials that exist for those tasks — the
formal-verification collection carries no Opus trials at all even though
claude-code/Opus ran on every one of its tasks. The dashboard therefore
can't show Opus pass@1 per category directly.
scripts/opus_pass1_report.py takes each collection as the task roster,
pulls /tasks/{id}/detail, and scores every trial whose model matches
--model-substring on the collection's pinned task version, using the
dashboard's pass@1 convention (reward == 1 passes; failed/skipped count
in the denominator; probes and superseded retries excluded). It renders
the per-category, per-task chart (docs/bytedance-samples-opus-pass1.png)
and an optional CSV.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HZgGch3FDFNMN5G86Rmczm
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
scripts/opus_pass1_report.py, which plots per-task Claude Opus 4.8 pass@1 for the four 2026-08-04 ByteDance sample collections, plus the generated chart atdocs/bytedance-samples-opus-pass1.png.Why a script (and not just the dashboard)
The sample experiments are collection experiments: they pin a task roster and a task version per task, but gather only a subset of the trials that exist for those tasks. The formal-verification collection (
ffe287bd) carries no Opus trials at all, even though claude-code/Opus ran on the pinned version of every one of its tasks in other experiments — so no single dashboard page can show Opus pass@1 per category.The script treats each collection as the task roster, pulls
/tasks/{id}/detail(combine copies already excluded server-side), and scores every trial whose model matches--model-substringon the collection's pinned task version, using the dashboard's pass@1 convention (reward == 1passes; failed/skipped trials stay in the denominator; probes and superseded retries excluded).Results (2026-08-06)
bc9dcf817fac77d1a32324bcffe287bd“Not run” rows mean no Opus trials exist on the sample's pinned task version (Opus may have run on other versions of the task).
Regenerate
ODDISH_API_KEY=... uv run scripts/opus_pass1_report.py \ --out-png docs/bytedance-samples-opus-pass1.png --out-csv opus-pass1.csvCollections, model substring, and API URL are flags; dependencies (matplotlib) resolve from the script's inline PEP 723 metadata.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HZgGch3FDFNMN5G86Rmczm
Generated by Claude Code