Skip to content

Commit 834472e

Browse files
songCNMSsongCNMS
authored
Merge pull request #358 from songCNMS/intern_nemotron_worker_4/task297_qwen_aime_v11_current_main_equivalence_review_s1
task297: hold for task296 equivalence evidence
2 parents 2d84ec7 + 6b46bfb commit 834472e

5 files changed

Lines changed: 438 additions & 5 deletions

File tree

‎workspace/interns/intern_nemotron_worker_4/status.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
# intern_nemotron_worker_4 - status
22

3-
<!-- METADATA:STATUS=Working,TASK=task294_qwen_aime_v11_task293_aime_gate_review_s1,ROLE=worker,TEAM_ID=nemotron -->
3+
<!-- METADATA:STATUS=Working,TASK=task297_qwen_aime_v11_current_main_equivalence_review_s1,ROLE=worker,TEAM_ID=nemotron -->
44

55
| Field | Value |
66
|------|-----|
77
| Name | intern_nemotron_worker_4 |
88
| Status | Working |
99
| Role | worker |
1010
| Team | nemotron |
11-
| Current Task | task294_qwen_aime_v11_task293_aime_gate_review_s1 |
12-
| PR | https://github.com/songCNMS/Nemotron/pull/357 |
13-
| Session | 1 |
14-
| Progress | Completed read-only review of task293 evidence source head `87de0a97e6c0406a4b67520faab6b11d91d9131e` and artifact root `/work-agents/intern_nemotron_worker_3/outputs/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1/run_20260602T085237Z`; decision `APPROVE_AIME_GATE_PASS_WITH_RESIDUAL` because FT `12/30 = 0.4` beats accepted base `11/30 = 0.36666666666666664`, with accepted residual `sampling_exact_parameter_match=false`. Opened review PR #357. No code edit, eval rerun, training, export, endpoint, promotion, task255, shared deletion, main push, 30B, or 8-GPU action was performed. |
11+
| Current Task | task297_qwen_aime_v11_current_main_equivalence_review_s1 |
12+
| PR | https://github.com/songCNMS/Nemotron/pull/358 |
13+
| Session | 5 |
14+
| Progress | Reconfirmed task296/#359 exact head `b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06` (`OPEN`/base `main`/`CLEAN`) and kept task297 decision `APPROVE_A_PROVED_NO_RERUN_WITH_RESIDUALS`; #358 remote was still old HOLD head `68bc1dfd3a54714a539211d288931ee6187330c7` before push. Prepared refreshed task297 report/status docs for commit, push, and official mailbox. No training, canary, AIME/task243 eval, export, endpoint, promotion, task255, AIME2025 train data, shared deletion, main push, merge, 30B, or 8-GPU action was performed. |

‎workspace/tasks/task297_qwen_aime_v11_current_main_equivalence_review_s1/README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# task297_qwen_aime_v11_current_main_equivalence_review_s1 - current-main equivalence review
2+
3+
<!-- METADATA:STATUS=InProgress,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 -->
4+
5+
## Background
6+
7+
task296 is assigned to worker_1 to determine whether the existing V11
8+
task285/task293 artifacts are product-code-equivalent to current main
9+
`2d84ec75960fb51ba9091427638b00083625e137` after coordinator PR #312 merged.
10+
11+
Lead needs an independent review before accepting path A and avoiding a fresh
12+
current-main data/training/evaluation rerun.
13+
14+
## Goal
15+
16+
Independently review the exact task296 report/head and decide whether lead may
17+
accept:
18+
19+
`APPROVE_A_PROVED_NO_RERUN_WITH_RESIDUALS`
20+
21+
or must request more evidence / launch path B:
22+
23+
`REQUEST_CHANGES` or `BLOCK_B_REQUIRED_RERUN`.
24+
25+
## Scope
26+
27+
- Wait for worker_1 task296 branch/report/mailbox. If no task296 report is
28+
visible, report `HOLD_WAITING_TASK296`.
29+
- Review the exact task296 head and report.
30+
- Reproduce the essential read-only git/GitHub checks:
31+
- `origin/main` equals `2d84ec75960fb51ba9091427638b00083625e137`;
32+
- #312 mergedAt/mergeCommit/head match coordinator report;
33+
- #312 changed-file list is coordinator docs only;
34+
- task285/task293 relevant code/script paths are unchanged or
35+
artifact-equivalent as claimed;
36+
- task285/task293 artifact roots/checksums and metrics are accurately carried.
37+
- Confirm residuals and boundaries are preserved:
38+
task276 sparse valid/test, task285 post-train eval RC=1, task292 detokenized
39+
fallback residual, task293 `sampling_exact_parameter_match=false`, no
40+
export/endpoint/promotion/further training/eval/task255/AIME2025 train
41+
data/shared deletion/30B/8-GPU.
42+
43+
## Boundaries
44+
45+
- Read-only review only.
46+
- Do not edit code or artifacts.
47+
- Do not run training, canary, AIME/task243 eval, export, endpoint, promotion,
48+
task255 reuse, AIME2025 train data, shared deletion, main push, merge, 30B, or
49+
8-GPU.
50+
51+
## Expected Output
52+
53+
- Worker branch:
54+
`intern_nemotron_worker_4/task297_qwen_aime_v11_current_main_equivalence_review_s1`
55+
- Review report under this task directory.
56+
- Mailbox report to lead with:
57+
- exact task296 head/report reviewed;
58+
- commands/checks run;
59+
- decision `APPROVE_A_PROVED_NO_RERUN_WITH_RESIDUALS`,
60+
`REQUEST_CHANGES`, `BLOCK_B_REQUIRED_RERUN`, or `HOLD_WAITING_TASK296`;
61+
- residual risks and boundary confirmation.
62+
63+
## Acceptance Criteria
64+
65+
- APPROVE: independent review confirms current main is product-code-equivalent
66+
for task285/task293 evidence and no rerun is needed.
67+
- REQUEST-CHANGES: task296 evidence is incomplete or ambiguous but fixable.
68+
- BLOCK: product-code equivalence is false or unprovable; lead should launch
69+
fresh current-main pipeline tasks.
70+
71+
## Assignment
72+
73+
- Team: `nemotron`
74+
- Team lead: `intern_nemotron_lead`
75+
- Worker: `intern_nemotron_worker_4`
76+
- Related task: task296
77+
- Related PRs: #312, #350, #351, #356, #357

‎workspace/tasks/task297_qwen_aime_v11_current_main_equivalence_review_s1/current_main_equivalence_review_report.md‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# task297 current-main equivalence review report
2+
3+
<!-- METADATA:STATUS=ReadyForPR,ASSIGNEE=intern_nemotron_worker_4,SESSION=5 -->
4+
5+
## Decision
6+
7+
- Decision: `APPROVE_A_PROVED_NO_RERUN_WITH_RESIDUALS`
8+
- Reviewed upstream PR: #359
9+
- Reviewed upstream head:
10+
`b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06`
11+
- Reviewed upstream state: `OPEN`, base `main`, merge state `CLEAN`
12+
- Task297 PR:
13+
`https://github.com/songCNMS/Nemotron/pull/358`
14+
- Current `origin/main`:
15+
`2d84ec75960fb51ba9091427638b00083625e137`
16+
17+
I approve task296's conclusion that current `origin/main` is equivalent to the
18+
accepted task285/task293 evidence for the scoped current-main no-rerun decision.
19+
This is not a fresh training/eval approval and does not authorize export,
20+
endpoint launch, promotion, task243/live AIME eval, task255 reuse, 30B, or
21+
8-GPU work.
22+
23+
## Scope And Freshness
24+
25+
The substantive task296 report was introduced at
26+
`b45308e99db75620dd421c4cdc44560cdcda8eec`. The current reviewed #359 head is
27+
`b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06`.
28+
29+
Freshness checks:
30+
31+
- `git diff --name-only b45308e9..b9c1af29 -- current_main_equivalence_audit_report.md`
32+
returned no output.
33+
- `git diff --name-status b45308e9..b9c1af29` changed only worker_1 status plus
34+
task296 `history_log.md` and `task_knowledge.md`.
35+
- `git diff --check b45308e9..b9c1af29` was clean.
36+
37+
Therefore the current #359 head is a metadata/status refresh over the same
38+
substantive task296 audit evidence.
39+
40+
## Commands And Checks
41+
42+
Read-only review commands included:
43+
44+
```bash
45+
git fetch origin main intern_nemotron_worker_1/task296_qwen_aime_v11_current_main_equivalence_audit_s1
46+
gh pr view 359 --json number,state,headRefName,headRefOid,baseRefName,mergeStateStatus,isDraft,url
47+
git rev-parse origin/main
48+
git log --oneline --decorate -5 origin/intern_nemotron_worker_1/task296_qwen_aime_v11_current_main_equivalence_audit_s1
49+
git diff --name-status b45308e99db75620dd421c4cdc44560cdcda8eec..b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06
50+
git diff --check b45308e99db75620dd421c4cdc44560cdcda8eec..b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06
51+
git show b45308e99db75620dd421c4cdc44560cdcda8eec:workspace/tasks/task296_qwen_aime_v11_current_main_equivalence_audit_s1/current_main_equivalence_audit_report.md
52+
gh pr view 312 --json number,state,baseRefName,headRefName,headRefOid,mergeCommit,mergedAt,url,files
53+
git diff --name-status 5d8b8d850d26e785332f8b707c772d99881a1b5d..2d84ec75960fb51ba9091427638b00083625e137
54+
git diff --stat 5d8b8d850d26e785332f8b707c772d99881a1b5d..2d84ec75960fb51ba9091427638b00083625e137
55+
git diff --name-status c53095a639f0ccf8ce34afcec1bdf302cf45add6..2d84ec75960fb51ba9091427638b00083625e137 -- src/nemotron/recipes/super3/stage1_sft/qwen_local_train.py src/nemotron/recipes/super3/stage1_sft/config/m1_agentic_train.yaml src tests workspace/tasks/task276_qwen_aime_v11_rematerialize_packed_qwen_s1 workspace/tasks/task283_qwen_aime_v11_bridge_runtime_remediation_preflight_s1 workspace/tasks/task285_qwen_aime_v11_bounded_qwen4b_sft_smoke_s1
56+
git diff --name-only c53095a639f0ccf8ce34afcec1bdf302cf45add6..2d84ec75960fb51ba9091427638b00083625e137 -- src tests | wc -l
57+
git diff --name-status 87de0a97e6c0406a4b67520faab6b11d91d9131e..2d84ec75960fb51ba9091427638b00083625e137 -- workspace/tasks/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1/run_no_export_aime_eval.py workspace/tasks/task291_qwen_aime_v11_no_export_canary_route_unblock_s1/run_no_export_canary_probe.py src tests workspace/tasks/task291_qwen_aime_v11_no_export_canary_route_unblock_s1 workspace/tasks/task292_qwen_aime_v11_task291_canary_route_review_s1 workspace/tasks/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1
58+
git diff --name-only 87de0a97e6c0406a4b67520faab6b11d91d9131e..2d84ec75960fb51ba9091427638b00083625e137 -- src tests | wc -l
59+
git diff --name-status 87de0a97e6c0406a4b67520faab6b11d91d9131e..2d84ec75960fb51ba9091427638b00083625e137 -- workspace/tasks/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1/run_no_export_aime_eval.py
60+
gh pr view 350 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url
61+
gh pr view 351 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url
62+
gh pr view 356 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url
63+
gh pr view 357 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url
64+
sha256sum <task285 report-listed files>
65+
sha256sum <task293 report-listed files>
66+
jq '.' <task285 fail_closed_pre_optimizer_preflight_manifest.json>
67+
jq '.' <task293 summary.json>
68+
jq '.' <task293 command_env_manifest.json>
69+
```
70+
71+
No training, canary, AIME/task243 eval, export, endpoint launch, promotion,
72+
artifact mutation, shared deletion, main push, merge, 30B, or 8-GPU command was
73+
run.
74+
75+
## Current Main Delta
76+
77+
PR #312 is merged into current main:
78+
79+
- PR: `https://github.com/songCNMS/Nemotron/pull/312`
80+
- State: `MERGED`
81+
- Head: `c7ada6134f63c88d1efcbf993452186d14ae24f3`
82+
- Merge commit: `2d84ec75960fb51ba9091427638b00083625e137`
83+
- MergedAt: `2026-06-02T12:13:44Z`
84+
85+
The delta from `5d8b8d850d26e785332f8b707c772d99881a1b5d` to current main
86+
changed only coordinator workspace documentation:
87+
88+
- `workspace/interns/intern_nemotron_coordinator/status.md`
89+
- `workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/history_log.md`
90+
- `workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/session16_aime2025_qwen_handoff.md`
91+
- `workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/task_knowledge.md`
92+
93+
No changed #312 file is product code, data prep, training, eval, harness,
94+
model/recipe code, `src/`, `tests/`, or task285/task293 runner code.
95+
96+
## Task285 Equivalence
97+
98+
Task285 source head:
99+
`c53095a639f0ccf8ce34afcec1bdf302cf45add6`.
100+
101+
Comparison to current main over `src`, `tests`,
102+
`qwen_local_train.py`, `m1_agentic_train.yaml`, task276, task283, and task285
103+
showed:
104+
105+
- `git diff --name-only ... -- src tests | wc -l` returned `0`.
106+
- `qwen_local_train.py` and `m1_agentic_train.yaml` had no diff.
107+
- The task276 packed-data task directory had no diff.
108+
- Diff output was limited to task283/task285 README/report/history/knowledge
109+
files.
110+
111+
Task285 local artifact root checked:
112+
`/work-agents/intern_nemotron_worker_2/outputs/task285_qwen_aime_v11_bounded_qwen4b_sft_smoke_s1/run_20260602T061036Z`.
113+
114+
Key recomputed hashes matched task296/task285 carried evidence:
115+
116+
| File | sha256 |
117+
|---|---|
118+
| `manifests/fail_closed_pre_optimizer_preflight_manifest.json` | `3b0a3bf3233eacfe8f727ad74e73c9062a86717be7a8c127e452b9fb6283c83c` |
119+
| `logs/bridge_import_base_proof.log` | `cb1523fffcd97d2b9e5e3b76141624d0d67ad9d2fb1d061e150f15fc7fbf66e6` |
120+
| `scripts/run_bounded_qwen4b_sft_smoke_retry3.sh` | `14ec9206372a292486ea2a5fff68ec9d35536b4ff80de5901a6e27ade2f12321` |
121+
| `logs/bounded_qwen4b_sft_smoke_retry3.log` | `096e622a94beae16c114afcf6d6cdd923b01f77d4f5a76200b22eed5fcf0767e` |
122+
| `manifests/smoke_checkpoints_retry3_inventory.tsv` | `d4cc3d1e5a047e321e98896996610f1ace0b5c45acd3cbe11bb0a8389ea97b78` |
123+
| `manifests/smoke_checkpoints_retry3_checksums.sha256` | `802ef28a30b7ae5a2359b481fc6c8882d1cc2804d0f1edd25cca84973f7794c4` |
124+
| `manifests/smoke_checkpoints_retry3_latest_iteration.txt` | `d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35` |
125+
126+
Pre-optimizer manifest evidence:
127+
128+
- `pre_optimizer_fail_closed.status`: `PASS`
129+
- train/valid/test rows: `279` / `1` / `0`
130+
- train input/supervised tokens: `1024646` / `228927`
131+
- Qwen packed chat contract: `PASS`
132+
- Qwen training pipeline contract: `PASS`
133+
- bounds: `CUDA_VISIBLE_DEVICES=0,1`, `train_iters=2`,
134+
`global_batch_size=2`, `micro_batch_size=1`, `lr=5e-7`, `min_lr=1e-7`
135+
136+
Task285 smoke log evidence:
137+
138+
- Iteration 1: LR `3.000000E-07`, lm loss `1.506399E+00`, skipped `0`,
139+
nan `0`.
140+
- Iteration 2: LR `1.000000E-07`, lm loss `8.874496E-01`, skipped `0`,
141+
nan `0`.
142+
- Latest checkpoint iteration: `2`.
143+
- Checkpoint size report: `105G`.
144+
- `SMOKE_RETRY3_COMMAND_RC=1` occurred after iteration-2 checkpoint save when
145+
the framework entered built-in validation/SIGTERM. This remains a residual,
146+
not an accepted eval pass.
147+
148+
## Task293 Equivalence
149+
150+
Task293 source head:
151+
`87de0a97e6c0406a4b67520faab6b11d91d9131e`.
152+
153+
Comparison to current main over `src`, `tests`, task291/task292/task293 docs,
154+
`run_no_export_aime_eval.py`, and `run_no_export_canary_probe.py` showed:
155+
156+
- `git diff --name-only ... -- src tests | wc -l` returned `0`.
157+
- Explicit diff for `run_no_export_aime_eval.py` returned no output.
158+
- Diff output was limited to task293 README/report/history/knowledge files.
159+
160+
Task293 local artifact root checked:
161+
`/work-agents/intern_nemotron_worker_3/outputs/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1/run_20260602T085237Z`.
162+
163+
Key recomputed hashes matched task296/task293 carried evidence:
164+
165+
| File | sha256 |
166+
|---|---|
167+
| `artifacts/aime_eval/summary.json` | `64a378ca54534ec426b92a7b6bc436edb4fddd2ea1ba831f61afeed4e1ad39b7` |
168+
| `artifacts/aime_eval/results.jsonl` | `4cbc2a9543a658df6a3e18e3128c5a5c9a173f9a575372095cfcbe5d6232aca5` |
169+
| `artifacts/aime_eval/full_completions.jsonl` | `5cb1e11ab8d331127c7c12f2cd8c04d83d2e6bd93445a5ebffc62363e2a818b4` |
170+
| `artifacts/manifests/aime_prompt_manifest.json` | `93146086fcc2214fc3c866354e23358d320377caddb6d2b5a2bd58954e85b919` |
171+
| `artifacts/manifests/checkpoint_load_manifest.json` | `243044f2e548e0c8b1b539e9c11fee17a39b4d45898e1a6601382716e4d90c74` |
172+
| `artifacts/manifests/command_env_manifest.json` | `5b128b5cc84159b8603b07fc92475ebc768152b7c0ea0fae0897c6635a502ccf` |
173+
| `artifacts/manifests/checksum_manifest.json` | `6a47e802433648248658010125db51474d0b4af565dc10c637d004900948e7d4` |
174+
| `logs/remote_no_export_aime_eval.log` | `c0dbfcd93cbb7c615c7f784b201a862e338c4eea23c0faf6d9dd9aa5bdcae4ab` |
175+
| `logs/remote_no_export_aime_eval_command.txt` | `39bfe804e49eb34ada919ef0ec557313a7cea7eed26c86ab18f746cf2fdd487b` |
176+
177+
Task293 summary evidence:
178+
179+
- disposition: `PASS`
180+
- total requests: `30`
181+
- FT score: `12/30 = 0.4`
182+
- accepted base score: `11/30 = 0.36666666666666664`
183+
- delta: `+1` correct, `+0.03333333333333338` accuracy
184+
- request status: `30/30 ok`
185+
- parsed rows: `21/30`
186+
- finish reasons: `stop=21`, `length=9`
187+
- prompt token mismatch count versus accepted task247 base: `0`
188+
- same AIME score cache sha:
189+
`c8b287d9784d1d4ae5d3ea593a70850aea69b289e3d42e05951c5488330eaf74`
190+
- same row count/denominator, prompt variant, max tokens, and parser/normalizer
191+
proof are present.
192+
- checkpoint load manifest has `load_megatron_model=PASS` and
193+
`model_eval=true`.
194+
- route: `direct_in_process_mcore_static_engine_no_export_no_endpoint_topk1_greedy_corrected_aime25`
195+
196+
Task293 boundary confirmations recorded no export, no endpoint, no promotion,
197+
no task255 reuse, no AIME2025 train prompts or labels, one GPU, no 30B, and no
198+
8-GPU.
199+
200+
## Residual Risks
201+
202+
- task285 smoke command has `SMOKE_RETRY3_COMMAND_RC=1` after iteration-2
203+
checkpoint save because built-in validation started and was interrupted by
204+
SIGTERM. The checkpoint/optimizer evidence is usable for the bounded smoke
205+
artifact, but this is not an eval pass.
206+
- task276 packed data remains sparse for validation/test: valid has one row and
207+
test has zero rows.
208+
- task292/task291 retained canary route carried the synthetic-row detokenized
209+
fallback residual.
210+
- task293 has `sampling_exact_parameter_match=false`: accepted base used SGLang
211+
endpoint settings while FT used the approved no-export/no-endpoint local MCore
212+
top-k-1 greedy route. I accept this only as semantic greedy equivalence, not
213+
byte-identical transport/sampling equivalence.
214+
- The approval is limited to current-main no-rerun equivalence. It does not
215+
authorize release, promotion, export, endpoint, task243/live AIME eval, fresh
216+
training/eval, task255 reuse, 30B, or 8-GPU work.
217+
218+
## Boundary Confirmation
219+
220+
This task297 refresh was review-only apart from updating task297/status docs on
221+
the worker_4 review branch. I did not train, run canary, run AIME/task243 eval,
222+
export, launch an endpoint, promote, reuse task255, use AIME2025 prompt/label
223+
train data, delete shared files, push main, merge, use 30B, or use 8-GPU.

0 commit comments

Comments
 (0)