|
1 | 1 | # task297 current-main equivalence review report |
2 | 2 |
|
3 | | -<!-- METADATA:STATUS=Hold,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 --> |
| 3 | +<!-- METADATA:STATUS=ReadyForPR,ASSIGNEE=intern_nemotron_worker_4,SESSION=5 --> |
4 | 4 |
|
5 | 5 | ## Decision |
6 | 6 |
|
7 | | -- Decision: `HOLD_WAITING_TASK296` |
8 | | -- Current main observed: |
9 | | - `2d84ec75960fb51ba9091427638b00083625e137` |
| 7 | +- Decision: `APPROVE_A_PROVED_NO_RERUN_WITH_RESIDUALS` |
| 8 | +- Reviewed upstream PR: #359 |
| 9 | +- Reviewed upstream head: |
| 10 | + `b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06` |
| 11 | +- Reviewed upstream state: `OPEN`, base `main`, merge state `CLEAN` |
10 | 12 | - Task297 PR: |
11 | 13 | `https://github.com/songCNMS/Nemotron/pull/358` |
12 | | -- Lead docs source: |
13 | | - `origin/intern_nemotron_lead/session1-recovery-task-docs` |
14 | | - `c01fb6147c4d711c2a4e5f55dcbe2366ee764709` |
| 14 | +- Current `origin/main`: |
| 15 | + `2d84ec75960fb51ba9091427638b00083625e137` |
| 16 | + |
| 17 | +I approve task296's conclusion that current `origin/main` is equivalent to the |
| 18 | +accepted task285/task293 evidence for the scoped current-main no-rerun decision. |
| 19 | +This is not a fresh training/eval approval and does not authorize export, |
| 20 | +endpoint launch, promotion, task243/live AIME eval, task255 reuse, 30B, or |
| 21 | +8-GPU work. |
| 22 | + |
| 23 | +## Scope And Freshness |
| 24 | + |
| 25 | +The substantive task296 report was introduced at |
| 26 | +`b45308e99db75620dd421c4cdc44560cdcda8eec`. The current reviewed #359 head is |
| 27 | +`b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06`. |
| 28 | + |
| 29 | +Freshness checks: |
| 30 | + |
| 31 | +- `git diff --name-only b45308e9..b9c1af29 -- current_main_equivalence_audit_report.md` |
| 32 | + returned no output. |
| 33 | +- `git diff --name-status b45308e9..b9c1af29` changed only worker_1 status plus |
| 34 | + task296 `history_log.md` and `task_knowledge.md`. |
| 35 | +- `git diff --check b45308e9..b9c1af29` was clean. |
| 36 | + |
| 37 | +Therefore the current #359 head is a metadata/status refresh over the same |
| 38 | +substantive task296 audit evidence. |
| 39 | + |
| 40 | +## Commands And Checks |
| 41 | + |
| 42 | +Read-only review commands included: |
| 43 | + |
| 44 | +```bash |
| 45 | +git fetch origin main intern_nemotron_worker_1/task296_qwen_aime_v11_current_main_equivalence_audit_s1 |
| 46 | +gh pr view 359 --json number,state,headRefName,headRefOid,baseRefName,mergeStateStatus,isDraft,url |
| 47 | +git rev-parse origin/main |
| 48 | +git log --oneline --decorate -5 origin/intern_nemotron_worker_1/task296_qwen_aime_v11_current_main_equivalence_audit_s1 |
| 49 | +git diff --name-status b45308e99db75620dd421c4cdc44560cdcda8eec..b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06 |
| 50 | +git diff --check b45308e99db75620dd421c4cdc44560cdcda8eec..b9c1af2986f5cdfec20c7091ffa2bc6c0b246f06 |
| 51 | +git show b45308e99db75620dd421c4cdc44560cdcda8eec:workspace/tasks/task296_qwen_aime_v11_current_main_equivalence_audit_s1/current_main_equivalence_audit_report.md |
| 52 | +gh pr view 312 --json number,state,baseRefName,headRefName,headRefOid,mergeCommit,mergedAt,url,files |
| 53 | +git diff --name-status 5d8b8d850d26e785332f8b707c772d99881a1b5d..2d84ec75960fb51ba9091427638b00083625e137 |
| 54 | +git diff --stat 5d8b8d850d26e785332f8b707c772d99881a1b5d..2d84ec75960fb51ba9091427638b00083625e137 |
| 55 | +git diff --name-status c53095a639f0ccf8ce34afcec1bdf302cf45add6..2d84ec75960fb51ba9091427638b00083625e137 -- src/nemotron/recipes/super3/stage1_sft/qwen_local_train.py src/nemotron/recipes/super3/stage1_sft/config/m1_agentic_train.yaml src tests workspace/tasks/task276_qwen_aime_v11_rematerialize_packed_qwen_s1 workspace/tasks/task283_qwen_aime_v11_bridge_runtime_remediation_preflight_s1 workspace/tasks/task285_qwen_aime_v11_bounded_qwen4b_sft_smoke_s1 |
| 56 | +git diff --name-only c53095a639f0ccf8ce34afcec1bdf302cf45add6..2d84ec75960fb51ba9091427638b00083625e137 -- src tests | wc -l |
| 57 | +git diff --name-status 87de0a97e6c0406a4b67520faab6b11d91d9131e..2d84ec75960fb51ba9091427638b00083625e137 -- workspace/tasks/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1/run_no_export_aime_eval.py workspace/tasks/task291_qwen_aime_v11_no_export_canary_route_unblock_s1/run_no_export_canary_probe.py src tests workspace/tasks/task291_qwen_aime_v11_no_export_canary_route_unblock_s1 workspace/tasks/task292_qwen_aime_v11_task291_canary_route_review_s1 workspace/tasks/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1 |
| 58 | +git diff --name-only 87de0a97e6c0406a4b67520faab6b11d91d9131e..2d84ec75960fb51ba9091427638b00083625e137 -- src tests | wc -l |
| 59 | +git diff --name-status 87de0a97e6c0406a4b67520faab6b11d91d9131e..2d84ec75960fb51ba9091427638b00083625e137 -- workspace/tasks/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1/run_no_export_aime_eval.py |
| 60 | +gh pr view 350 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url |
| 61 | +gh pr view 351 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url |
| 62 | +gh pr view 356 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url |
| 63 | +gh pr view 357 --json number,state,baseRefName,headRefOid,mergeCommit,mergedAt,url |
| 64 | +sha256sum <task285 report-listed files> |
| 65 | +sha256sum <task293 report-listed files> |
| 66 | +jq '.' <task285 fail_closed_pre_optimizer_preflight_manifest.json> |
| 67 | +jq '.' <task293 summary.json> |
| 68 | +jq '.' <task293 command_env_manifest.json> |
| 69 | +``` |
| 70 | + |
| 71 | +No training, canary, AIME/task243 eval, export, endpoint launch, promotion, |
| 72 | +artifact mutation, shared deletion, main push, merge, 30B, or 8-GPU command was |
| 73 | +run. |
| 74 | + |
| 75 | +## Current Main Delta |
| 76 | + |
| 77 | +PR #312 is merged into current main: |
| 78 | + |
| 79 | +- PR: `https://github.com/songCNMS/Nemotron/pull/312` |
| 80 | +- State: `MERGED` |
| 81 | +- Head: `c7ada6134f63c88d1efcbf993452186d14ae24f3` |
| 82 | +- Merge commit: `2d84ec75960fb51ba9091427638b00083625e137` |
| 83 | +- MergedAt: `2026-06-02T12:13:44Z` |
| 84 | + |
| 85 | +The delta from `5d8b8d850d26e785332f8b707c772d99881a1b5d` to current main |
| 86 | +changed only coordinator workspace documentation: |
| 87 | + |
| 88 | +- `workspace/interns/intern_nemotron_coordinator/status.md` |
| 89 | +- `workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/history_log.md` |
| 90 | +- `workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/session16_aime2025_qwen_handoff.md` |
| 91 | +- `workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/task_knowledge.md` |
| 92 | + |
| 93 | +No changed #312 file is product code, data prep, training, eval, harness, |
| 94 | +model/recipe code, `src/`, `tests/`, or task285/task293 runner code. |
| 95 | + |
| 96 | +## Task285 Equivalence |
| 97 | + |
| 98 | +Task285 source head: |
| 99 | +`c53095a639f0ccf8ce34afcec1bdf302cf45add6`. |
| 100 | + |
| 101 | +Comparison to current main over `src`, `tests`, |
| 102 | +`qwen_local_train.py`, `m1_agentic_train.yaml`, task276, task283, and task285 |
| 103 | +showed: |
| 104 | + |
| 105 | +- `git diff --name-only ... -- src tests | wc -l` returned `0`. |
| 106 | +- `qwen_local_train.py` and `m1_agentic_train.yaml` had no diff. |
| 107 | +- The task276 packed-data task directory had no diff. |
| 108 | +- Diff output was limited to task283/task285 README/report/history/knowledge |
| 109 | + files. |
| 110 | + |
| 111 | +Task285 local artifact root checked: |
| 112 | +`/work-agents/intern_nemotron_worker_2/outputs/task285_qwen_aime_v11_bounded_qwen4b_sft_smoke_s1/run_20260602T061036Z`. |
| 113 | + |
| 114 | +Key recomputed hashes matched task296/task285 carried evidence: |
| 115 | + |
| 116 | +| File | sha256 | |
| 117 | +|---|---| |
| 118 | +| `manifests/fail_closed_pre_optimizer_preflight_manifest.json` | `3b0a3bf3233eacfe8f727ad74e73c9062a86717be7a8c127e452b9fb6283c83c` | |
| 119 | +| `logs/bridge_import_base_proof.log` | `cb1523fffcd97d2b9e5e3b76141624d0d67ad9d2fb1d061e150f15fc7fbf66e6` | |
| 120 | +| `scripts/run_bounded_qwen4b_sft_smoke_retry3.sh` | `14ec9206372a292486ea2a5fff68ec9d35536b4ff80de5901a6e27ade2f12321` | |
| 121 | +| `logs/bounded_qwen4b_sft_smoke_retry3.log` | `096e622a94beae16c114afcf6d6cdd923b01f77d4f5a76200b22eed5fcf0767e` | |
| 122 | +| `manifests/smoke_checkpoints_retry3_inventory.tsv` | `d4cc3d1e5a047e321e98896996610f1ace0b5c45acd3cbe11bb0a8389ea97b78` | |
| 123 | +| `manifests/smoke_checkpoints_retry3_checksums.sha256` | `802ef28a30b7ae5a2359b481fc6c8882d1cc2804d0f1edd25cca84973f7794c4` | |
| 124 | +| `manifests/smoke_checkpoints_retry3_latest_iteration.txt` | `d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35` | |
| 125 | + |
| 126 | +Pre-optimizer manifest evidence: |
| 127 | + |
| 128 | +- `pre_optimizer_fail_closed.status`: `PASS` |
| 129 | +- train/valid/test rows: `279` / `1` / `0` |
| 130 | +- train input/supervised tokens: `1024646` / `228927` |
| 131 | +- Qwen packed chat contract: `PASS` |
| 132 | +- Qwen training pipeline contract: `PASS` |
| 133 | +- bounds: `CUDA_VISIBLE_DEVICES=0,1`, `train_iters=2`, |
| 134 | + `global_batch_size=2`, `micro_batch_size=1`, `lr=5e-7`, `min_lr=1e-7` |
| 135 | + |
| 136 | +Task285 smoke log evidence: |
| 137 | + |
| 138 | +- Iteration 1: LR `3.000000E-07`, lm loss `1.506399E+00`, skipped `0`, |
| 139 | + nan `0`. |
| 140 | +- Iteration 2: LR `1.000000E-07`, lm loss `8.874496E-01`, skipped `0`, |
| 141 | + nan `0`. |
| 142 | +- Latest checkpoint iteration: `2`. |
| 143 | +- Checkpoint size report: `105G`. |
| 144 | +- `SMOKE_RETRY3_COMMAND_RC=1` occurred after iteration-2 checkpoint save when |
| 145 | + the framework entered built-in validation/SIGTERM. This remains a residual, |
| 146 | + not an accepted eval pass. |
| 147 | + |
| 148 | +## Task293 Equivalence |
| 149 | + |
| 150 | +Task293 source head: |
| 151 | +`87de0a97e6c0406a4b67520faab6b11d91d9131e`. |
| 152 | + |
| 153 | +Comparison to current main over `src`, `tests`, task291/task292/task293 docs, |
| 154 | +`run_no_export_aime_eval.py`, and `run_no_export_canary_probe.py` showed: |
| 155 | + |
| 156 | +- `git diff --name-only ... -- src tests | wc -l` returned `0`. |
| 157 | +- Explicit diff for `run_no_export_aime_eval.py` returned no output. |
| 158 | +- Diff output was limited to task293 README/report/history/knowledge files. |
15 | 159 |
|
16 | | -No substantive task296 equivalence review can be completed yet because no |
17 | | -worker_1 task296 branch, PR, report, or mailbox evidence is visible in the |
18 | | -local/remote sources checked during acceptance. |
| 160 | +Task293 local artifact root checked: |
| 161 | +`/work-agents/intern_nemotron_worker_3/outputs/task293_qwen_aime_v11_task285_same_harness_aime_eval_s1/run_20260602T085237Z`. |
19 | 162 |
|
20 | | -## Read-only checks |
| 163 | +Key recomputed hashes matched task296/task293 carried evidence: |
21 | 164 |
|
22 | | -- `git fetch origin main intern_nemotron_lead/session1-recovery-task-docs` |
23 | | -- `git rev-parse origin/main` |
24 | | -- `git rev-parse origin/intern_nemotron_lead/session1-recovery-task-docs` |
25 | | -- `git ls-remote --heads origin '*task296*'` |
26 | | -- `gh pr list --state all --search "task296" --json number,state,headRefName,headRefOid,baseRefName,mergeStateStatus,isDraft,title,url --limit 20` |
| 165 | +| File | sha256 | |
| 166 | +|---|---| |
| 167 | +| `artifacts/aime_eval/summary.json` | `64a378ca54534ec426b92a7b6bc436edb4fddd2ea1ba831f61afeed4e1ad39b7` | |
| 168 | +| `artifacts/aime_eval/results.jsonl` | `4cbc2a9543a658df6a3e18e3128c5a5c9a173f9a575372095cfcbe5d6232aca5` | |
| 169 | +| `artifacts/aime_eval/full_completions.jsonl` | `5cb1e11ab8d331127c7c12f2cd8c04d83d2e6bd93445a5ebffc62363e2a818b4` | |
| 170 | +| `artifacts/manifests/aime_prompt_manifest.json` | `93146086fcc2214fc3c866354e23358d320377caddb6d2b5a2bd58954e85b919` | |
| 171 | +| `artifacts/manifests/checkpoint_load_manifest.json` | `243044f2e548e0c8b1b539e9c11fee17a39b4d45898e1a6601382716e4d90c74` | |
| 172 | +| `artifacts/manifests/command_env_manifest.json` | `5b128b5cc84159b8603b07fc92475ebc768152b7c0ea0fae0897c6635a502ccf` | |
| 173 | +| `artifacts/manifests/checksum_manifest.json` | `6a47e802433648248658010125db51474d0b4af565dc10c637d004900948e7d4` | |
| 174 | +| `logs/remote_no_export_aime_eval.log` | `c0dbfcd93cbb7c615c7f784b201a862e338c4eea23c0faf6d9dd9aa5bdcae4ab` | |
| 175 | +| `logs/remote_no_export_aime_eval_command.txt` | `39bfe804e49eb34ada919ef0ec557313a7cea7eed26c86ab18f746cf2fdd487b` | |
27 | 176 |
|
28 | | -Observed: |
| 177 | +Task293 summary evidence: |
29 | 178 |
|
30 | | -- `origin/main` is |
31 | | - `2d84ec75960fb51ba9091427638b00083625e137`. |
32 | | -- Lead docs branch is |
33 | | - `c01fb6147c4d711c2a4e5f55dcbe2366ee764709`. |
34 | | -- No remote branch matched `*task296*`. |
35 | | -- GitHub PR search for `task296` returned an empty list. |
| 179 | +- disposition: `PASS` |
| 180 | +- total requests: `30` |
| 181 | +- FT score: `12/30 = 0.4` |
| 182 | +- accepted base score: `11/30 = 0.36666666666666664` |
| 183 | +- delta: `+1` correct, `+0.03333333333333338` accuracy |
| 184 | +- request status: `30/30 ok` |
| 185 | +- parsed rows: `21/30` |
| 186 | +- finish reasons: `stop=21`, `length=9` |
| 187 | +- prompt token mismatch count versus accepted task247 base: `0` |
| 188 | +- same AIME score cache sha: |
| 189 | + `c8b287d9784d1d4ae5d3ea593a70850aea69b289e3d42e05951c5488330eaf74` |
| 190 | +- same row count/denominator, prompt variant, max tokens, and parser/normalizer |
| 191 | + proof are present. |
| 192 | +- checkpoint load manifest has `load_megatron_model=PASS` and |
| 193 | + `model_eval=true`. |
| 194 | +- route: `direct_in_process_mcore_static_engine_no_export_no_endpoint_topk1_greedy_corrected_aime25` |
36 | 195 |
|
37 | | -## Boundary confirmation |
| 196 | +Task293 boundary confirmations recorded no export, no endpoint, no promotion, |
| 197 | +no task255 reuse, no AIME2025 train prompts or labels, one GPU, no 30B, and no |
| 198 | +8-GPU. |
38 | 199 |
|
39 | | -This acceptance/HOLD pass was read-only. No training, canary, AIME/task243 eval, |
40 | | -export, endpoint, promotion, task255 reuse, AIME2025 train data, shared deletion, |
41 | | -main push, merge, 30B, or 8-GPU action was performed. |
| 200 | +## Residual Risks |
42 | 201 |
|
43 | | -## Next review trigger |
| 202 | +- task285 smoke command has `SMOKE_RETRY3_COMMAND_RC=1` after iteration-2 |
| 203 | + checkpoint save because built-in validation started and was interrupted by |
| 204 | + SIGTERM. The checkpoint/optimizer evidence is usable for the bounded smoke |
| 205 | + artifact, but this is not an eval pass. |
| 206 | +- task276 packed data remains sparse for validation/test: valid has one row and |
| 207 | + test has zero rows. |
| 208 | +- task292/task291 retained canary route carried the synthetic-row detokenized |
| 209 | + fallback residual. |
| 210 | +- task293 has `sampling_exact_parameter_match=false`: accepted base used SGLang |
| 211 | + endpoint settings while FT used the approved no-export/no-endpoint local MCore |
| 212 | + top-k-1 greedy route. I accept this only as semantic greedy equivalence, not |
| 213 | + byte-identical transport/sampling equivalence. |
| 214 | +- The approval is limited to current-main no-rerun equivalence. It does not |
| 215 | + authorize release, promotion, export, endpoint, task243/live AIME eval, fresh |
| 216 | + training/eval, task255 reuse, 30B, or 8-GPU work. |
44 | 217 |
|
45 | | -Refresh once worker_1 publishes task296 exact branch/head/report or official |
46 | | -mailbox evidence. The required downstream decision remains one of: |
| 218 | +## Boundary Confirmation |
47 | 219 |
|
48 | | -- `APPROVE_A_PROVED_NO_RERUN_WITH_RESIDUALS` |
49 | | -- `REQUEST_CHANGES` |
50 | | -- `BLOCK_B_REQUIRED_RERUN` |
| 220 | +This task297 refresh was review-only apart from updating task297/status docs on |
| 221 | +the worker_4 review branch. I did not train, run canary, run AIME/task243 eval, |
| 222 | +export, launch an endpoint, promote, reuse task255, use AIME2025 prompt/label |
| 223 | +train data, delete shared files, push main, merge, use 30B, or use 8-GPU. |
0 commit comments