Skip to content

Commit 6baa63f

Browse files
author
intern_nemotron_worker_1
committed
Report task308 all SFT inventory audit
1 parent 348cba4 commit 6baa63f

5 files changed

Lines changed: 227 additions & 9 deletions

File tree

‎workspace/interns/intern_nemotron_worker_1/status.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,5 @@
1010
| Team | nemotron |
1111
| Current Task | task308_qwen_all_sft_pipeline_inventory_audit_s1 |
1212
| PR | |
13-
| Session | 85 |
14-
| Progress | Accepted task308 on worker branch from current origin/main; reading current-main all-SFT pipeline and trainable SFT inventory with no-run/no-mutation boundaries. |
13+
| Session | 86 |
14+
| Progress | Completed task308 all-SFT pipeline/inventory audit report and worker-owned manifest; preparing PR and official mailbox, with task309 fail-closed recommendation for unmaterialized generic stage1_sft raw sources. |

‎workspace/tasks/task308_qwen_all_sft_pipeline_inventory_audit_s1/README.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,16 @@
11
# task308_qwen_all_sft_pipeline_inventory_audit_s1 - Qwen all-SFT pipeline and trainable data inventory audit
22

3-
<!-- METADATA:STATUS=InProgress,ASSIGNEE=intern_nemotron_worker_1,SESSION=85 -->
3+
<!-- METADATA:STATUS=ReadyForReview,ASSIGNEE=intern_nemotron_worker_1,SESSION=86 -->
44

55
## Background
66

77
Coordinator requested a new gate-driven all-SFT pipeline review/run from current
8-
`origin/main` `ecb14173a820df377270273b9f7d9d92cb5076d2` after the 30B AIME
9-
V11 fail closeout:
8+
`origin/main` after the 30B AIME V11 fail closeout. The original coordinator
9+
baseline was `ecb14173a820df377270273b9f7d9d92cb5076d2`; current `origin/main`
10+
is `172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`, which lead verified as a
11+
docs-only task310 task-doc commit. Treat `172cd0e7` as the branch base and
12+
`ecb14173` as the product-code baseline unless newer product-code changes
13+
appear.
1014

1115
- task300 accepted Qwen3-30B-A3B base AIME2025 score: `15/30 = 0.5`.
1216
- task306 evaluated task301 FT checkpoint `iter_0000035` at `14/30 =
@@ -24,7 +28,8 @@ attempt.
2428

2529
## Scope
2630

27-
- Inspect current `origin/main` `ecb14173a820df377270273b9f7d9d92cb5076d2`.
31+
- Inspect current `origin/main` `172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`;
32+
product code is unchanged from `ecb14173a820df377270273b9f7d9d92cb5076d2`.
2833
- Map current pipeline stages and entrypoints:
2934
- data prep and source registry;
3035
- packing / `packed_qwen` materialization;
@@ -86,7 +91,8 @@ attempt.
8691
- Team: `nemotron`
8792
- Team lead: `intern_nemotron_lead`
8893
- Worker: `intern_nemotron_worker_1`
89-
- Current main: `ecb14173a820df377270273b9f7d9d92cb5076d2`
94+
- Current main: `172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`
95+
- Product-code baseline: `ecb14173a820df377270273b9f7d9d92cb5076d2`
9096
- Downstream tasks: task309, task310, task311, task312
9197
- Gate state: no training starts until task308 and task309 produce acceptable
9298
evidence.

‎workspace/tasks/task308_qwen_all_sft_pipeline_inventory_audit_s1/all_sft_pipeline_inventory_audit_report.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# task308 all-SFT pipeline inventory audit report
2+
3+
<!-- METADATA:STATUS=ReadyForReview,ASSIGNEE=intern_nemotron_worker_1,SESSION=86 -->
4+
5+
## Decision
6+
7+
Decision: `PASS_AUDIT_WITH_TASK309_FAIL_CLOSED_CONSTRAINTS`.
8+
9+
Current `origin/main` is
10+
`172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`. Product code is unchanged from
11+
baseline `ecb14173a820df377270273b9f7d9d92cb5076d2`; the diff only adds
12+
task310 docs. This audit did not train, pack final data, run eval, export,
13+
launch an endpoint, claim promotion, reuse task255, write AIME2025 train rows,
14+
delete shared files, edit product code, push main, or merge.
15+
16+
The checksum-backed trainable Qwen/M1 V11 sources are ready for task309 as
17+
inputs to a packed-data contract. The generic `stage1_sft/data_blend_raw`
18+
sources are eligible in principle but not task309-ready from current evidence:
19+
their repo/file checksums are known, but exact row counts, no-AIME/decontam
20+
proof, Qwen chat-template packing evidence, and supervised-token counts are not
21+
materialized in this audit.
22+
23+
## Artifact
24+
25+
Task-owned artifact root:
26+
`/work-agents/intern_nemotron_worker_1/outputs/task308_qwen_all_sft_pipeline_inventory_audit_s1/run_20260603T144136Z`
27+
28+
Inventory manifest:
29+
`/work-agents/intern_nemotron_worker_1/outputs/task308_qwen_all_sft_pipeline_inventory_audit_s1/run_20260603T144136Z/all_sft_inventory_manifest.json`
30+
31+
Manifest sha256:
32+
`4f629e015d4e7a8965899f1fb6c1a5e22e4e666fff28c5bfa69d9d9b31f97a61`
33+
34+
## Commands And Environment
35+
36+
Repository:
37+
`/work-agents/intern_nemotron_worker_1/Nemotron`
38+
39+
Lead docs seen:
40+
`5f4167dc819f5313e7db7fc43e57cec113306cc4`
41+
42+
Commands run were read-only except for this report/status and worker-owned
43+
task308 output artifacts:
44+
45+
```bash
46+
git fetch origin main intern_nemotron_lead/session1-recovery-task-docs
47+
git rev-parse origin/main origin/intern_nemotron_lead/session1-recovery-task-docs
48+
git diff --name-status ecb14173a820df377270273b9f7d9d92cb5076d2 172cd0e7ceaba8ad2b412d1145441dbb4c5fd122
49+
rg/sed over stage1_sft, qwen_chat_contract, M1 planner, training, canary, and eval files
50+
jq over task246/task251/task262/task276/task299 manifests
51+
wc -l and sha256sum over local worker-owned sidecar/heldout files
52+
python3 HfApi.dataset_info(files_metadata=True) for public HF repo/file metadata only
53+
python3 task308 inventory manifest writer
54+
```
55+
56+
No HF source JSONL was downloaded and no final packed data was produced.
57+
58+
## Pipeline Map
59+
60+
| Stage | Current entrypoints | Audit status |
61+
| --- | --- | --- |
62+
| SFT raw source registry | `src/nemotron/recipes/super3/stage1_sft/config/data_prep/data_blend_raw.json`, `default.yaml` | Registry exists; generic HF data is not materialized/count/decontam proven for task309. |
63+
| Qwen M1 data prep / packing | `stage1_sft/data_prep.py`, `config/data_prep/qwen_agentic_v0.yaml`, `qwen_chat_contract.py` | Qwen tokenizer-native contract is proven by task276/task299 artifacts. |
64+
| Training planner / launch | `plan_m1_agentic_sft_training.py`, `m1_agentic_train.yaml`, `qwen3_30b_a3b_local_train.py` | Mapped only; no training launch. |
65+
| Checkpoint load / canary | `qwen_aime2025_base_vs_ft_gate.py`, `qwen_v11_export_load_canary_prompts.yaml` | Mapped only; no canary/export/endpoint. |
66+
| Corrected benchmark eval | `stage3_eval/config/m1_corrected_math_comparison.yaml`, `qwen_aime2025_base_vs_ft_gate.py` | Mapped only; no benchmark eval. |
67+
68+
## Trainable Inventory
69+
70+
### Checksum-backed V11 M1 JSONL sources
71+
72+
| Source | Path | Rows | sha256 | Eligibility |
73+
| --- | --- | ---: | --- | --- |
74+
| `m1-agentic-sft-v11-from-m0` | `/work-agents/intern_nemotron_worker_2/outputs/task251_qwen_aime_v10_hotpotqa_loader_unblock_s1/m1_agentic_sft/agentic_sft_v0_train.jsonl` | 1100 | `994166eeb83ffb5ebd213db9cc0d6cdd90208251bd2aab9dbb70cec7bf96691a` | Eligible for task309 packing with task262/task276/task299 decontam contract. |
75+
| `m1-agentic-sft-v11-math-final-answer` | `/work-agents/intern_nemotron_worker_2/outputs/task251_qwen_aime_v10_hotpotqa_loader_unblock_s1/m1_agentic_sft/agentic_sft_v0_math_final_answer_train.jsonl` | 200 | `0e5485eae86bf716d0c2e04e8e02595564b38a949d71d31a42874d6e87ef1731` | Eligible; task262 full token 8-gram final-answer scan passed. |
76+
| `m1-agentic-sft-v11-math-hard-verified-full-solution` | `/work-agents/intern_nemotron_worker_2/outputs/task251_qwen_aime_v10_hotpotqa_loader_unblock_s1/m1_agentic_sft/agentic_sft_v0_math_hard_verified_full_solution_train.jsonl` | 8 | `2039b67b2bcf5cf74b576a640f1f3a198d675e3fbd64a886da4be5753ad515d9` | Eligible but sparse. |
77+
78+
Environment counts for the 1100-row base source are 100 rows each across
79+
11 environments: code execution, general tool calling, math competition,
80+
math reasoning, multi-turn tool use, grounded QA, multihop QA, structured JSON,
81+
SWE pivot, terminal shell, and tool-call repair.
82+
83+
### M0 math sidecar provenance
84+
85+
Task246 M0 V10 math sidecar:
86+
`/work-agents/intern_nemotron_worker_1/outputs/task246_qwen_aime_v10_real_decontam_corpus_s1/m0_v10_math_sidecar`
87+
88+
| File | Rows | sha256 |
89+
| --- | ---: | --- |
90+
| `math_competition_numeric/train-split.jsonl` | 8 | `01ac5d1c8571dc956bbae12b7f1a00a4e759d59e503abbf2ddfba3b85aa324e3` |
91+
| `math_competition_numeric/val-split.jsonl` | 0 | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` |
92+
| `manifest.json` | n/a | `ca7864ce5ddbec20c0e0b1e67fdaefb2b09ef884f430b68fe7158c5b62951477` |
93+
94+
Source dataset: `AI-MO/NuminaMath-CoT`, revision
95+
`9d8d210c9f6a36c8f3cd84045668c9b7800ef517`; task246 scanned 859494 source
96+
rows, selected 8 V10 candidate train rows, and blocked 0 rows by decontam.
97+
98+
### Packed Qwen roots
99+
100+
Task276 Qwen3-4B packed root:
101+
`/work-agents/intern_nemotron_worker_2/outputs/task276_qwen_aime_v11_rematerialize_packed_qwen_s1/run_20260602T034648Z/packed_qwen`
102+
103+
Task299 Qwen3-30B-ready packed root:
104+
`/work-agents/intern_nemotron_worker_1/outputs/task299_qwen_aime_v11_30b_data_packing_contract_s1/run_20260602T150941Z/packed_qwen_30b`
105+
106+
Packed split counts are identical:
107+
108+
| Split | Shards | Rows | Input tokens | Supervised tokens |
109+
| --- | ---: | ---: | ---: | ---: |
110+
| train | 46 | 279 | 1024646 | 228927 |
111+
| valid | 1 | 1 | 1491 | 1428 |
112+
| test | 1 | 0 | 0 | 0 |
113+
114+
Train split source counts:
115+
116+
| Source | Shards | Rows | Input tokens | Supervised tokens |
117+
| --- | ---: | ---: | ---: | ---: |
118+
| `m1-agentic-sft-v11-from-m0` | 16 | 244 | 942062 | 167555 |
119+
| `m1-agentic-sft-v11-math-final-answer` | 16 | 28 | 75305 | 54821 |
120+
| `m1-agentic-sft-v11-math-hard-verified-full-solution` | 14 | 7 | 7279 | 6551 |
121+
122+
Task299 30B top manifest sha256:
123+
`59ee4432b5ddf776f82ee5dff6f45f1a9c1f8f9c7ad99a29d8fcfb96c7e50f3d`
124+
125+
Task299 contract status: tokenizer asset/API/chat-template equivalence passed,
126+
split parity passed, Qwen packed/training contract validators passed, decontam
127+
status passed.
128+
129+
## Generic stage1_sft/data_blend_raw Inventory
130+
131+
Config sha256:
132+
`1c76013e66643972cfa1c96aca86571a666c81d7fd6cf95b024e081f0836503d`
133+
134+
The 12 source entries are present in current main and have HF repo/file
135+
metadata. Exact row counts and decontam status were not produced because this
136+
task did not download or materialize those large JSONL files.
137+
138+
| Source | Repo SHA | File sha256 | Exact rows | Task309 status |
139+
| --- | --- | --- | --- | --- |
140+
| `instruction-following-chat` | `83dcd3aded0d289b0bbc018d3f9af4c5dd4005df` | `37f9ecc3c41dc5e97cfd6fca962a94afbc8713349900ea6f413c040df549ddb8` | unknown | Block until materialized/count/decontam scanned. |
141+
| `instruction-following-structured` | `83dcd3aded0d289b0bbc018d3f9af4c5dd4005df` | `03e4cc9a657f9f193860d82fe49764acc6b298f6ce6811497412aa5a0181ec77` | unknown | Block until materialized/count/decontam scanned. |
142+
| `competitive-cpp-00` | `d6e7c6b404ed5db6e1104b41d0f80a0c7dad7bf8` | `1081e0a650ecdc02df1b4b8b4fecf4b3d39828908874b4bf1a4015e638005c62` | unknown | Block until materialized/count/decontam scanned. |
143+
| `competitive-cpp-01` | `d6e7c6b404ed5db6e1104b41d0f80a0c7dad7bf8` | `4500b6db059765aa6146d3c3247fdde1ce8b5cc762a7687ff4355b45e1701afa` | unknown | Block until materialized/count/decontam scanned. |
144+
| `competitive-python-00` | `d6e7c6b404ed5db6e1104b41d0f80a0c7dad7bf8` | `8314b37b7d42b32fb658c3be1fb974eb0814f44a856ccf2d90ec2d38856a7f5d` | unknown | Block until materialized/count/decontam scanned. |
145+
| `competitive-python-01` | `d6e7c6b404ed5db6e1104b41d0f80a0c7dad7bf8` | `988cc7a00686d6212b3f8fbef95919c8e72bbda81c9f859dd556df789bf44b30` | unknown | Block until materialized/count/decontam scanned. |
146+
| `swe` | `0fe17a965b297a9c943a59050a14c42d5f0083ce` | `1e0fb6d9a8d955fb0f2160e44a4946e5f2c4eb3931e80dadb724ff823cdbc14c` | unknown | Block until materialized/count/decontam scanned. |
147+
| `math-proofs-lean` | `97229c590831adfe96202f5cd071d444d535bf91` | `b423525d35ad16c791863670cbad76b27d8463e2574770732e2cf5bf70661a2e` | unknown | Block until materialized/count/decontam scanned. |
148+
| `agentic-interactive` | `650d590978ca35c8f1ecea2faf136e5fac421b62` | `dcfeda22372fa707c979cab29ddfe896b89a933f15ed4acbb4f16e7e3787d9dd` | unknown | Block until materialized/count/decontam scanned. |
149+
| `agentic-tool-calling` | `650d590978ca35c8f1ecea2faf136e5fac421b62` | `f537a901d38a999627b8fe59e77a1007af0d79d71a892ad9a4a3d80456e5601b` | unknown | Block until materialized/count/decontam scanned. |
150+
| `infinibyte-00` | `d6e7c6b404ed5db6e1104b41d0f80a0c7dad7bf8` | `7d6cc0943a9264696ba177f152fd12c60cc2e1b042787a205221abcd4059c9e7` | unknown | Block until materialized/count/decontam scanned. |
151+
| `infinibyte-01` | `d6e7c6b404ed5db6e1104b41d0f80a0c7dad7bf8` | `0124e374453dce8fa7a6e7ecd75356160f2bde525ba97b246d2b39e8479c4ef3` | unknown | Block until materialized/count/decontam scanned. |
152+
153+
## Exclusions
154+
155+
- AIME2025 prompts and labels remain held out for eval/decontam only.
156+
- Task246 heldout corpus is excluded from trainable data: 560 rows, corpus
157+
sha256 `614b2b347d33c1ec00cfd2c33222c26ad1d99b8b837bd7e48ea11fd4fedae6f9`.
158+
- Task246 prompt hashes are excluded from trainable data: 560 hashes, sha256
159+
`a2a348ee12f962d5dd7ed7cf0e5d034ebea4a76a2287804e97ade331b552a78d`.
160+
- Task251 `agentic_sft_v0_val_shadow.jsonl` is not a train source.
161+
- Task251 `agentic_sft_v0_math_heldout_eval.jsonl` is empty and excluded.
162+
- Task253 stale packed data is excluded because task262 documented split
163+
materialization mismatch.
164+
- Task255 artifacts are excluded entirely and were not reused.
165+
166+
## Task309 Recommendation
167+
168+
Recommend task309 proceed fail-closed:
169+
170+
1. Use task299 as the immediate checksum-backed Qwen3-30B-ready seed only if
171+
task309 wants the already proven V11 M1/math set.
172+
2. If task309 must be "all-SFT" including generic `stage1_sft/data_blend_raw`,
173+
first materialize and count every HF source, run heldout prompt-hash and
174+
full n-gram decontam against AIME25/HMMT/MATH heldouts where relevant,
175+
prove no task255 or AIME2025 prompt/label train rows, and then pack with
176+
`qwen_agentic_v0.yaml` against
177+
`/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507`.
178+
3. Emit split manifest, row/token/supervised-token/source counts,
179+
intended-vs-exposed parity, shard checksums, Qwen chat-template/tokenizer
180+
contract proof, and residual valid/test sparsity risk before task310.
181+
182+
Without step 2, generic `stage1_sft/data_blend_raw` should remain excluded
183+
from the next training attempt.

‎workspace/tasks/task308_qwen_all_sft_pipeline_inventory_audit_s1/history_log.md‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# task308_qwen_all_sft_pipeline_inventory_audit_s1 - History Log
22

3-
<!-- METADATA:SESSION=85 -->
3+
<!-- METADATA:SESSION=86 -->
44

55
## Session 77 - 2026-06-03 UTC - Assigned
66

@@ -21,3 +21,25 @@
2121
- Audit boundaries acknowledged: no training, packing final data, benchmark
2222
eval, export, endpoint, promotion, task255 reuse, AIME2025 train rows,
2323
shared deletion, product-code edits, main push, or merge.
24+
25+
## Session 86 - 2026-06-03 UTC - Inventory audit report
26+
27+
- Fetched current `origin/main`
28+
`172cd0e7ceaba8ad2b412d1145441dbb4c5fd122` and updated lead docs
29+
`5f4167dc819f5313e7db7fc43e57cec113306cc4`.
30+
- Verified product code is unchanged from baseline
31+
`ecb14173a820df377270273b9f7d9d92cb5076d2`; diff to current main only adds
32+
task310 docs.
33+
- Produced report
34+
`workspace/tasks/task308_qwen_all_sft_pipeline_inventory_audit_s1/all_sft_pipeline_inventory_audit_report.md`.
35+
- Produced task-owned inventory manifest
36+
`/work-agents/intern_nemotron_worker_1/outputs/task308_qwen_all_sft_pipeline_inventory_audit_s1/run_20260603T144136Z/all_sft_inventory_manifest.json`
37+
with sha256
38+
`4f629e015d4e7a8965899f1fb6c1a5e22e4e666fff28c5bfa69d9d9b31f97a61`.
39+
- Decision: `PASS_AUDIT_WITH_TASK309_FAIL_CLOSED_CONSTRAINTS`. Task299/V11
40+
Qwen packed root and task262/task276/task246 decontam evidence are
41+
checksum-backed; generic `stage1_sft/data_blend_raw` sources remain blocked
42+
for task309 until materialized, counted, decontam-scanned, and Qwen-packed.
43+
- Boundaries maintained: no training, final packing, benchmark eval, export,
44+
endpoint, promotion, task255 reuse, AIME2025 train rows, shared deletion,
45+
product-code edits, main push, or merge.

‎workspace/tasks/task308_qwen_all_sft_pipeline_inventory_audit_s1/task_knowledge.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# task308_qwen_all_sft_pipeline_inventory_audit_s1 - Task Knowledge
22

3-
<!-- METADATA:SESSION=85 -->
3+
<!-- METADATA:SESSION=86 -->
44

55
## Knowledge Entries
66

@@ -11,3 +11,10 @@
1111
3. The inventory must include `stage1_sft` `data_blend_raw`, task276/task299
1212
packed-data evidence, M1 agentic/math sidecars, and any other eligible SFT
1313
source before task309 can produce a final all-SFT packed contract.
14+
4. Task308 result: current-main generic `stage1_sft/data_blend_raw` has repo
15+
and file checksums but no exact row counts or heldout/AIME decontam proof in
16+
current evidence, so task309 must materialize/count/scan it before including
17+
it in all-SFT packing.
18+
5. Task308 checksum-backed seed for task309 is the task299 30B-ready V11
19+
packed root plus task262/task276/task246 decontam evidence; task253 and
20+
task255 remain excluded.

0 commit comments

Comments
 (0)