Skip to content

Commit 019f632

Browse files
author
intern_nemotron_worker_4
committed
task336 review task335 launch preflight blocker
1 parent e4bc330 commit 019f632

5 files changed

Lines changed: 215 additions & 6 deletions

File tree

‎workspace/interns/intern_nemotron_worker_4/status.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
# intern_nemotron_worker_4 - status
22

3-
<!-- METADATA:STATUS=Working,TASK=task336_qwen_all_sft_task335_independent_review_s1,ROLE=worker,TEAM_ID=nemotron -->
3+
<!-- METADATA:STATUS=ReadyForPR,TASK=task336_qwen_all_sft_task335_independent_review_s1,ROLE=worker,TEAM_ID=nemotron -->
44

55
| Field | Value |
66
|------|-----|
77
| Name | intern_nemotron_worker_4 |
8-
| Status | Working |
8+
| Status | ReadyForPR |
99
| Role | worker |
1010
| Team | nemotron |
1111
| Current Task | task336_qwen_all_sft_task335_independent_review_s1 |
1212
| PR | pending |
1313
| Session | 1 |
14-
| Progress | Accepted task336 on branch `intern_nemotron_worker_4/task336_qwen_all_sft_task335_independent_review_s1` from required `origin/main` `76b9ebf98e623cb85075378ca9980ba6ee11c8ed`, with lead docs imported from `a2e4b731f63bbce9d074f994720abc57db976ac8`. Review target is #398 exact head `0a094483458f01813b50e4fb13e2ddefdbdc4517` and task335 artifact root `/work-agents/intern_nemotron_worker_2/outputs/task335_qwen_all_sft_task333_30b_launch_preflight_s1/run_20260604T090300Z`. Scope is independent read-only review of `BLOCK_LAUNCH_PREFLIGHT` / missing `megatron.energon` blocker evidence plus pass subchecks/checksums. No task335 artifact or worker_2 branch mutation; no training/eval/export/endpoint/promotion/30B/task310/task255/AIME2025 train rows/shared deletion/main push/merge/self-merge. |
14+
| Progress | Completed independent read-only review of #398 exact head `0a094483458f01813b50e4fb13e2ddefdbdc4517` and task335 artifact root `/work-agents/intern_nemotron_worker_2/outputs/task335_qwen_all_sft_task333_30b_launch_preflight_s1/run_20260604T090300Z`. Disposition: `APPROVE_TASK335_BLOCKER_DOCS_CLOSEOUT`. Verified PR scope/diff-check, helper compile, artifact checksum manifest, train-only shard checksum manifest, final summary `BLOCK_LAUNCH_PREFLIGHT`, remote probe `BLOCK_RUNTIME_MISSING_IMPORT`, pass subchecks, and exact missing `megatron.energon` blocker. No task335 artifact or worker_2 branch mutation; no training/eval/export/endpoint/promotion/30B/task310/task255/AIME2025 train rows/shared deletion/main push/merge/self-merge. |

‎workspace/tasks/task336_qwen_all_sft_task335_independent_review_s1/README.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# task336_qwen_all_sft_task335_independent_review_s1 - Review task335 launch preflight
22

3-
<!-- METADATA:STATUS=InProgress,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 -->
3+
<!-- METADATA:STATUS=ReadyForPR,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 -->
44

55
## Background
66

@@ -78,6 +78,19 @@ Return one of:
7878
- Mailbox closeout with branch/head/PR, commands run, pass/fail findings,
7979
residuals, and exact decision for #398.
8080

81+
## Worker_4 Review Result
82+
83+
- Report:
84+
`workspace/tasks/task336_qwen_all_sft_task335_independent_review_s1/task335_independent_review_report.md`
85+
- Reviewed #398 exact head:
86+
`0a094483458f01813b50e4fb13e2ddefdbdc4517`
87+
- Disposition: `APPROVE_TASK335_BLOCKER_DOCS_CLOSEOUT`
88+
- Summary: #398 accurately documents a no-training fail-closed launch preflight.
89+
Data/model/resource/Qwen-contract/validation-route subchecks and checksum
90+
manifests pass; the exact launch blocker is missing `megatron.energon` during
91+
import of `megatron.bridge.recipes.qwen.qwen3_moe`. This keeps task310/30B
92+
training HOLD.
93+
8194
## Boundaries
8295

8396
- Read-only review only.

‎workspace/tasks/task336_qwen_all_sft_task335_independent_review_s1/history_log.md‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# task336_qwen_all_sft_task335_independent_review_s1 - history
22

3-
<!-- METADATA:STATUS=InProgress,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 -->
3+
<!-- METADATA:STATUS=ReadyForPR,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 -->
44

55
## 2026-06-04 UTC - Assigned
66

@@ -26,3 +26,24 @@
2626
- Boundaries: no task335 artifact or worker_2 branch mutation; no training,
2727
eval, export, endpoint, promotion, 30B release, task310 release, task255,
2828
AIME2025 train rows, shared deletion, main push, merge, or self-merge.
29+
30+
## Session 1 - 2026-06-04 UTC - Independent review complete
31+
32+
- Rechecked #398 exact head
33+
`0a094483458f01813b50e4fb13e2ddefdbdc4517`: `OPEN`, non-draft, base
34+
`main`, `CLEAN`/`MERGEABLE`.
35+
- Verified PR diff scope and `git diff --check`; helper compile from PR head
36+
passed.
37+
- From assigned artifact root, verified `sha256sum -c
38+
manifests/artifact_checksums.sha256` and `sha256sum -c
39+
manifests/train_only_shard_checksums.sha256` both pass.
40+
- Verified final summary `BLOCK_LAUNCH_PREFLIGHT`, remote probe
41+
`BLOCK_RUNTIME_MISSING_IMPORT`, train-only split exposure 84/0/0,
42+
Qwen3-MoE model/chat-template proof, Qwen contract pass, validation
43+
`do_validation=false` route, and 8 H200 GPU probe.
44+
- Verified exact blocker:
45+
`megatron.bridge.recipes.qwen.qwen3_moe` fails with
46+
`ModuleNotFoundError("No module named 'megatron.energon'")` while base
47+
imports pass.
48+
- Recorded disposition in `task335_independent_review_report.md`:
49+
`APPROVE_TASK335_BLOCKER_DOCS_CLOSEOUT`.

‎workspace/tasks/task336_qwen_all_sft_task335_independent_review_s1/task335_independent_review_report.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# task336 independent review of task335/#398
2+
3+
<!-- METADATA:STATUS=ReadyForPR,DISPOSITION=APPROVE_TASK335_BLOCKER_DOCS_CLOSEOUT,SESSION=1 -->
4+
5+
Generated: 2026-06-04T09:29:39Z
6+
7+
## Decision
8+
9+
`APPROVE_TASK335_BLOCKER_DOCS_CLOSEOUT` for #398 exact head
10+
`0a094483458f01813b50e4fb13e2ddefdbdc4517`.
11+
12+
The reviewed evidence supports accepting #398/task335 as no-training
13+
fail-closed blocker documentation. The data/model/resource/Qwen-contract/
14+
validation-route subchecks are documented and checksum-backed, while the launch
15+
remains blocked by the exact NemTron runtime import failure:
16+
17+
`megatron.bridge.recipes.qwen.qwen3_moe` -> `ModuleNotFoundError("No module named 'megatron.energon'")`.
18+
19+
This approval is blocker-docs closeout only. It does not release task310, 30B
20+
training, optimizer steps, eval, export, endpoint, promotion, task255 reuse,
21+
AIME2025 train rows, shared deletion, main push, merge, or self-merge.
22+
23+
## Target Reviewed
24+
25+
- PR: #398 `https://github.com/songCNMS/Nemotron/pull/398`
26+
- Exact head reviewed: `0a094483458f01813b50e4fb13e2ddefdbdc4517`
27+
- PR state observed: `OPEN`, non-draft, base `main`, `CLEAN`/`MERGEABLE`
28+
- Base observed: `origin/main` `76b9ebf98e623cb85075378ca9980ba6ee11c8ed`
29+
- Local artifact root:
30+
`/work-agents/intern_nemotron_worker_2/outputs/task335_qwen_all_sft_task333_30b_launch_preflight_s1/run_20260604T090300Z`
31+
- Report reviewed:
32+
`workspace/tasks/task335_qwen_all_sft_task333_30b_launch_preflight_s1/task333_30b_launch_preflight_report.md`
33+
34+
## Commands And Checks
35+
36+
Commands were run from
37+
`/work-agents/intern_nemotron_worker_4/Nemotron_task336` unless noted.
38+
39+
```bash
40+
git fetch origin main +pull/398/head:refs/remotes/origin/pr/398
41+
gh pr view 398 --json number,state,isDraft,baseRefName,headRefName,headRefOid,mergeStateStatus,mergeable,url,title
42+
git diff --name-status origin/main...origin/pr/398
43+
git diff --check origin/main...origin/pr/398
44+
tmp=$(mktemp -d)
45+
git show origin/pr/398:workspace/tasks/task335_qwen_all_sft_task333_30b_launch_preflight_s1/build_task335_30b_launch_preflight.py > "$tmp/build_task335_30b_launch_preflight.py"
46+
PYTHONDONTWRITEBYTECODE=1 python3 -m py_compile "$tmp/build_task335_30b_launch_preflight.py"
47+
rm -rf "$tmp"
48+
cd /work-agents/intern_nemotron_worker_2/outputs/task335_qwen_all_sft_task333_30b_launch_preflight_s1/run_20260604T090300Z
49+
sha256sum -c manifests/artifact_checksums.sha256
50+
sha256sum -c manifests/train_only_shard_checksums.sha256
51+
python3 <read-only JSON manifest and split exposure summary>
52+
tail -n 80 logs/remote_no_training_preflight_probe.log
53+
```
54+
55+
Results:
56+
57+
- #398 exact head is
58+
`0a094483458f01813b50e4fb13e2ddefdbdc4517`.
59+
- Diff scope is limited to worker_2 status plus task335 README/history/
60+
task_knowledge, task-local helper, and task335 report.
61+
- `git diff --check origin/main...origin/pr/398`: clean.
62+
- Helper compile from PR head: `PASS`.
63+
- `sha256sum -c manifests/artifact_checksums.sha256`: `PASS`, 16 entries.
64+
- `sha256sum -c manifests/train_only_shard_checksums.sha256`: `PASS`, 84
65+
train-only shard entries.
66+
- `remote_no_training_preflight_probe` log ends with
67+
`TASK335_REMOTE_PREFLIGHT=BLOCK`.
68+
69+
## Artifact And Checksum Verdict
70+
71+
Important artifact checksums match the task335 report and local manifests:
72+
73+
- `manifests/final_summary.json`:
74+
`80a4ddce65f43af87ff269b760db73e5520644b9c528530f2e0df267b9968b6d`.
75+
- `manifests/artifact_checksums.sha256`:
76+
`fedeea0f279cd716ed24d7c352a464b010e7577876d75bb5d156ade292665297`.
77+
- `manifests/remote_no_training_preflight_probe.json`:
78+
`cf0cacc2a42c3e13a8677edcdfd804f27f97e5b7b1cc2b57a5369304409560d8`.
79+
- `manifests/later_launch_contract.json`:
80+
`476b28337526d2057278f82de8e0917b9b33e418d75ac15adda8a8a81c860d6b`.
81+
- `logs/remote_no_training_preflight_probe.log`:
82+
`8fa6724d984d38402324f6a3e91e2ba53a95fd11fcb3eb46b9a3dd925616a210`.
83+
- `manifests/train_only_shard_checksums.sha256`:
84+
`e5abfbdfebe341b8f346c17f33f4b95ff8fe5750411a40efac1b079fa66bb937`.
85+
86+
The checksum manifests validate the files and train-only shards in the assigned
87+
artifact root. I did not modify task335 artifacts.
88+
89+
## Passing Subchecks
90+
91+
The final summary and probe manifests support these pass conditions:
92+
93+
- Final disposition: `BLOCK_LAUNCH_PREFLIGHT`.
94+
- Remote probe disposition: `BLOCK_RUNTIME_MISSING_IMPORT`.
95+
- Qwen3-30B model path exists:
96+
`/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507`.
97+
- Model metadata: `model_type=qwen3_moe`,
98+
architecture `Qwen3MoeForCausalLM`.
99+
- Tokenizer chat template present; tokenizer/model probing used
100+
`trust_remote_code=false`.
101+
- task333 full packed root exists and exposes train/valid/test as 84/6/6.
102+
- task333 artifact checksum rc `0`, packed shard checksum rc `0`, and accepted
103+
Qwen3-30B packed contract pass are carried in the local probe.
104+
- Task-owned train-only launch view exposes 84 train parquet files and 0
105+
valid/test parquet files.
106+
- Train-only metrics: 78,168 rows, 300,046,415 input tokens,
107+
33,477,337 supervised tokens, 154,008,682 bytes.
108+
- Remote Qwen packed/training contract status: `PASS`.
109+
- Validation route: `valid_parquet_count=0`,
110+
`do_validation_expected=false`, source contains the `has_validation_data =
111+
False` route and returns `do_validation=has_validation_data`.
112+
- GPU resource probe: 8 NVIDIA H200 GPUs, each 143,771 MiB, 0% observed
113+
utilization during the probe.
114+
- Later launch contract is no-training preflight only and keeps required
115+
placeholders unset for train iters, LR/min LR/warmup/save interval, and
116+
lead-approved Bridge checkpoint path.
117+
118+
## Blocker Verdict
119+
120+
The exact blocker is correctly classified:
121+
122+
- `megatron`: `PASS`
123+
- `megatron.bridge`: `PASS`
124+
- `megatron.bridge.training.config`: `PASS`
125+
- `torch`: `PASS`
126+
- `omegaconf`: `PASS`
127+
- `nemotron.recipes.super3.stage1_sft.qwen3_30b_a3b_local_train`: `PASS`
128+
- `nemotron.recipes.super3.stage1_sft.qwen_chat_contract`: `PASS`
129+
- `megatron.bridge.recipes.qwen.qwen3_moe`: `FAIL`
130+
131+
Failure:
132+
`ModuleNotFoundError("No module named 'megatron.energon'")`.
133+
134+
This supports `BLOCK_LAUNCH_PREFLIGHT` and keeps task310/all-SFT 30B launch on
135+
HOLD until a later lead-approved runtime remediation makes the Qwen3 MoE Bridge
136+
recipe import successfully in the same task-owned NemTron `/root` sync route.
137+
138+
## Consistency Notes
139+
140+
- The artifact command-env manifest records worker_2 branch head
141+
`76227ae1ccf483579f19a3288778ced2f32262c6`, while #398 exact head is
142+
`0a094483458f01813b50e4fb13e2ddefdbdc4517`.
143+
- I checked `76227ae1..0a094483`: the drift is task335 docs/status/report/
144+
helper publication, and `git diff --check` is clean.
145+
- The remote synced repo used for runtime debug is current main
146+
`76b9ebf98e623cb85075378ca9980ba6ee11c8ed`, matching the task335 report.
147+
148+
## Residual Risks
149+
150+
- This is blocker evidence only; no actual optimizer/training launch happened.
151+
- The later launch template is intentionally not runnable without lead-supplied
152+
placeholders and a lead-approved imported Bridge checkpoint path.
153+
- The task-owned train-only view intentionally omits valid/test shards to keep
154+
validation disabled; future training must preserve that fail-closed route or
155+
rerun an equivalent no-training preflight.
156+
- The runtime remains unusable for launch until `megatron.energon` is available
157+
in the NemTron route and `megatron.bridge.recipes.qwen.qwen3_moe` imports.
158+
- Any future checkpoint, canary, or same-harness eval requires separate lead
159+
assignment and independent review.
160+
161+
## Boundary Confirmation
162+
163+
Confirmed from the report/manifests and my own actions:
164+
165+
- No task335 artifact or worker_2 branch mutation.
166+
- No optimizer steps or training loop.
167+
- No benchmark eval, AIME eval, or task243 eval.
168+
- No export, endpoint, promotion, 30B release, or task310 release.
169+
- No task255 reuse and no AIME2025 train rows.
170+
- No shared deletion/mutation.
171+
- No main push, merge, or self-merge.

‎workspace/tasks/task336_qwen_all_sft_task335_independent_review_s1/task_knowledge.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# task336_qwen_all_sft_task335_independent_review_s1 - task knowledge
22

3-
<!-- METADATA:STATUS=InProgress,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 -->
3+
<!-- METADATA:STATUS=ReadyForPR,ASSIGNEE=intern_nemotron_worker_4,SESSION=1 -->
44

55
1. Review target is #398 exact head
66
`0a094483458f01813b50e4fb13e2ddefdbdc4517`.
@@ -17,3 +17,7 @@
1717
from `origin/main` `76b9ebf98e623cb85075378ca9980ba6ee11c8ed`; lead docs
1818
source is `origin/intern_nemotron_lead/session1-recovery-task-docs`
1919
`a2e4b731f63bbce9d074f994720abc57db976ac8`.
20+
7. Independent review result for #398 exact head
21+
`0a094483458f01813b50e4fb13e2ddefdbdc4517` is
22+
`APPROVE_TASK335_BLOCKER_DOCS_CLOSEOUT`: #398 is accurate blocker evidence
23+
for missing `megatron.energon`; task310/30B launch remains HOLD.

0 commit comments

Comments
 (0)