Skip to content

Commit 80d1737

Browse files
author
intern_nemotron_worker_5
committed
task310 accept all-sft training gate
1 parent 172cd0e commit 80d1737

5 files changed

Lines changed: 262 additions & 31 deletions

File tree

‎workspace/interns/intern_nemotron_worker_5/status.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
# intern_nemotron_worker_5 - status
22

3-
<!-- METADATA:STATUS=Working,TASK=task301_qwen_aime_v11_30b_full_sft_training_s1,ROLE=worker,TEAM_ID=nemotron -->
3+
<!-- METADATA:STATUS=Working,TASK=task310_qwen_all_sft_30b_full_training_s1,ROLE=worker,TEAM_ID=nemotron -->
44

55
| Field | Value |
66
|------|-----|
77
| Name | intern_nemotron_worker_5 |
88
| Status | Working |
99
| Role | worker |
1010
| Team | nemotron |
11-
| Current Task | task301_qwen_aime_v11_30b_full_sft_training_s1 |
12-
| PR | #362 |
13-
| Session | 15 |
14-
| Last Update | Lead-cleared SIGTERM terminated the validation hang after task301 reached iter `35/35`; wrapper wrote `train_rc=1` and `train_end=2026-06-02T16:58:51Z`, GPUs released, and `iter_0000035` is inventoried/checksummed as `TRAINING_LOOP_COMPLETE__VALIDATION_HANG_TERMINATED__CHECKPOINT_SALVAGE_CANDIDATE` rather than training PASS. |
11+
| Current Task | task310_qwen_all_sft_30b_full_training_s1 |
12+
| PR | pending |
13+
| Session | 2 |
14+
| Last Update | Accepted task310 from refreshed lead docs including `5f4167dc` on branch `intern_nemotron_worker_5/task310_qwen_all_sft_30b_full_training_s1`; current gate disposition is `BLOCK_PRETRAINING_GATE` because task308/task309 worker branches are visible but no PRs or required PASS reports/artifacts are visible. No training/eval/export/endpoint/promotion/product-code edits/shared deletion/main push/merge. |

‎workspace/tasks/task310_qwen_all_sft_30b_full_training_s1/README.md‎

Lines changed: 93 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,105 @@
11
# task310_qwen_all_sft_30b_full_training_s1 - Qwen all-SFT 30B full training gate
22

3-
<!-- METADATA:STATUS=Open,ASSIGNEE= -->
3+
<!-- METADATA:STATUS=Blocked,ASSIGNEE=intern_nemotron_worker_5,SESSION=2 -->
44

5-
## 背景
5+
## Background
66

7-
Coordinator authorized attempting a full all-SFT data->training->testing pipeline, prioritizing current 30B if resource/runtime gates remain valid. This is not a promotion claim.
7+
The user requested a full pipeline on all SFT data and evaluation on available
8+
benchmarks. The coordinator authorized an attempted gate-driven run, not a
9+
promotion claim. The previous 30B AIME attempt failed the hard non-regression
10+
rule: task301/task306 FT `14/30` was below task300 base `15/30`.
811

9-
## 任务目标
12+
## Goal
1013

11-
Launch full Qwen3-30B-A3B all-SFT training only after task308/task309 data gates and runtime/resource gates pass; otherwise fail closed with exact blocker.
14+
Launch full Qwen3-30B-A3B all-SFT training only after data and runtime gates
15+
pass; otherwise fail closed with exact resource/runtime/data blocker.
1216

13-
## 实现说明
17+
## Scope
1418

15-
Use /mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507 unless task308/task309/task298-equivalent review blocks. Do not silently downgrade. No AIME2025 train data, task255 reuse, shared deletion, export/endpoint/promotion, main push, merge, or product-code edits.
19+
- Use current `origin/main` `172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`.
20+
Lead verified this is a docs-only task310 task-doc advance from product-code
21+
baseline `ecb14173a820df377270273b9f7d9d92cb5076d2`.
22+
- Prioritize target model path:
23+
`/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507`.
24+
- Before training, verify:
25+
- task308 `PASS_AUDIT`;
26+
- task309 `PASS_PACKED_CONTRACT`;
27+
- current 30B runtime/resource route is still valid or explicitly refreshed
28+
from task298/task300/task301/task306 evidence;
29+
- exact GPUs, host, parallelism, launch entrypoint, config, output roots, and
30+
checkpoint policy.
31+
- If a required gate is missing, return `BLOCK_PRETRAINING_GATE` or a more
32+
specific blocker; do not silently downgrade to 4B or a different Qwen path.
33+
- When gates pass, run full all-SFT training and record:
34+
- command/env;
35+
- model path and tokenizer;
36+
- packed root and source manifest;
37+
- LR, optimizer, train steps, schedule, seed, precision, parallelism;
38+
- loss and validation summaries;
39+
- checkpoint paths and checksum manifests;
40+
- resource usage and failure/retry notes;
41+
- handoff for task311 canary/benchmark eval.
1642

17-
## 验收标准
43+
## Boundaries
1844

19-
- PASS_TRAINING: usable checkpoint, LR/loss/validation evidence, commands/env, resources, logs, checksums, and handoff for canary/benchmark eval are complete.
20-
- BLOCK: resource/runtime/data gates fail, checkpoint is unusable, or safe 30B training cannot be launched without boundary violations.
45+
- No AIME2025 prompts or labels in training data.
46+
- No task255 reuse.
47+
- No shared deletion under `/mnt/cephfs/data/processing/lei.song`.
48+
- No export, endpoint, promotion, direct main push, merge, or product-code
49+
modification.
50+
- If export/endpoint becomes needed, task311 must treat it as eval-only and
51+
document it separately.
2152

22-
## 分配信息
53+
## Expected Output
2354

24-
- Team：nemotron
25-
- Team lead：intern_nemotron_lead
26-
- Worker：intern_nemotron_worker_5
27-
- 分配方式：team_lead 创建本 task 文档后，通知 worker 接受该 task。
55+
- Worker branch:
56+
`intern_nemotron_worker_5/task310_qwen_all_sft_30b_full_training_s1`.
57+
- Report:
58+
`workspace/tasks/task310_qwen_all_sft_30b_full_training_s1/all_sft_30b_full_training_report.md`.
59+
- Remote training root and local evidence root with launch scripts/configs,
60+
logs, loss/validation summaries, checkpoint manifests, checksums, and task311
61+
handoff.
62+
- Mailbox report with branch/head/PR or blocker, upstream gate references,
63+
exact commands/env, GPU/resource evidence, checkpoint paths, metrics, and
64+
residual risks.
65+
66+
## Acceptance Criteria
67+
68+
- `PASS_TRAINING`: required gates passed, training completed with a usable
69+
Qwen3-30B-A3B all-SFT checkpoint, finite loss/validation evidence, complete
70+
commands/env/logs/checksums, and task311 handoff.
71+
- `REQUEST_CHANGES`: report/artifacts lack required config, metrics, checksums,
72+
validation, or handoff evidence.
73+
- `BLOCK`: data/runtime/resource gates are absent, launch fails without usable
74+
checkpoint, contamination risk appears, shared deletion would be needed, or
75+
the run would require unauthorized downgrade/promotion/export.
76+
77+
## Assignment
78+
79+
- Team: `nemotron`
80+
- Team lead: `intern_nemotron_lead`
81+
- Worker: `intern_nemotron_worker_5`
82+
- Current main: `172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`
83+
- Product-code baseline: `ecb14173a820df377270273b9f7d9d92cb5076d2`
84+
- Upstream dependencies: task308, task309, prior 30B runtime/resource evidence
85+
- Downstream tasks: task311, task312
86+
- Gate state: full training is HOLD until upstream evidence is accepted.
87+
88+
## Current Worker Disposition
89+
90+
Session 2 disposition: `BLOCK_PRETRAINING_GATE`.
91+
92+
Training was not launched because the required upstream gates are not yet
93+
present as accepted evidence:
94+
95+
- task308 has a visible worker branch
96+
`intern_nemotron_worker_1/task308_qwen_all_sft_pipeline_inventory_audit_s1`
97+
at `348cba44c02043cd6310a36ec722a68278288db2`, but no PR or
98+
`all_sft_pipeline_inventory_audit_report.md` PASS artifact is visible.
99+
- task309 has a visible worker branch
100+
`intern_nemotron_worker_2/task309_qwen_all_sft_packed_data_contract_s1`
101+
at `d054925b1792a5365738247eeb8bdec462e1e6c6`, but no PR or
102+
`all_sft_packed_data_contract_report.md` PASS artifact is visible.
103+
104+
Report:
105+
`workspace/tasks/task310_qwen_all_sft_30b_full_training_s1/all_sft_30b_full_training_report.md`.

‎workspace/tasks/task310_qwen_all_sft_30b_full_training_s1/all_sft_30b_full_training_report.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# task310 Qwen all-SFT 30B full training report
2+
3+
<!-- METADATA:STATUS=Blocked,ASSIGNEE=intern_nemotron_worker_5,SESSION=2 -->
4+
5+
## Disposition
6+
7+
`BLOCK_PRETRAINING_GATE`.
8+
9+
No Qwen3-30B-A3B all-SFT training was launched. The task310 launch gate requires
10+
task308 `PASS_AUDIT`, task309 `PASS_PACKED_CONTRACT`, and a valid current 30B
11+
runtime/resource route. The runtime/resource history is present from prior
12+
merged work, but the fresh task308/task309 PASS artifacts are not visible or
13+
accepted, so task310 fails closed before any training command.
14+
15+
## Checked revisions
16+
17+
- Current `origin/main`: `172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`.
18+
- Product-code baseline carried by lead:
19+
`ecb14173a820df377270273b9f7d9d92cb5076d2`.
20+
- Lead docs branch:
21+
`origin/intern_nemotron_lead/session1-recovery-task-docs` at
22+
`9f838e94feccd0aad4b916dc8f29a6e4d0c80133`; requested task310 refresh
23+
commit `5f4167dc` is an ancestor and has no further task310 diff at current
24+
lead head.
25+
- Task310 worker branch:
26+
`intern_nemotron_worker_5/task310_qwen_all_sft_30b_full_training_s1`.
27+
- Target model path, unchanged and not downgraded:
28+
`/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507`.
29+
30+
## Upstream gate findings
31+
32+
| Gate | Required state | Observed state | Task310 effect |
33+
|---|---|---|---|
34+
| task308 all-SFT pipeline inventory audit | `PASS_AUDIT` with inventory report and accepted evidence | Branch `intern_nemotron_worker_1/task308_qwen_all_sft_pipeline_inventory_audit_s1` exists at `348cba44c02043cd6310a36ec722a68278288db2`; only task README/history/knowledge/status files are visible; no PR and no `all_sft_pipeline_inventory_audit_report.md` found | Blocks training |
35+
| task309 all-SFT packed-data contract | `PASS_PACKED_CONTRACT` with packed root, manifests, decontam proof, checksums, and accepted evidence | Branch `intern_nemotron_worker_2/task309_qwen_all_sft_packed_data_contract_s1` exists at `d054925b1792a5365738247eeb8bdec462e1e6c6`; only task README/history/knowledge/status files are visible; no PR and no `all_sft_packed_data_contract_report.md` found | Blocks training |
36+
| task298 30B runtime/resource/base-load | Valid 30B route or refreshed equivalent | Merged evidence on main records `PASS_RUNTIME_RESOURCE_BASE_LOAD_GATE_WITH_TRAINING_LAUNCH_RESIDUALS`, model path exists, host `lg-cmc-b7r201-f08u26-h200-000126`, 8x H200, Bridge import rc `0`, import root `/root/task298_qwen_aime_v11_30b_runtime_resource_base_load_s1/run_20260602T143838Z/qwen3_30b_bridge_import_iter0` | Carried as prior route evidence, but insufficient without task308/task309 |
37+
| task300 30B base comparator | Accepted base comparator for any future judgment | Merged evidence on main records base `15/30 = 0.5` for `/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507` | Reference only; not a training launch gate substitute |
38+
| task301/task306 previous 30B FT attempt | Must not be reused as success evidence | task301 was salvage closeout only; task306 measured task301 FT `14/30 = 0.4666666666666667`, below task300 base `15/30 = 0.5`, disposition `FAIL` | Confirms previous checkpoint is not a promotion/training-success basis |
39+
| task311/task312 downstream gates | Wait for task310 handoff/evidence | Branches exist for task311 at `dd59d5448c44ba9d04facd2af2ddc4a02b54f899` and task312 at `21bfe2045ec5270775239eecf9474f6044272e7c` as acceptance/planning docs | No effect until task310 produces a usable checkpoint |
40+
41+
## Commands and checks run
42+
43+
Read-only repository and PR visibility checks:
44+
45+
```bash
46+
git fetch origin main
47+
git fetch origin intern_nemotron_lead/session1-recovery-task-docs
48+
git merge-base --is-ancestor 5f4167dc origin/intern_nemotron_lead/session1-recovery-task-docs
49+
git diff --name-status 5f4167dc origin/intern_nemotron_lead/session1-recovery-task-docs -- workspace/tasks/task310_qwen_all_sft_30b_full_training_s1
50+
git rev-parse origin/main origin/intern_nemotron_lead/session1-recovery-task-docs
51+
git ls-tree -r --name-only origin/main workspace/tasks | rg 'task(298|300|301|306|308|309|310|311|312)'
52+
git ls-remote --heads origin '*task308*' '*task309*' '*task310*' '*task311*' '*task312*'
53+
gh pr list --state all --search 'task308 OR task309 OR task310 OR task311 OR task312' --json number,state,headRefName,headRefOid,baseRefName,mergeStateStatus,title
54+
gh pr list --state open --limit 200 --json number,state,headRefName,headRefOid,baseRefName,mergeStateStatus,title | jq '.[] | select(.headRefName|test("task30[89]|task31[012]"))'
55+
```
56+
57+
Read-only upstream branch inspections:
58+
59+
```bash
60+
git fetch origin intern_nemotron_worker_1/task308_qwen_all_sft_pipeline_inventory_audit_s1:refs/remotes/origin/intern_nemotron_worker_1/task308_qwen_all_sft_pipeline_inventory_audit_s1
61+
git fetch origin intern_nemotron_worker_2/task309_qwen_all_sft_packed_data_contract_s1:refs/remotes/origin/intern_nemotron_worker_2/task309_qwen_all_sft_packed_data_contract_s1
62+
git fetch origin intern_nemotron_worker_3/task311_qwen_all_sft_benchmark_eval_s1:refs/remotes/origin/intern_nemotron_worker_3/task311_qwen_all_sft_benchmark_eval_s1
63+
git fetch origin intern_nemotron_worker_4/task312_qwen_all_sft_independent_review_runbook_s1:refs/remotes/origin/intern_nemotron_worker_4/task312_qwen_all_sft_independent_review_runbook_s1
64+
git ls-tree -r --name-only origin/intern_nemotron_worker_1/task308_qwen_all_sft_pipeline_inventory_audit_s1 workspace/tasks/task308_qwen_all_sft_pipeline_inventory_audit_s1 workspace/interns/intern_nemotron_worker_1/status.md
65+
git ls-tree -r --name-only origin/intern_nemotron_worker_2/task309_qwen_all_sft_packed_data_contract_s1 workspace/tasks/task309_qwen_all_sft_packed_data_contract_s1 workspace/interns/intern_nemotron_worker_2/status.md
66+
git show origin/intern_nemotron_worker_1/task308_qwen_all_sft_pipeline_inventory_audit_s1:workspace/interns/intern_nemotron_worker_1/status.md
67+
git show origin/intern_nemotron_worker_2/task309_qwen_all_sft_packed_data_contract_s1:workspace/interns/intern_nemotron_worker_2/status.md
68+
git show origin/main:workspace/tasks/task298_qwen_aime_v11_30b_runtime_resource_base_load_s1/30b_runtime_resource_base_load_report.md
69+
git show origin/main:workspace/tasks/task300_qwen_aime_v11_30b_same_harness_testing_s1/30b_base_aime2025_report.md
70+
git show origin/main:workspace/tasks/task306_qwen_aime_v11_30b_task301_same_harness_aime_eval_s1/30b_task301_same_harness_aime_eval_report.md
71+
```
72+
73+
## Launch status
74+
75+
No launch was attempted.
76+
77+
| Requested launch field | Value |
78+
|---|---|
79+
| Training command/env | Not applicable because the pretraining gate is blocked |
80+
| GPUs/parallelism | Not allocated |
81+
| LR/steps/seed | Not selected for execution |
82+
| Loss/validation metrics | Not produced |
83+
| Checkpoint/log roots | Not produced |
84+
| Checksums/artifact inventory | This report is the only task310 artifact so far |
85+
| task311 handoff | Blocked until task310 has a usable checkpoint |
86+
87+
## Boundary assessment
88+
89+
- No training, optimizer step, evaluation, canary, export, endpoint, promotion,
90+
or product-code edit was performed.
91+
- No AIME2025 prompt or label rows were used for training.
92+
- No task255 artifacts were used.
93+
- No files under `/mnt/cephfs/data/processing/lei.song` were deleted or
94+
overwritten.
95+
- No direct push to main or merge was performed.
96+
- No model-path downgrade or substitution was made.
97+
98+
## Residual risk and unblock conditions
99+
100+
Task310 remains blocked until task308 publishes and receives acceptance for a
101+
complete all-SFT pipeline inventory audit and task309 publishes and receives
102+
acceptance for the packed-data contract. After those gates pass, task310 should
103+
refresh the 30B runtime/resource assumptions against the current main and exact
104+
packed root before launching.

‎workspace/tasks/task310_qwen_all_sft_30b_full_training_s1/history_log.md‎

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,39 @@
11
# task310_qwen_all_sft_30b_full_training_s1 - History Log
22

3-
<!-- METADATA:SESSION=0 -->
3+
<!-- METADATA:SESSION=2 -->
44

5-
## Session 0 - 2026-06-03 UTC - Task created by team lead
5+
## Session 0 - 2026-06-03 UTC - Assigned
66

7-
- Team lead `intern_nemotron_lead` 为 worker `intern_nemotron_worker_5` 创建本 task。
8-
- Worker 应接受本 task，按普通 task/PR 流程开发、测试、提交，并在 PR merge 后完成 task。
7+
- Created by `intern_nemotron_lead` for the all-SFT 30B full training gate.
8+
- Assigned to `intern_nemotron_worker_5`.
9+
- Training is explicitly blocked until task308/task309 and runtime/resource
10+
gates pass; no silent downgrade, promotion, export, endpoint, task255 reuse,
11+
AIME2025 train data, shared deletion, direct main push, or merge is allowed.
12+
13+
## Session 1 - 2026-06-03 UTC - Acceptance and fail-closed gate check
14+
15+
- Accepted task310 on branch
16+
`intern_nemotron_worker_5/task310_qwen_all_sft_30b_full_training_s1` from
17+
current `origin/main` `172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`.
18+
- Verified lead docs branch
19+
`origin/intern_nemotron_lead/session1-recovery-task-docs` at
20+
`3e715c7349c9a944eab621193053a45a0363db46`.
21+
- Found task308 and task309 worker branches visible, but no PRs or required
22+
PASS reports/artifacts visible; disposition is `BLOCK_PRETRAINING_GATE`.
23+
- Did not launch training, allocate GPUs, run eval/canary, export, endpoint,
24+
promotion, product-code edits, shared deletion, direct main push, or merge.
25+
26+
## Session 2 - 2026-06-03 UTC - Lead-doc refresh and remote acceptance report
27+
28+
- Refreshed `origin/main` to
29+
`172cd0e7ceaba8ad2b412d1145441dbb4c5fd122`.
30+
- Refreshed lead docs branch to
31+
`9f838e94feccd0aad4b916dc8f29a6e4d0c80133`; requested task310 update
32+
`5f4167dc` is included, and there is no task310 file diff from that commit
33+
to current lead docs head.
34+
- Carried lead's product-code baseline note:
35+
`ecb14173a820df377270273b9f7d9d92cb5076d2`.
36+
- Kept task310 at `BLOCK_PRETRAINING_GATE`: task308/task309 branches are
37+
visible, but accepted PASS reports/PRs are not visible.
38+
- Prepared branch push and mailbox report with no training/eval/export/endpoint
39+
promotion, product-code edit, shared deletion, direct main push, or merge.

‎workspace/tasks/task310_qwen_all_sft_30b_full_training_s1/task_knowledge.md‎

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,30 @@
11
# task310_qwen_all_sft_30b_full_training_s1 - Task Knowledge
22

3-
<!-- METADATA:SESSION=0 -->
4-
5-
## 记录规则
6-
7-
- 只记录本任务相关的事实、决策、踩坑和验证结果。
8-
- 每条尽量一句话，避免重复 README 的完整内容。
3+
<!-- METADATA:SESSION=2 -->
94

105
## Knowledge Entries
116

12-
1. 本 task 由 team_lead `intern_nemotron_lead` 创建并分配给 worker `intern_nemotron_worker_5`。
7+
1. The previous 30B FT checkpoint cannot be promoted or reused as success
8+
evidence because task306 scored `14/30`, below the accepted task300 base
9+
`15/30`.
10+
2. The selected target is
11+
`/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507` unless a
12+
task308/task309/runtime gate reports an exact blocker.
13+
3. Full training must fail closed rather than downgrading to 4B or switching
14+
data/model paths silently.
15+
4. As of Session 1, task308 branch
16+
`intern_nemotron_worker_1/task308_qwen_all_sft_pipeline_inventory_audit_s1`
17+
exists at `348cba44c02043cd6310a36ec722a68278288db2`, but no PR or
18+
`all_sft_pipeline_inventory_audit_report.md` PASS artifact is visible.
19+
5. As of Session 1, task309 branch
20+
`intern_nemotron_worker_2/task309_qwen_all_sft_packed_data_contract_s1`
21+
exists at `d054925b1792a5365738247eeb8bdec462e1e6c6`, but no PR or
22+
`all_sft_packed_data_contract_report.md` PASS artifact is visible.
23+
6. Task310 training must remain unlaunched until task308 and task309 have
24+
accepted PASS evidence and the current 30B runtime/resource assumptions are
25+
refreshed against the exact packed root.
26+
7. Lead docs commit `5f4167dc` updated task310 to current `origin/main`
27+
`172cd0e7ceaba8ad2b412d1145441dbb4c5fd122` and product-code baseline
28+
`ecb14173a820df377270273b9f7d9d92cb5076d2`; current lead docs head
29+
`9f838e94feccd0aad4b916dc8f29a6e4d0c80133` contains no additional task310
30+
file changes beyond that commit.

0 commit comments

Comments
 (0)