coordinator: record Qwen AIME scaleup and all-SFT pipeline requests

workspace/interns/intern_nemotron_coordinator/status.md

@@ -9,6 +9,6 @@
 | Role | coordinator |
 | Team | N/A |
 | Current Task | task_coordinator_nemotron_coordinator_06b9acba |
-| PR | #312 |
-| Session | 44 |
-| Last Update | 2026-06-02T12:11:47Z - User requested PR merge and full current-code pipeline run; verified Qwen AIME V11 pipeline evidence on main and prepared coordinator PR #312 merge plus lead rerun/confirmation goal. |
+| PR | #360 |
+| Session | 45 |
+| Last Update | 2026-06-02T14:26:28Z - User requested 30B full training/testing; created Session 45 branch and delivered lead goal for gated Qwen 30B AIME V11 scale-up. |

workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/history_log.md

@@ -1,6 +1,6 @@
 # task_coordinator_nemotron_coordinator_06b9acba - History Log
 
-<!-- METADATA:SESSION=44 -->
+<!-- METADATA:SESSION=45 -->
 
 ## Session 0 - Created with coordinator
 
@@ -619,3 +619,43 @@
   - no export, endpoint, promotion, task255 reuse, AIME2025 train data, shared deletion, 30B, or 8-GPU is authorized.
 - Prepared to merge #312 with exact-head protection after committing/pushing this Session 44 coordinator record.
 - Planned lead handoff after #312 merge: set a current-main full-pipeline confirmation/rerun goal requiring lead to use the post-merge main, confirm whether task285/task293 artifacts are code-equivalent to current main or launch a fresh current-code run, and report final data/training/eval artifacts, metrics, residuals, and blockers.
+
+## Session 45 - 30B full training and testing requested
+
+- Received user instruction `在 30b 模型上进行完整的训练和测试`.
+- Because PR #312 had been merged in Session 44, created a fresh coordinator branch from current `origin/main`: `intern_nemotron_coordinator/session45-30b-scaleup-coordination`.
+- Fetched `origin`; current state at dispatch time:
+  - `origin/main` is `31137bc1e28f7d08d4c6b5aa2448487d95aa07d7`;
+  - lead branch `origin/intern_nemotron_lead/session1-recovery-task-docs` is `b537d87b`;
+  - open GitHub PR list is empty.
+- Read latest lead Session 75 history and task296/task297 evidence. Current-main 4B equivalence request from Session 44 is closed as path A:
+  - task296/#359 merged at `31137bc1e28f7d08d4c6b5aa2448487d95aa07d7`;
+  - task297/#358 merged at `834472e69b23dc71b49824cda57f866a60839c0a`;
+  - decision is `A_PROVED_NO_RERUN` / `APPROVE_A_PROVED_NO_RERUN_WITH_RESIDUALS`;
+  - existing task285/task293 artifacts are accepted as product-code-equivalent to current main, so a fresh 4B rerun is not required.
+- Preserved known 4B residuals before 30B scale-up:
+  - task285 smoke command ended `RC=1` after iter2 checkpoint during built-in validation/SIGTERM;
+  - task276 valid/test split is sparse;
+  - task292 carries detokenized fallback residual;
+  - task293 `sampling_exact_parameter_match=false` is accepted only as semantic greedy equivalence.
+- Searched available model paths and found relevant local 30B candidates:
+  - `/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507`;
+  - `/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Base`;
+  - `/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Thinking-2507`;
+  - `/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507-FP8`.
+- Searched repository and found existing 30B scale-up clues:
+  - training entrypoint `src/nemotron/recipes/super3/stage1_sft/qwen3_30b_a3b_local_train.py`;
+  - tests around `qwen30b_a3b_local_train`;
+  - prior task071/task075 Qwen3-30B-A3B train/export/eval runbook evidence and notes.
+- Sent delivered peer message to `intern_nemotron_lead` with the user instruction and requested gated 30B full data, training, and testing startup from current main.
+- Set delivered lead pressing goal `qwen-aime-v11-30b-full-train-test-session45`.
+- Created coordinator PR #360 for the Session 45 record, scoped to coordinator status/history/task_knowledge only.
+- Required lead sequence in the 30B handoff:
+  1. 30B runtime/resource/base-load proof with exact model path, GPU/parallelism, entrypoint, and testing route;
+  2. same-harness 30B base AIME2025 score before judging FT;
+  3. task276 V11 data reuse or adaptation only if Qwen 30B chat-template/decontam contract holds;
+  4. full 30B training with checkpoints, LR/loss/validation, logs, env, and checksums;
+  5. non-AIME canary and corrected AIME2025 FT-vs-base testing with completions and parser diagnostics;
+  6. independent review and runbook/provenance.
+- Boundaries preserved in the lead handoff: AIME2025 prompts/labels remain held-out eval/decontam only; no task255 reuse; no shared `/mnt/cephfs/data/processing/lei.song` deletion; any export/endpoint is eval-only if needed, not promotion, unless separately approved.
+- Coordinator did not run 30B training, eval, export, endpoint, promotion, shared deletion, or artifact mutation directly.

workspace/tasks/task_coordinator_nemotron_coordinator_06b9acba/task_knowledge.md

@@ -1,6 +1,6 @@
 # task_coordinator_nemotron_coordinator_06b9acba - Task Knowledge
 
-<!-- METADATA:SESSION=44 -->
+<!-- METADATA:SESSION=45 -->
 
 ## Knowledge Entries
 
@@ -127,3 +127,9 @@
 121. task293 metric evidence: task285 iter2 FT scored corrected AIME2025 `12/30 = 0.4` versus accepted base `11/30 = 0.36666666666666664`, delta `+1/30`, with task294 decision `APPROVE_AIME_GATE_PASS_WITH_RESIDUAL`.
 122. task293/task294 residual: `sampling_exact_parameter_match=false`; accepted only as deterministic greedy semantic match for metric-gate evidence. No export, endpoint, promotion, further training/eval, task255 reuse, AIME2025 train data, shared deletion, 30B, or 8-GPU is authorized.
 123. Session 44 user request requires post-merge current-code confirmation/rerun from lead: use post-merge main, confirm task285/task293 artifact code-equivalence to current code or run a fresh current-code full data/training/eval pass, and report artifacts/metrics/residuals.
+124. Session 45 supersedes the current-code rerun uncertainty: task296/task297 merged into current main `31137bc1e28f7d08d4c6b5aa2448487d95aa07d7`, accepting path A that task285/task293 artifacts are product-code-equivalent to current main after #312.
+125. User Session 45 authorizes 30B full training/testing. Coordinator delivered lead goal `qwen-aime-v11-30b-full-train-test-session45`.
+126. Relevant discovered 30B Qwen path for lead verification: `/mnt/cephfs/data/stable/models/Qwen/Qwen3-30B-A3B-Instruct-2507`; nearby candidates include `Qwen3-30B-A3B-Base`, `Qwen3-30B-A3B-Thinking-2507`, and `Qwen3-30B-A3B-Instruct-2507-FP8`.
+127. Existing repo 30B entrypoint clue: `src/nemotron/recipes/super3/stage1_sft/qwen3_30b_a3b_local_train.py`, with tests around `qwen30b_a3b_local_train` and prior task071/task075 Qwen3-30B-A3B scale-up/eval evidence.
+128. 30B launch sequence must fail closed: prove 30B resource/runtime/base-load, establish 30B same-harness base AIME2025 score before FT judgment, verify data/chat-template/decontam contract, run full 30B train, then non-AIME canary and corrected AIME2025 FT-vs-base test with independent review.
+129. Session 45 30B boundaries: AIME2025 prompts/labels remain held-out eval/decontam only; no task255 reuse; no shared `/mnt/cephfs/data/processing/lei.song` deletion; export/endpoint is eval-only if needed and not promotion unless separately approved.