[task160][intern_nem_dev_2] Pin Omni3 Valor32k QA ZIP revision (#268)

* [task160] Pin Valor32k QA ZIP revision

* [task160] Add PR bookkeeping

---------

Co-authored-by: intern_nem_dev_2 <intern_nem_dev_2@intern.local>

Author: songCNMS

src/nemotron/recipes/omni3/stage0_sft/data_prep.py

@@ -107,7 +107,11 @@
 # All knowledge of Valor32k's source layout (URL, tar structure, filename
 # convention, QA shape) lives in this file. The shared SFT-Omni recipe
 # (nemotron.data_prep.recipes.sft_omni) is generic over these.
-VALOR32K_QA_ZIP_URL = "https://github.com/inesriahi/valor32k-avqa-2/raw/refs/heads/main/data.zip"
+VALOR32K_QA_ZIP_REVISION = "a1eeb58e16fbe84f43a3886fd72fe61fd208b7b2"
+VALOR32K_QA_ZIP_URL = (
+    "https://github.com/inesriahi/valor32k-avqa-2/raw/"
+    f"{VALOR32K_QA_ZIP_REVISION}/data.zip"
+)
 # Valor32k filenames have the form {youtube_id}_{start.sss}_{end.sss}.mp4.
 # QA records key by the bare youtube_id, so we strip the timestamp suffix.
 _VALOR32K_TS_SUFFIX_RE = re.compile(r"^(.+)_\d+\.\d+_\d+\.\d+$")
@@ -604,6 +608,7 @@ def _run_valor32k_pipeline(
             "dataset_path": str(dataset_path),
             "raw_dir": str(raw_dir),
             "source_tar": str(cfg.source_tar) if cfg.source_tar else None,
+            "qa_zip_url": cfg.qa_zip_url,
             "samples_per_shard": cfg.samples_per_shard,
             "sample": cfg.sample,
             "split_sample_counts": format_result.split_sample_counts,

tests/recipes/omni3/test_stage0_sft_valor32k_revision_pin.py

@@ -0,0 +1,77 @@
+from __future__ import annotations
+
+import ast
+import re
+from pathlib import Path
+
+from nemotron.recipes.omni3.stage0_sft.data_prep import (
+    VALOR32K_QA_ZIP_URL,
+    Omni3SFTDataPrepConfig,
+)
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+DATA_PREP = REPO_ROOT / "src/nemotron/recipes/omni3/stage0_sft/data_prep.py"
+EXPECTED_REVISION = "a1eeb58e16fbe84f43a3886fd72fe61fd208b7b2"
+EXPECTED_URL = (
+    "https://github.com/inesriahi/valor32k-avqa-2/raw/"
+    f"{EXPECTED_REVISION}/data.zip"
+)
+FLOATING_REF_FRAGMENTS = (
+    "refs/heads/main",
+    "/main/",
+    "refs/heads/master",
+    "/master/",
+)
+
+
+def _module_tree() -> ast.Module:
+    return ast.parse(DATA_PREP.read_text(encoding="utf-8"))
+
+
+def _module_string_constant(tree: ast.Module, name: str) -> str:
+    for node in tree.body:
+        if not isinstance(node, ast.Assign):
+            continue
+        if not any(isinstance(target, ast.Name) and target.id == name for target in node.targets):
+            continue
+        assert isinstance(node.value, ast.Constant)
+        assert isinstance(node.value.value, str)
+        return node.value.value
+    raise AssertionError(f"missing module constant {name}")
+
+
+def _has_manifest_qa_zip_url_entry(tree: ast.Module) -> bool:
+    for node in ast.walk(tree):
+        if not isinstance(node, ast.Dict):
+            continue
+        for key, value in zip(node.keys, node.values, strict=True):
+            if not (isinstance(key, ast.Constant) and key.value == "qa_zip_url"):
+                continue
+            if isinstance(value, ast.Attribute) and value.attr == "qa_zip_url":
+                return True
+    return False
+
+
+def test_valor32k_qa_zip_default_is_pinned_to_exact_revision() -> None:
+    source = DATA_PREP.read_text(encoding="utf-8")
+    tree = _module_tree()
+
+    revision = _module_string_constant(tree, "VALOR32K_QA_ZIP_REVISION")
+    assert revision == EXPECTED_REVISION
+    assert re.fullmatch(r"[0-9a-f]{40}", revision)
+    assert VALOR32K_QA_ZIP_URL == EXPECTED_URL
+    for fragment in FLOATING_REF_FRAGMENTS:
+        assert fragment not in VALOR32K_QA_ZIP_URL
+    assert "refs/heads/main/data.zip" not in source
+
+
+def test_valor32k_config_uses_pinned_qa_zip_default_without_download() -> None:
+    assert Omni3SFTDataPrepConfig().qa_zip_url == EXPECTED_URL
+
+
+def test_valor32k_lineage_records_effective_qa_zip_url() -> None:
+    source = DATA_PREP.read_text(encoding="utf-8")
+    tree = _module_tree()
+
+    assert "source_uri_parts = [cfg.qa_zip_url]" in source
+    assert _has_manifest_qa_zip_url_entry(tree)

workspace/interns/intern_nem_dev_2/status.md

@@ -1,13 +1,13 @@
 # intern_nem_dev_2 - 状态
 
-<!-- METADATA:STATUS=Working,TASK=task158_nemotron_cc_fasttext_hf_revision_pin_s1,ROLE=independent -->
+<!-- METADATA:STATUS=Working,TASK=task160_omni3_valor32k_qa_zip_revision_pin_s1,ROLE=independent -->
 
 | 字段 | 值 |
 |------|-----|
 | Name | intern_nem_dev_2 |
 | Status | Working |
-| Current Task | task158_nemotron_cc_fasttext_hf_revision_pin_s1 |
-| PR | https://github.com/songCNMS/Nemotron/pull/265 |
+| Current Task | task160_omni3_valor32k_qa_zip_revision_pin_s1 |
+| PR | https://github.com/songCNMS/Nemotron/pull/268 |
 | Session | 1 |
 
-最近进展：Opened PR #265 for `task158_nemotron_cc_fasttext_hf_revision_pin_s1`: https://github.com/songCNMS/Nemotron/pull/265. Base `0b31358436c38e698c7c2bc3a89871df273df21c`; implementation head before PR bookkeeping `53ee587143f5c596eddd1d464b9e8eb8dfc1cc6e`. Pinned Nemotron-CC FastText `hf_hub_download` to revision `cd8b714a90f2dbcd3b02cf5fc972e5d7c7f4f107` with static/AST coverage only. Checks passed: focused pytest (`1 passed`), py_compile, Ruff, structured static/AST probe, diff checks, and added-line live-surface scan. No live HF download, Nemotron-CC curation run, Ray/Curator pipeline execution, train/eval, endpoint, W&B, cluster, deploy, artifact upload/download, main/master push, or self-merge.
+最近进展：Opened PR #268 for `task160_omni3_valor32k_qa_zip_revision_pin_s1`: https://github.com/songCNMS/Nemotron/pull/268. Base `9efec596f0401ab2fbe4909ac54e82be8872ec55`; implementation head before PR bookkeeping `bf8b9e61855fa37f9d08749e99d341d81f1c076d`. Pinned the Omni3 Valor32k QA ZIP default away from floating `refs/heads/main` to exact commit `a1eeb58e16fbe84f43a3886fd72fe61fd208b7b2`, preserved operator overrides, and added focused static/AST tests. Checks passed: focused Valor32k pytest (`13 passed`), py_compile, Ruff, structured static/AST probe, diff checks, and added-line live-surface scan. No live Valor32k QA ZIP download, HF/dataset download, SFT data prep, ffmpeg/audio extraction, train/eval, endpoint, W&B, cluster, deploy, artifact upload/download, main/master push, or self-merge.

workspace/tasks/task160_omni3_valor32k_qa_zip_revision_pin_s1/README.md

@@ -0,0 +1,36 @@
+# task160_omni3_valor32k_qa_zip_revision_pin_s1
+
+<!-- METADATA:STATUS=ReadyForGate,ASSIGNEE=intern_nem_dev_2 -->
+
+Status: Ready for PM gate
+Owner: intern_nem_dev_2
+Branch: `intern_nem_dev_2/task160_omni3_valor32k_qa_zip_revision_pin_s1`
+Base: `9efec596f0401ab2fbe4909ac54e82be8872ec55`
+PR: https://github.com/songCNMS/Nemotron/pull/268
+
+## Summary
+
+Pin the Omni3 Valor32k QA ZIP default URL to the PM-provided upstream commit
+instead of the floating `refs/heads/main` ref.
+
+## Scope
+
+- `src/nemotron/recipes/omni3/stage0_sft/data_prep.py`
+- Focused static/AST Valor32k tests under `tests/recipes/omni3/`
+- Task/status docs for `intern_nem_dev_2`
+
+## Boundaries
+
+- No live Valor32k QA ZIP download, HF/dataset download, SFT data prep,
+  ffmpeg/audio extraction, train/eval, endpoint, W&B, cluster, deploy,
+  artifact upload/download, direct `main`/`master` push, or self-merge.
+
+## Acceptance Checks
+
+- PASS: `PYTHONPATH=src /work-agents/.venv/bin/python -m pytest -q tests/recipes/omni3/test_stage0_sft_valor32k_revision_pin.py tests/recipes/omni3/test_stage0_sft_valor_tar_guard.py tests/recipes/omni3/test_stage0_sft_valor32k_config_portability.py` (13 passed)
+- PASS: `/work-agents/.venv/bin/python -m py_compile src/nemotron/recipes/omni3/stage0_sft/data_prep.py tests/recipes/omni3/test_stage0_sft_valor32k_revision_pin.py`
+- PASS: `/work-agents/.venv/bin/ruff check src/nemotron/recipes/omni3/stage0_sft/data_prep.py tests/recipes/omni3/test_stage0_sft_valor32k_revision_pin.py`
+- PASS: structured static/AST QA ZIP probe
+- PASS: `git diff --check`
+- PASS: `git diff --cached --check`
+- PASS: added-line live-surface scan showed static revision pin, manifest metadata, tests, and task/status docs only

workspace/tasks/task160_omni3_valor32k_qa_zip_revision_pin_s1/history_log.md

@@ -0,0 +1,21 @@
+# task160_omni3_valor32k_qa_zip_revision_pin_s1 history
+
+<!-- METADATA:SESSION=1 -->
+
+## Session 1 - 2026-05-29
+
+- Accepted PM assignment and created branch
+  `intern_nem_dev_2/task160_omni3_valor32k_qa_zip_revision_pin_s1` from
+  `origin/main` at `9efec596f0401ab2fbe4909ac54e82be8872ec55`.
+- Added `VALOR32K_QA_ZIP_REVISION` with PM-provided SHA
+  `a1eeb58e16fbe84f43a3886fd72fe61fd208b7b2`.
+- Changed the default Valor32k QA ZIP URL away from floating
+  `refs/heads/main` to the exact commit URL.
+- Preserved operator `qa_zip_url` overrides and recorded effective
+  `cfg.qa_zip_url` in both artifact source lineage and staging manifest
+  metadata.
+- Added focused static/AST tests that do not perform a live URL download.
+- Verified focused Valor32k pytest (`13 passed`), `py_compile`, Ruff,
+  structured static/AST QA ZIP probe, `git diff --check`,
+  `git diff --cached --check`, and added-line live-surface scan.
+- Opened PR #268 to `main`: https://github.com/songCNMS/Nemotron/pull/268.

workspace/tasks/task160_omni3_valor32k_qa_zip_revision_pin_s1/task_knowledge.md

@@ -0,0 +1,13 @@
+# task160_omni3_valor32k_qa_zip_revision_pin_s1 knowledge
+
+<!-- METADATA:SESSION=1 -->
+
+## Working Notes
+
+- Valor32k QA ZIP revision is
+  `a1eeb58e16fbe84f43a3886fd72fe61fd208b7b2`.
+- Pinned default URL is
+  `https://github.com/inesriahi/valor32k-avqa-2/raw/a1eeb58e16fbe84f43a3886fd72fe61fd208b7b2/data.zip`.
+- Tests must not call `urlopen` or perform live Valor32k/HF downloads.
+- Operator `qa_zip_url` overrides remain supported through
+  `Omni3SFTDataPrepConfig.qa_zip_url`.