Update vLLM to pinned version 0.18.1 by chrisalexiuk-nvidia · Pull Request #135 · NVIDIA-NeMo/Nemotron

chrisalexiuk-nvidia · 2026-04-09T18:47:01Z

Bug reported here, pinning to known "best" version.

Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com>

chtruong814 · 2026-04-09T20:22:52Z

/claude review

claude

LGTM

Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com>

…on agentic steps (#179) * docs: Swap our super image (#128) * docs: Swap our super image Signed-off-by: oliver könig <okoenig@nvidia.com> * also sft Signed-off-by: oliver könig <okoenig@nvidia.com> --------- Signed-off-by: oliver könig <okoenig@nvidia.com> * Add Spark Deployment Guide (#129) * Add Spark Deployment Guide Signed-off-by: chrisalexiuk-nvidia <calexiuk@nvidia.com> * Nemotron cc docs (#95) * update readme and docs Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * add substring dedup to docs Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Fix doc build errors and readme Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add missing links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * more missing links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add more missing links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Apply suggestions from code review Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add note about sdg models Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> * Update Spark Deployment Configs for Nemotron 3 Super (#131) * Update Spark Configs Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Add embedding recipe: build domain-specific embeddings from raw documents (#85) * Add embedding recipe for fine-tuning, evaluation, and deployment Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: Add embedding recipe documentation to docs site Move detailed documentation from the recipe README into docs/nemotron/embed/ to follow the nano3/super3 pattern. Add grid card and toctree entry in docs/index.md. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Move sample corpus to HuggingFace with hf:// URI support Remove bundled sample data from the repo and download it on demand from HuggingFace (nvidia/Retrieval-Synthetic-NVDocs-v1). The SDG stage now supports hf:// URIs in corpus_dir config, e.g.: hf://nvidia/Retrieval-Synthetic-NVDocs-v1@<sha>/sample_corpus/nv_pp_random This keeps the repo lightweight while preserving zero-config quick start — the default config auto-downloads the sample corpus on first run. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Fix embed recipe tests: correct CLI module paths and required fields - conftest.py: Fix cli_module paths from nemotron.cli.embed.* to nemotron.cli.commands.embed.* to match actual module locations - test_config_models.py: Provide required sdg_input_path for DataPrepConfig tests that construct with defaults - test_module_exports.py: Update expected exports to match actual API (SCRIPT_PATH, SPEC, META instead of SCRIPT_LOCAL, CONFIG_DIR, etc.) Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: Fix embed model specs (dimension 2048, max seq len 8192) Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: Clarify corpus file formats are configurable, not restricted Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * fix(embed): use python entrypoint consistently and remove dead variable - finetune.py: use entrypoint='python' (was 'python3') to match all other embed commands and the nano3/super3 pattern - sdg.py: remove unused for_remote variable Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * fix(embed): fix file handle leak, export patch, unused import, and docs - stage1_data_prep: use context manager for open() to avoid file handle leak - stage4_export: remove unused `from functools import partial` inside export_to_onnx - stage4_export: restore torch.onnx.export after patching using try/finally - docs: update corpus_dir default in README to match actual default.yaml HF URI Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Track uv.lock files and revert eval monkey-patch - Remove uv.lock from .gitignore and track stage lock files for reproducible builds - Revert AutoTokenizer monkey-patch in eval stage, no longer needed after upstream checkpoint fix (rope_theta added to rope_scaling) Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs(embed): add pre-generated dataset section from embed-finetune-recipe - Add 'Using NVIDIA's Pre-Generated Dataset' section showing how to skip Stage 0 using nvidia/Retrieval-Synthetic-NVDocs-v1 - Add dataset link to Further Reading Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * chore(embed): bump sample corpus revision to 1c0d185 Update hf:// URI revision for Retrieval-Synthetic-NVDocs-v1 in the default SDG config and the corresponding test. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> --------- Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Remove Nemotron 3 Super Advanced Deployment Guide (#133) * Remove Advanced Deployment Guide and Reference to Guide from README.md Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * ci: Add claude review to nemotron (#138) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update vLLM to pinned version 0.18.1 (#135) Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Move vllm version to nightly (#140) * Update vLLM to pinned version 0.18.1 Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Update vLLM to pinned version nightly Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Readd Language For Nightly Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Revert Version to Nightly and Re-Add Nightly language Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> --------- Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Expand embedding recipe README with comprehensive documentation (#134) * Expand embedding recipe README with comprehensive documentation Add installation guide, corpus preparation instructions, execution profiles, detailed stage descriptions, FAQ, and troubleshooting sections to make the embed recipe self-contained and user-friendly. Signed-off-by: Steve Han <sthan@nvidia.com> Made-with: Cursor * update docs readme Signed-off-by: Steve Han <sthan@nvidia.com> * fix docs build warning Signed-off-by: Steve Han <sthan@nvidia.com> * fix review comment Signed-off-by: Steve Han <sthan@nvidia.com> --------- Signed-off-by: Steve Han <sthan@nvidia.com> Co-authored-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * fix(super3): add missing data subcommand package (closes #100) (#145) Adds the complete `data` subpackage for the super3 CLI command, mirroring the existing nano3 structure. Includes `data prep` (pretrain/sft/rl) and `data import` (pretrain/sft/rl) subcommands wired to the super3 recipe scripts. Also fixes .gitignore negation paths that used the wrong directory level and adds the super3/data exclusion. https://claude.ai/code/session_01N723md8zykwcTpwyUsAw1m Signed-off-by: Doondi-Ashlesh <doondiashlesh@gmail.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> * Fix retriever SDG dedup plugin circular import (#156) * Fix retriever SDG dedup plugin initialization Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Install vendored retriever-sdg as editable Ensures source edits under vendor/retriever-sdg/ are picked up on the next run without rebuilding the cached wheel. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> --------- Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Fix embedding SDG dedup compatibility. (#147) * Fix embedding SDG dedup compatibility. Use Data Designer's public embedding facade API so Stage 0 dedup continues working after the private router interface change. Made-with: Cursor Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> * Pin data-designer for SDG dedup compatibility. Require data-designer>=0.5.3 so the retriever-sdg dependency floor matches the ModelFacade embedding API change used by the dedup fix. Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> Made-with: Cursor Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> --------- Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> Co-authored-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: add SECURITY.md (#160) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * fix(super3): point SFT HF conversion at Nemotron-3-Super base (#158) * fix(super3): point SFT HF conversion at Nemotron-3-Super base The SFT convert_to_hf step was pointed at the previous-generation Llama-3_3-Nemotron-Super-49B-v1 as a placeholder while the real HF repo was unpublished, which would have yielded a misconfigured export (wrong tokenizer/architecture) for the Nemotron 3 Super MoE/Mamba hybrid. Swap it to the now-published NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16 and link all four published HF model repos from the Super3 README. Signed-off-by: Shashank Verma <shashankv@nvidia.com> * docs(super3): link published Nemotron 3 Super Technical Report Replace all five "Tech Report (coming soon)" references across the Super3 docs with the published PDF at research.nvidia.com. Signed-off-by: Shashank Verma <shashankv@nvidia.com> --------- Signed-off-by: Shashank Verma <shashankv@nvidia.com> * fix(embed): propagate query/passage_max_length to hard negative mining (#163) The stage1_data_prep recipe accepts query_max_length and passage_max_length via the YAML config and CLI overrides, but run_mining() did not forward these values to the mine_hard_negatives.py subprocess. As a result, the mining step always fell back to the defaults (512) baked into scripts/mining_config.yaml, regardless of what the user configured. Add --mining.query_max_length and --mining.passage_max_length to the subprocess command so user overrides propagate correctly. Made-with: Cursor Made-with: Cursor Signed-off-by: Steve Han <sthan@nvidia.com> * Add Nemotron 3 Nano Omni Document Intelligence cookbook (#169) Self-contained notebook pairing Nemotron Parse with Nemotron 3 Nano Omni on the hosted catalog endpoint (https://integrate.api.nvidia.com/v1) for an end-to-end document AI pipeline: layout extraction, per-picture transcription, and multi-page reasoning. Runs on a free NVIDIA_API_KEY -- no GPU, no Docker, no local model weights required. Files added under usage-cookbook/Nemotron-3-Nano-Omni/doc-intelligence-with-parse/: - doc_intelligence_cookbook.ipynb (20 cells, end-to-end runnable) - README.md (model overview, requirements, uv-based quick start) - .gitignore (auto-downloaded PDFs and per-page artefacts) Signed-off-by: Chia-Chih Chen <chiachihc@nvidia.com> * Add Nemotron-3-Nano Omni deployment cookbooks (#172) Signed-off-by: Anusha Pant <apant@nvidia.com> * docs(automodel): add Nemotron-Nano-Omni V3 fine-tuning cookbook (#170) Add the end-to-end NeMo AutoModel fine-tuning cookbook for Nemotron-Nano-Omni (V3, 30B-A3B-Reasoning) on CORD-v2 receipts. Covers full SFT and LoRA PEFT, environment setup, dataset exploration, training, and inference. This sits next to the existing GRPO assets under usage-cookbook/Nemotron-3-Nano-Omni/ as the AutoModel-side counterpart. Source: docs/guides/vlm/nemotron-omni.md in NVIDIA-NeMo/Automodel. Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Nemotron Nano Omni V3 RL assets (#168) * adding omni-v3 RL assets * fix bash cells * remove bash magic * adding omni v3 sft cookbook (#171) * Adding Omni-3 recipes (#173) * refactor(recipes): move data curation under data tree Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * refactor(runspec): support archive container schemes Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): scaffold build command and app Keep the _smoke_test_stage placeholder in-tree for wave-1 CLI wiring smoke coverage; replace or remove it in the next wave when the real omni3 stages land. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(data/sdg): add long-document SDG pipeline scaffold Add the long-document SDG recipe directory with a full README and nine script stubs.\n\nThe upstream GitLab recipe source currently redirects to sign-in from this environment, so the script bodies remain explicit release-time TODO stubs for now. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add sft training path Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add sft model lifecycle commands Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add rl stages (mpo, text, vision) Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add eval stage Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * docs(omni3): add family docs + update cli/nemo-run references Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add pipe command Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * fix(omni3): expand $USER in container paths via OmegaConf Replaces 19 occurrences of the literal $USER in OCI-archive container paths with ${oc.env:USER}. OmegaConf does not expand shell variables, so the literal was being passed through to ensure_squashed_image and would fail to resolve the archive path on any real cluster. Spotted across: - stage0_sft configs (default, tiny, audio_text, image_text_sft, image_text_peft, peft_valor32k) and README.md - stage1_rl sub-stages (mpo, text, vision) configs + train.py runspec comments - stage2_eval default config No functional change for any config that was previously overriding the container field; only fixes the out-of-the-box default path resolution. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * fix(omni3): P0 stub allocation + multi-node torchrun rendezvous Two critical fixes before the branch can survive real cluster usage: 1. stage3_vision_rl/train.py runspec declared nodes=16, gpus_per_node=8 on a stub that raises NotImplementedError. A stray `nemotron omni3 rl vision --batch` would allocate 128 GPUs, wait, and then crash. Dropped to nodes=1, gpus_per_node=0; the real footprint gets restored when the upstream launcher lands alongside the real main() body. 2. stage0_sft/train.py invoked torchrun with only --nproc-per-node, so a multi-node run (`nodes=2` in the runspec) launched two independent torchrun processes that never rendezvoused — each spanning its own 8 GPUs, no cross-node collectives. Now reads SLURM_NNODES/SLURM_NODEID and resolves the first SLURM_STEP_NODELIST hostname into MASTER_ADDR, passing --nnodes / --node-rank / --rdzv-endpoint through to torchrun. Single-node runs fall back to the static default. Emits a warning when nnodes>1 but MASTER_ADDR can't be resolved. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * fix(omni3): pipe stub gating + data-prep config clobber and tempfile leak Three fixes that turn "works on a happy-path stable SSH session" into "survives real operational conditions": 1. omni3/pipe.py now auto-detects the vision stub via its degenerate resource footprint (gpus_per_node=0, set in the previous commit) and skips stage 4 by default, surfacing omni3-rl-text-model:latest as the final usable artifact. Users can pass `pipe.force_vision=true` via dotlist to override once the upstream launcher lands and the runspec footprint has been bumped back. Previously, `omni3 pipe --run` burned through stages 1-3 (days of compute) before crashing at stage 4's NotImplementedError. 2. omni3/data/prep/{sft,rl}.py used $CWD/config.yaml as a staging path for the resolved recipe config. That silently overwrote whatever the user happened to have at that path — a data-loss foot-gun for anyone iterating on config.yaml in the repo root. Now stages under a job-scoped filename (`.nemotron-data-prep-<job>.yaml`). The remote path inside the Ray workdir stays `config.yaml` so the `cmd` template is unchanged. 3. Same two files leaked one `NamedTemporaryFile(delete=False)` per invocation in /tmp for the Ray runtime_env yaml. Both that file and the new job-scoped config copy are now cleaned up in a try/finally around ray_job.start + tunnel.put. Test updates: - test_pipe_dry_run_succeeds now asserts the stub-skip path (rl text as final artifact). - test_pipe_dry_run_force_vision covers the override path. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * chore(gitignore): re-ignore __pycache__ under un-ignored data trees The broad `!src/nemotron/cli/commands/omni3/data/**` and `!src/nemotron/recipes/data/**` exceptions needed to defeat the top-level `data/` rule also un-masked `__pycache__/` directories inside those trees. Four `.pyc` files leaked into the previous commit. Re-ignore explicitly and remove the tracked artifacts. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * refactor(runspec): extract build-context precedence into shared helpers Six sites previously inlined the same `build_partition > run_partition > partition` / `build_time > time` / `build_image > default` precedence chain (flagged across the oracle review of commits 2, 3, 5, and 6 — the single largest duplication on the omni3 branch). Consolidate into four public helpers in `nemo_runspec.squash`: - resolve_build_partition(env) -> str | None - resolve_build_time(env, default) -> str - resolve_build_image(env, default) -> str - build_salloc_args(env, **opts) -> list[str] (salloc argv shape) Migrate call sites: - nemo_runspec.squash.ensure_squashed_image (was 20 lines, now 1) - nemotron.cli.kit.squash (was 20 lines, now 1) - nemotron.cli.commands.omni3.build (two display/executor sites, both now use the three resolve_* helpers directly since the SlurmExecutor kwargs differ from salloc argv) Behavior is unchanged: the partition/time precedence and defaults match what was written inline in each site. Adds 13 unit tests covering the helpers' precedence rules; existing 11 scheme-passthrough tests keep passing. Note: the data-prep Ray submissions (omni3/data/prep/{sft,rl}.py) and the model lifecycle helper (omni3/model/_base.py) use a different, mode-driven precedence (run_partition vs batch_partition depending on attached mode) and are intentionally not touched by this commit. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * docs(omni3): add Known Limitations sections to family docs Oracle review of commit 9 flagged that the omni3 family docs described the design-doc state of the branch rather than the shipped state — the vision RL stub, RL Dockerfile placeholder, SFT data-prep staging scaffold, SDG script stubs, text-only eval default, and upstream-private branch resolution were either glossed over or absent entirely. Fix: - README.md: prominent "Current Limitations" section listing each stub/placeholder with the exact path, the auto-detect/override mechanism where one exists, and a pointer to the per-stage guide. - sft.md: blockquote callout covering data-prep staging scaffold + the pending upstream dev/nomni branch. - rl.md: blockquote callout covering the vision stub auto-skip in pipe, the Dockerfile placeholder body, and the pending nano-v3-omni-recipes branch. - evaluate.md: blockquote callout explaining text-only default task list + how to override the default model artifact while vision is stubbed. No code changes. The rl.md vision-stub acknowledgment that already existed is preserved and expanded. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * refactor(omni3/sft): honor PEP 723 cmd, let nemo-run handle torchrun The previous shape had three problems the user surfaced: 1. `uv run` leaked into the PEP 723 cmd template — that's a Dockerfile implementation detail (we use uv to manage the venv), not part of the script's declared interface. 2. `train.py` hand-rolled a subprocess dance that reconstructed the torchrun command, duplicated SLURM_NNODES/SLURM_NODEID parsing, and flattened OmegaConf overrides back to argv — work that nemo-run's Torchrun launcher already does. 3. `train.py` ran in the base image's Python while the subprocess ran in the uv venv — a split that made imports fragile. Split concerns cleanly: **Dockerfile** (`stage0_sft/Dockerfile`): - `ENV PATH=/workspace/Megatron-Bridge/.venv/bin:$PATH` makes the uv-synced venv the default Python environment. Downstream tooling no longer needs to know `uv` was how it got built. - Drop `uv run` prefix from the sanity-check RUN lines. **Runspec** (`stage0_sft/train.py` PEP 723): - `launch = "torchrun"` (was `"direct"`) — nemo-run now wraps with torchrun and injects --nproc-per-node / --nnodes / --node-rank / the rendezvous flags from SLURM env vars automatically. - `cmd = "python {script} --recipe {recipe} --step_func {step_func} --config {config}"` — no `uv run`, matches nano3's declarative style. **train.py body** — collapsed to a ~15-line `os.execvp` forwarder that accepts the CLI flags and hands them to `/workspace/Megatron-Bridge/scripts/training/run_recipe.py`. Deletes `_build_command`, `_torchrun_rendezvous_args`, and `_flatten_overrides` (~90 lines). No nemotron.* imports so it runs cleanly in the uv venv. **CLI** (`commands/omni3/sft.py`): - Reads `recipe.name` / `recipe.step_func` from the compiled job config and passes them as `--recipe` / `--step_func` args alongside `--config REMOTE_CONFIG`. - `torchrun=(SPEC.run.launch == "torchrun")` — honors the runspec's launch mode for local execution too (was hardcoded `False`). Net effect: the hand-rolled multi-node rendezvous logic from the previous fix commit becomes unnecessary (nemo-run's Torchrun launcher does it); the `uv run` / base-venv-vs-uv-venv split disappears; the runspec is portable across dependency-management tools (swap uv for conda tomorrow: only the Dockerfile changes). Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3/sft): open-dataset default via CORD-v2, Valor32k as opt-in The shipped default for Omni SFT pointed at Valor32k-AVQA, an NVIDIA-internal Energon dataset — so a fresh user running `nemotron omni3 sft --run <cluster>` out of the box hit `Energon dataset not found at /datasets/valor32k/energon` with no path to recovery. Flipping the default makes the recipe runnable from open data on a public clone. Config swaps (explicit git mv so history follows): - default.yaml → valor32k.yaml (internal Energon SFT, same contents; added header documenting OMNI3_VALOR32K_ENERGON_PATH) - image_text_sft.yaml → default.yaml (CORD-v2 via HF `vlm-hf` loader; train_iters bumped 20 → 1000; save path aligned to /nemo_run/omni3-sft-model; header documents the projector-only trade-off per QA guide §5.2.2) tiny.yaml repointed at CORD-v2 so `-c tiny` smoke tests work without internal dataset access. Same for data_prep/tiny.yaml. data_prep flow now supports both modes: - hf_dataset_id set → manifest-only (training container pulls from Hub) - dataset_path set → Energon validation (existing behavior) - both → ValueError - neither → ValueError pointing at the two config exemplars data_prep.py: - Omni3EnergonDataPrepConfig gains optional hf_dataset_id; dataset_path is now Optional[Path] so callers can choose the flow via which field they set. - run_data_prep dispatches to _run_hf_prep or _run_energon_prep; both emit a manifest.json carrying `source: huggingface|energon` for downstream artifact lineage. - PEP 723 runspec default flipped "valor32k" → "default" so `omni3 data prep sft` picks the HF config by default. New config: data_prep/default.yaml (CORD-v2 HF flow). Docs: sft.md and README.md Known Limitations updated to reflect the open default and the Valor32k opt-in. Stays honest about the projector-only constraint on single-node CORD-v2. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Adding omni 3 recipes Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Update rl recipe and remove eval Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Fixing some docs issues Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Fixing some docs issues Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> --------- Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Update TRT-LLM setup for Nemotron-3-Nano-Omni (#175) * Add support for Lepton and DGX Cloud executors (#162) * Add Lepton and DGX Cloud executor support for nano3/super3 recipes Extends Nemotron executor layer to run on NVIDIA DGX Cloud Lepton and DGX Cloud (run:ai) alongside the existing Slurm path, including multi-node Ray-based RL (GRPO) via nemo-run's unified RayCluster + RayJob classes. Changes ------- * nemo-run pinned to the PR #480 build of rapaul-nv/Run (adds DGXCloudRayJob / LeptonRayCluster backends + client_credentials auth). * New nemo_runspec.execution: - execute_cloud : inline Script submission for non-Ray cloud jobs (data prep, torchrun-based SFT/pretrain). - execute_cloud_ray: RayCluster + RayJob path for launch="ray" recipes; used by nano3/super3 RL commands when on Lepton or DGX Cloud with nodes > 1. - _create_lepton_executor / _create_dgxcloud_executor factories, aligned with nemo-run 0.10 (client_id/client_secret, kube_apiserver_url, ray_version). - get_executor_type helper shared across all CLI command dispatchers. * CLI commands (nano3 + super3 sft/pretrain/rl + data/prep) route cloud jobs through the new execute_cloud / execute_cloud_ray paths. * env.toml profiles: [lepton], [lepton_gcp], [lepton_sft*], [lepton_rl], [dgxcloud] plus a shared [wandb] section. * Pipeline config accepts "dgxcloud"/"lepton" literals. * Tests: tests/nemo_runspec/test_execution.py covers executor creation (local/docker/slurm/lepton/dgxcloud), OmegaConf <-> plain-dict conversion, cloud workspace derivation, and git-mount command generation. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Ship local source to cloud pods without git install Airgap-friendly source transport for Lepton and DGX Cloud executors: local src/ is tarballed and delivered over the same Job API the launcher already uses, replacing the old `pip install nemotron @ git+…` step. - Lepton: chunk tarball across env vars (envp bypasses the 128 KiB argv MAX_ARG_STRLEN). NODE_RANK=0 extracts on shared NFS; other pods wait on a chunk-count-suffixed marker file to avoid the multi-pod rm-rf-then-tar race. - DGX Cloud (run:AI): drop tarball into job_dir as one `.tgz` file so move_data chunks only that file — torchrun_job.sh stays small. Env-var chunks would otherwise blow up the per-file deploy count. - Fallback (Slurm / others): nemo-run native packager extraction. Infrastructure: - New nemo_runspec.data_mover module with SourcePackager, Plan, plan_for. - run.py: patch_cloud_data_mover_skip_configs() excludes configs/ from nemo-run's inline-base64 tarball on both Lepton and DGXCloud, keeping the data-mover command under the kernel's MAX_ARG_STRLEN. - execute_cloud / execute_cloud_ray dispatch through a single plan_for() call; removed ~250 lines of branching packager/env-var/flag code. - env.toml knobs: `repo_root` (repo override), `dgxcloud_max_args_chars` (tune run:AI per-workload Args budget, default 9500). Tests: tests/nemo_runspec/test_data_mover.py covers include scoping, pycache filtering, all three transport branches, and the NODE_RANK gate. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Refactor Slurm executor construction and fix auto-mount behavior Slurm, Lepton, and DGX Cloud now share the same factory pattern. The previous ``create_executor`` had ~80 lines of inline Slurm setup while Lepton and DGX Cloud already used dedicated helpers. Five CLI commands also maintained near-identical copies of ``_build_slurm_executor``. Changes ------- * Extract ``create_slurm_executor`` as a public helper (with a ``launcher`` kwarg so Ray-based flows can pass ``None``). Also extract ``_create_local_executor`` and ``_create_docker_executor``. * Add ``_resolve_container_image`` and ``_resolve_nodes_gpus`` shared helpers that replace the three duplicated resolution blocks. * ``create_executor`` is now a small dispatch function. * Accept an ``identity`` field in env.toml so the SSH tunnel can be given an explicit private-key path when the agent isn't available. Bug fix for Slurm auto-mounts ----------------------------- On Slurm, ``${auto_mount:git+...}`` entries in a recipe config only register the requested repo into the ``get_git_mounts()`` registry at the moment OmegaConf resolves them. The previous code cloned the repos *before* the mounts list was accessed, so the registry was empty and the resulting sbatch script was missing the bind mounts. The container then silently fell back to the pre-baked Megatron-LM / Megatron-Bridge versions, causing training to run against the wrong code. ``_create_slurm_executor`` now reads ``env.mounts`` before cloning, which triggers resolution and populates the registry. CLI commands ------------ The following commands no longer carry their own Slurm executor setup and delegate to ``create_slurm_executor`` (with ``launcher=None`` for Ray flows): * ``nano3/data/prep/sft.py`` * ``nano3/data/prep/rl.py`` * ``nano3/data/prep/pretrain.py`` * ``nano3/rl.py`` * ``super3/rl/_base.py`` (persistent_cache and sif_dir extras applied on top of the returned executor) Misc ---- * Fix stale ``.gitignore`` whitelist: the CLI data directory moved from ``src/nemotron/cli/nano3/data/`` to ``src/nemotron/cli/commands/nano3/data/`` but the override entry was never updated, so the files were silently ignored. Verification ------------ * 131/131 unit tests pass. * Lint clean on all touched files. * End-to-end Slurm SFT data prep succeeds on CW DFW CS-001 (2 nodes, 10k samples, 47M tokens). Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Update dependencies and improve data mover functionality - Updated the `nemo-run` dependency to point to the main branch of the NVIDIA-NeMo repository. - Enhanced the `SourcePackager` class to streamline tarball creation for job directories. - Refactored the `plan_for` function to unify environment variable chunking for Lepton and DGX Cloud executors, improving data transport efficiency. - Introduced new patches to handle legacy keyword arguments and optimize environment variable handling in DGX Cloud. - Added a manifest resolver to facilitate configuration management for data preparation outputs. This commit ensures better compatibility with upstream changes and enhances the overall performance of the data mover functionality across different execution environments. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Enhance data preparation for cloud execution in super3 recipes - Added support for executing data preparation commands (pretrain, rl, sft) on DGX Cloud and Lepton using the new `execute_cloud` function. - Implemented logic to handle single-node CPU work for cloud environments, ensuring compatibility with the existing data preparation workflow. - Introduced functions to create train/val splits and update the manifest with per-split paths in the RL data preparation process. This update improves the flexibility and efficiency of data preparation across different execution environments. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * fix(nemo_runspec): address PR #162 review comments - docs: document required kube_apiserver_url; promote client_id/client_secret as primary auth fields with app_id/app_secret as legacy aliases - run.py: use argv list form for tar to avoid shell-metachar issues in nemo_run_dir paths - execution.py: honor explicit gpus_per_node=0 when picking nproc - execution.py: rename shadowed local launch -> launch_cmd - execution.py: narrow cluster.start except to "already exists" idempotency, re-raise auth/quota/image-pull failures Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Rakesh Paul <rapaul@nvidia.com> --------- Signed-off-by: Rakesh Paul <rapaul@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Nemotron 3 Nano Omni deployment cookbooks (#181) * Add Nemotron-3-Nano Omni deployment cookbooks Signed-off-by: Anusha Pant <apant@nvidia.com> * Update TRT-LLM setup for Nemotron-3-Nano-Omni Signed-off-by: Anusha Pant <apant@nvidia.com> * Update TRT-LLM setup for Nemotron-3-Nano-Omni Signed-off-by: Anusha Pant <apant@nvidia.com> * Update Nemotron-3-Nano-Omni deployment cookbooks Signed-off-by: Anusha Pant <apant@nvidia.com> --------- Signed-off-by: Anusha Pant <apant@nvidia.com> * First pass over nemotron skills Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Splitting up skill Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(steps): add Nemotron steps for SFT, PEFT, RL, pretraining, SDG, and optimization This commit brings the following capabilities into the Nemotron steps: - SFT and PEFT workflows with AutoModel and Megatron Bridge runners. - RL workflows for DPO, RLHF, RLVR, and NeMo Gym-style GRPO execution. - Pretraining workflows with AutoModel and Megatron Bridge support. - Data preparation steps for SFT packing, pretraining, and RL data. - SDG workflows through the Data Designer step. - ModelOpt optimization steps for quantization, pruning, and distillation. - Evaluation, conversion, curation, and translation steps under the same discoverable execution model. - A common `nemotron step list/show/run` CLI for step discovery and launch. - Backend-aware execution for local, Slurm, and Lepton/cloud job submission. - Portable code packaging so remote jobs can run from packaged step sources. - Standardized `default` and `tiny` configs for fast bring-up and validation. - NVIDIA-backed tiny data blends for SFT and pretraining data prep. - Improved `nemo_runspec` execution and data movement support. - Updated docs, skills, and tests for extending and operating the step system. - Standalone data-prep steps now use the regular job backend path instead of assuming RayCluster provisioning. Validation: - SFT data prep successfully tested on Lepton. - SFT training submission successfully tested on Lepton. - DPO submission successfully tested on Lepton. - AutoModel SFT test successfully completed on Lepton. - Pretraining data prep successfully tested on Lepton. - Added/updated step test suite is passing with 100% pass rate. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * refactor(modelopt): standardize argument handling and improve configuration structure - Transitioning to a unified `args:` section in YAML configurations for upstream script arguments, enhancing clarity and usability. - Updating the Python scripts to accommodate the new argument structure, ensuring backward compatibility with legacy flat configurations. - Enhancing documentation to reflect the new configuration format and usage patterns. - Adding tests to validate the new argument mapping and ensure correct CLI flag translation. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(wandb): enhance W&B integration across various components - Added support for additional W&B environment variables such as WANDB_ENABLED, WANDB_GROUP, WANDB_JOB_TYPE, and WANDB_TAGS in the execution environment. - Updated the `init_wandb_from_env` function to utilize new W&B variables for improved logging and tracking. - Refactored YAML configuration files to pull W&B settings from environment variables, ensuring consistency and flexibility in project and entity naming. - Introduced a new function `init_prep_wandb` to initialize W&B with specific tags for data preparation steps, enhancing observability during the preparation phase. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(automodel): update configurations for Qwen3.6-35B-A3B model for demonstrating the support of non nemotron models using automodel - Enhanced step.toml and default.yaml files across PEFT, SFT, and pretraining workflows to include Qwen3.6-35B-A3B as the default model, optimizing for larger GPU setups. - Updated descriptions to clarify the use of LoRA adapters with qwen models for various training scenarios. - Ensured backward compatibility by maintaining existing model options while introducing the new larger model for improved performance. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(steps): add tiny configs for functional validation - Add low-scale tiny.yaml configs across prep, SFT, PEFT, pretrain, RL, synth, and ModelOpt steps with working values tested on A100 GPUs. - Align defaults with current launcher/runtime behavior, including NeMo-RL dataset schemas and policy defaults. - Update AutoModel configs for Qwen3 MoE settings. - Switch ModelOpt quantization configs to Nano-friendly recipes. - Use packaged SDG seed files and output paths. - Replace Megatron-Bridge packed-data paths with shared/env-driven locations. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * docs(steps): teach agents to use step metadata - Add step-level `SKILL.md` guidance across Nemotron stages - Instruct agents to read `step.toml` before changing configs or code - Document `step.toml` as the flow, artifact, parameter, strategy, failure-mode, and reference contract - Expand step and pattern docs for agentic navigation across prep, SFT, PEFT, pretrain, RL, synth, and ModelOpt workflows Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * `feat(runspec): harden Lepton and Ray step launches` - Add simple `defaults:` YAML inheritance for step configs - Preserve YAML and profile `env_vars` when merging env profiles - Support `NEMOTRON_ENV_FILE` for alternate env profile files - Redact sensitive config/env values in CLI display - Stage cloud source into unique per-run directories to avoid shared filesystem races - Keep `/nemo_run/code/src` valid when cloud source is staged elsewhere - Mask large transport env vars before Ray workers inherit them - Poll Ray job status instead of streaming logs for attached Ray launches - Add focused runspec tests for config inheritance, env override, transport, and Ray helpers Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * `feat(steps): harden functional step runtimes` - Add compatibility patches for AutoModel, Megatron-Bridge checkpointing, and NeMo-RL/NeMo-Gym API drift - Improve ModelOpt launch behavior for distributed runs, rank-0-only W&B logging, and safer optional checkout refresh - Emit pretrain prep `blend.json` split metadata for downstream pretraining steps - Update SFT, RL, pretrain, quantize, and Data Designer step entrypoints for current runtime behavior - Add RL prep and synthetic data seed files for tiny functional validation - Expand step tests for ModelOpt runner behavior, Data Designer schemas/results, and updated step contracts Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Add env profile generator and guidance Add env/env_toml for generating Lepton and Slurm profile examples as env.lepton.toml and env.slurm.toml, preserving any existing root env.toml. Include step-linked profiles for prep, pretrain, SFT, RL, and ModelOpt quantization, plus guidance for NEMOTRON_ENV_FILE, force=true, inheritance, runtime env vars, Ray placement, W&B, and site-private placeholders. Update step skills to make env profile setup the first check for non-local Lepton, Slurm, Ray, and batch runs. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Make data-prep env profiles CPU-only - Set Lepton and Slurm data prep profiles to request CPU resources. - Add Slurm CPU build/import settings for data-prep profiles. - Document the data-prep CPU-only convention in env skills. - Allow squash builds to opt into GPU allocations with build_include_gpus and build_gpus_per_node. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * fix: address PR review comments and harden staged steps - fix PR review findings around AutoModel model defaults, SFT/PEFT contracts, placeholder paths, and pretrain backend docs - update NeMo-RL DPO/RLVR/RLHF configs for v0.6 runtime and NeMo-Gym contracts - harden ModelOpt args precedence, W&B failure reporting, setup timeouts, and prune compatibility patching - bound cloud source-transport marker waits and fail fast on extraction failures - improve env profile merging, cloud script path handling, and Data Designer dataset creation - make Data Designer tool-chat generation parquet-safe - generate customer-support tool conversations as JSON text instead of nested structured columns - add tests for review fixes, RL configs, cloud paths, source transport, config coherence, and synth JSON handling Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * fix: address review comments for SDG Data Designer - rename the Data Designer step namespace from synth to sdg across steps, skills, docs, and tests - make customer-support tool-chat generation parquet-safe by avoiding escaped nested JSON in model output - serialize generated tool arguments and tool responses back to OpenAI-compatible string fields during projection - keep malformed generated rows skippable while failing when the full batch is invalid - add Data Designer tests for JSON parsing, tool payload normalization, and bad-row handling Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * refactor(skills, steps, patterns): one home per fact, sovereign-AI-focused catalog Make skills/nemotron-customize/ a thin orchestrator over src/nemotron/steps/. Realign the pattern catalog around the sovereign-AI customization journey (curate → SDG → prep → CPT/SFT/PEFT → RL → convert → eval). Strengthen every category and per-step SKILL with consistent structure. What changed - Customize SKILL.md rewritten around a numbered four-phase workflow (Orient → Plan → Act → Verify) with CLI-first discovery; act/PROJECT.md and act/STAGE.md tightened with concrete pointers into _runners/. - 10 raw upstream-doc dumps (~50K lines, regeneratable) deleted from context/. 5 authored extracts remain (≤250 lines each), grounded in actual step.toml + step.py + config defaults: automodel-pretrain, data-designer-sdg, modelopt-optimization, nemo-rl-alignment, nemotron-data-prep. - 11 step.toml files cleaned of dead `skills = [...]` lists and inline `skill = "..."` pointers to deleted packs. URLs in [reference] now carry the upstream pointers. - 5 missing per-step SKILL.md files added (eval/model_eval, curate/nemo_curator, translate/nemo_skills, plus a single category convert/SKILL.md covering the three thin I/O bridges). - prep/guide.md and sft/guide.md merged into their category SKILL.md. Pattern catalog - 14 patterns total (was 14). Refocused on sovereign-AI customization. - Removed: distill-after-structural-compression, representative-calibration-before-optimization, production-export-trt (compression / serving — not customization journey). - Added: cpt-data-blend-scoping, sft-data-blending, byob-benchmark-design. - Renamed for self-explanatory <topic>-<rule> form: eval-bookends → eval-before-and-after-training, pack-variable-length → sft-sequence-packing, small-dataset-lora → sft-small-dataset-prefer-lora, validate-rl-rewards-before-scale → rl-validate-rewards-before-scale, version-sdg-pipeline → sdg-pipeline-versioning, prepared-data-is-tokenizer-locked → prep-data-is-tokenizer-locked, checkpoint-before-convert → convert-checkpoint-safety, adapter-artifact-before-merge → peft-adapter-merge-discipline. - Quality-pass rewrites: convert-checkpoint-safety, pretrain-token-budget-before-scale, sdg-pipeline-versioning. - eval-before-and-after-training steps[] populated (was empty); every pattern now has populated cross-references. Category + per-step SKILLs - All 9 category SKILLs (prep, sft, peft, pretrain, rl, sdg, optimize + optimize/modelopt + rl/nemo_rl) follow the same structure: identity, steps table, decision tree, pipeline placement, pre-conditions, workflow, smoke commands, patterns to cite, guardrails. - Fixed peft/automodel/SKILL.md frontmatter typo ("suggest ---") that silently broke parsing. Verification - nv-base: 232 errors → 0 errors, 0 warnings. SCHEMA, SECURITY, PII, HYGIENE, UNICODE all pass. - discover_steps() = 23, discover_patterns() = 14, all step.toml valid. - All 32 SKILL.md files have valid YAML frontmatter. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * context(skills): add Megatron-Bridge SFT/PEFT pack, clarify AutoModel scope, fix Super3 model id - New pack mbridge-sft.txt covers sft/megatron_bridge + peft/megatron_bridge (runner flow, recipe selection, MoE knobs, packed-sequence rules, full vs LoRA). - AutoModel pack now explicitly states it's the path for non-Nemotron and non-MB-recipe models; MB stays for the curated recipe families. - Fix sft/megatron_bridge [[models]]: correct HF ids (NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, NVIDIA-Nemotron-3-Super-120B-A12B-BF16) and Super3 description (hybrid Mamba LatentMoE, not "dense 49B" — that conflated the older Llama-Nemotron-Super). - Wire both step.tomls + index.toml to the new pack. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(execution): add cloud script path handling for pod-local source - Introduced `_cloud_script_path` function to convert script paths for cloud pod compatibility, ensuring that file-based commands reference the correct pod-local paths. - Updated `execute_cloud` and `execute_cloud_ray` functions to utilize the new path handling, enhancing support for concurrent jobs and avoiding shared mutable symlinks. - Added unit tests for `_cloud_script_path` to verify correct path transformations based on source layout. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: chrisalexiuk-nvidia <calexiuk@nvidia.com> Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Steve Han <sthan@nvidia.com> Signed-off-by: Doondi-Ashlesh <doondiashlesh@gmail.com> Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> Signed-off-by: Shashank Verma <shashankv@nvidia.com> Signed-off-by: Chia-Chih Chen <chiachihc@nvidia.com> Signed-off-by: Anusha Pant <apant@nvidia.com> Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> Signed-off-by: Rakesh Paul <rapaul@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Chris Alexiuk <161380339+chrisalexiuk-nvidia@users.noreply.github.com> Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com> Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Co-authored-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Steve Han <150830061+shan-nvidia@users.noreply.github.com> Co-authored-by: Doondi Ashlesh <126656890+Doondi-Ashlesh@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: hoangnguyenN94 <hoannguyen@nvidia.com> Co-authored-by: Shashank Verma <shashankv@nvidia.com> Co-authored-by: Chia-Chih Chen <chiachihc@nvidia.com> Co-authored-by: Anusha Pant <67914344+anushapant@users.noreply.github.com> Co-authored-by: Huiying <willwin.lee@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Vinh Nguyen <vinhn@nvidia.com> Co-authored-by: Marc Romeyn <marcromeyn@gmail.com>

* First pass over nemotron skills Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Splitting up skill Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Byob agentic capabilities (#183) * Add agentic BYOB benchmark step Signed-off-by: rkalani <rkalani@nvidia.com> * Move BYOB manifest into step package Signed-off-by: rkalani <rkalani@nvidia.com> * Align BYOB step with contribution conventions Signed-off-by: rkalani <rkalani@nvidia.com> * Document BYOB step layout convention Signed-off-by: rkalani <rkalani@nvidia.com> * Add BYOB CLI command Signed-off-by: rkalani <rkalani@nvidia.com> * Add BYOB use case notebook Signed-off-by: rkalani <rkalani@nvidia.com> * Use Curator for BYOB translation * Use Curator experimental translation for BYOB * Update BYOB Curator dedup flow * Refactor BYOB family pipeline and dependencies --------- Signed-off-by: rkalani <rkalani@nvidia.com> * Fix Data Designer Bugs (#190) * Fix BYOB Data Designer 0.5.5 integration Signed-off-by: rkalani <rkalani@nvidia.com> * Remove BYOB asset-only test Signed-off-by: rkalani <rkalani@nvidia.com> --------- Signed-off-by: rkalani <rkalani@nvidia.com> * Add Curator based translation and skills (#180) * Add Curator translation agentic step Signed-off-by: rkalani <rkalani@nvidia.com> * Add translation planning patterns Signed-off-by: rkalani <rkalani@nvidia.com> * Add translation CLI command Signed-off-by: rkalani <rkalani@nvidia.com> --------- Signed-off-by: rkalani <rkalani@nvidia.com> Co-authored-by: rapaul-nv <rapaul@nvidia.com> * Add SFT, PEFT, RL, Pretraining, SDG, and Model Optimization to Nemotron agentic steps (#179) * docs: Swap our super image (#128) * docs: Swap our super image Signed-off-by: oliver könig <okoenig@nvidia.com> * also sft Signed-off-by: oliver könig <okoenig@nvidia.com> --------- Signed-off-by: oliver könig <okoenig@nvidia.com> * Add Spark Deployment Guide (#129) * Add Spark Deployment Guide Signed-off-by: chrisalexiuk-nvidia <calexiuk@nvidia.com> * Nemotron cc docs (#95) * update readme and docs Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * add substring dedup to docs Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Fix doc build errors and readme Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add missing links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * more missing links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add more missing links Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Apply suggestions from code review Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> * Add note about sdg models Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> --------- Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> * Update Spark Deployment Configs for Nemotron 3 Super (#131) * Update Spark Configs Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Add embedding recipe: build domain-specific embeddings from raw documents (#85) * Add embedding recipe for fine-tuning, evaluation, and deployment Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: Add embedding recipe documentation to docs site Move detailed documentation from the recipe README into docs/nemotron/embed/ to follow the nano3/super3 pattern. Add grid card and toctree entry in docs/index.md. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Move sample corpus to HuggingFace with hf:// URI support Remove bundled sample data from the repo and download it on demand from HuggingFace (nvidia/Retrieval-Synthetic-NVDocs-v1). The SDG stage now supports hf:// URIs in corpus_dir config, e.g.: hf://nvidia/Retrieval-Synthetic-NVDocs-v1@<sha>/sample_corpus/nv_pp_random This keeps the repo lightweight while preserving zero-config quick start — the default config auto-downloads the sample corpus on first run. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Fix embed recipe tests: correct CLI module paths and required fields - conftest.py: Fix cli_module paths from nemotron.cli.embed.* to nemotron.cli.commands.embed.* to match actual module locations - test_config_models.py: Provide required sdg_input_path for DataPrepConfig tests that construct with defaults - test_module_exports.py: Update expected exports to match actual API (SCRIPT_PATH, SPEC, META instead of SCRIPT_LOCAL, CONFIG_DIR, etc.) Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: Fix embed model specs (dimension 2048, max seq len 8192) Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: Clarify corpus file formats are configurable, not restricted Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * fix(embed): use python entrypoint consistently and remove dead variable - finetune.py: use entrypoint='python' (was 'python3') to match all other embed commands and the nano3/super3 pattern - sdg.py: remove unused for_remote variable Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * fix(embed): fix file handle leak, export patch, unused import, and docs - stage1_data_prep: use context manager for open() to avoid file handle leak - stage4_export: remove unused `from functools import partial` inside export_to_onnx - stage4_export: restore torch.onnx.export after patching using try/finally - docs: update corpus_dir default in README to match actual default.yaml HF URI Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Track uv.lock files and revert eval monkey-patch - Remove uv.lock from .gitignore and track stage lock files for reproducible builds - Revert AutoTokenizer monkey-patch in eval stage, no longer needed after upstream checkpoint fix (rope_theta added to rope_scaling) Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs(embed): add pre-generated dataset section from embed-finetune-recipe - Add 'Using NVIDIA's Pre-Generated Dataset' section showing how to skip Stage 0 using nvidia/Retrieval-Synthetic-NVDocs-v1 - Add dataset link to Further Reading Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * chore(embed): bump sample corpus revision to 1c0d185 Update hf:// URI revision for Retrieval-Synthetic-NVDocs-v1 in the default SDG config and the corresponding test. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> --------- Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Remove Nemotron 3 Super Advanced Deployment Guide (#133) * Remove Advanced Deployment Guide and Reference to Guide from README.md Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * ci: Add claude review to nemotron (#138) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * Update vLLM to pinned version 0.18.1 (#135) Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Move vllm version to nightly (#140) * Update vLLM to pinned version 0.18.1 Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Update vLLM to pinned version nightly Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Readd Language For Nightly Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Revert Version to Nightly and Re-Add Nightly language Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> --------- Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> * Expand embedding recipe README with comprehensive documentation (#134) * Expand embedding recipe README with comprehensive documentation Add installation guide, corpus preparation instructions, execution profiles, detailed stage descriptions, FAQ, and troubleshooting sections to make the embed recipe self-contained and user-friendly. Signed-off-by: Steve Han <sthan@nvidia.com> Made-with: Cursor * update docs readme Signed-off-by: Steve Han <sthan@nvidia.com> * fix docs build warning Signed-off-by: Steve Han <sthan@nvidia.com> * fix review comment Signed-off-by: Steve Han <sthan@nvidia.com> --------- Signed-off-by: Steve Han <sthan@nvidia.com> Co-authored-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * fix(super3): add missing data subcommand package (closes #100) (#145) Adds the complete `data` subpackage for the super3 CLI command, mirroring the existing nano3 structure. Includes `data prep` (pretrain/sft/rl) and `data import` (pretrain/sft/rl) subcommands wired to the super3 recipe scripts. Also fixes .gitignore negation paths that used the wrong directory level and adds the super3/data exclusion. https://claude.ai/code/session_01N723md8zykwcTpwyUsAw1m Signed-off-by: Doondi-Ashlesh <doondiashlesh@gmail.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> * Fix retriever SDG dedup plugin circular import (#156) * Fix retriever SDG dedup plugin initialization Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Install vendored retriever-sdg as editable Ensures source edits under vendor/retriever-sdg/ are picked up on the next run without rebuilding the cached wheel. Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> --------- Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Fix embedding SDG dedup compatibility. (#147) * Fix embedding SDG dedup compatibility. Use Data Designer's public embedding facade API so Stage 0 dedup continues working after the private router interface change. Made-with: Cursor Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> * Pin data-designer for SDG dedup compatibility. Require data-designer>=0.5.3 so the retriever-sdg dependency floor matches the ModelFacade embedding API change used by the dedup fix. Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> Made-with: Cursor Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> --------- Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> Co-authored-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * docs: add SECURITY.md (#160) Signed-off-by: Charlie Truong <chtruong@nvidia.com> * fix(super3): point SFT HF conversion at Nemotron-3-Super base (#158) * fix(super3): point SFT HF conversion at Nemotron-3-Super base The SFT convert_to_hf step was pointed at the previous-generation Llama-3_3-Nemotron-Super-49B-v1 as a placeholder while the real HF repo was unpublished, which would have yielded a misconfigured export (wrong tokenizer/architecture) for the Nemotron 3 Super MoE/Mamba hybrid. Swap it to the now-published NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16 and link all four published HF model repos from the Super3 README. Signed-off-by: Shashank Verma <shashankv@nvidia.com> * docs(super3): link published Nemotron 3 Super Technical Report Replace all five "Tech Report (coming soon)" references across the Super3 docs with the published PDF at research.nvidia.com. Signed-off-by: Shashank Verma <shashankv@nvidia.com> --------- Signed-off-by: Shashank Verma <shashankv@nvidia.com> * fix(embed): propagate query/passage_max_length to hard negative mining (#163) The stage1_data_prep recipe accepts query_max_length and passage_max_length via the YAML config and CLI overrides, but run_mining() did not forward these values to the mine_hard_negatives.py subprocess. As a result, the mining step always fell back to the defaults (512) baked into scripts/mining_config.yaml, regardless of what the user configured. Add --mining.query_max_length and --mining.passage_max_length to the subprocess command so user overrides propagate correctly. Made-with: Cursor Made-with: Cursor Signed-off-by: Steve Han <sthan@nvidia.com> * Add Nemotron 3 Nano Omni Document Intelligence cookbook (#169) Self-contained notebook pairing Nemotron Parse with Nemotron 3 Nano Omni on the hosted catalog endpoint (https://integrate.api.nvidia.com/v1) for an end-to-end document AI pipeline: layout extraction, per-picture transcription, and multi-page reasoning. Runs on a free NVIDIA_API_KEY -- no GPU, no Docker, no local model weights required. Files added under usage-cookbook/Nemotron-3-Nano-Omni/doc-intelligence-with-parse/: - doc_intelligence_cookbook.ipynb (20 cells, end-to-end runnable) - README.md (model overview, requirements, uv-based quick start) - .gitignore (auto-downloaded PDFs and per-page artefacts) Signed-off-by: Chia-Chih Chen <chiachihc@nvidia.com> * Add Nemotron-3-Nano Omni deployment cookbooks (#172) Signed-off-by: Anusha Pant <apant@nvidia.com> * docs(automodel): add Nemotron-Nano-Omni V3 fine-tuning cookbook (#170) Add the end-to-end NeMo AutoModel fine-tuning cookbook for Nemotron-Nano-Omni (V3, 30B-A3B-Reasoning) on CORD-v2 receipts. Covers full SFT and LoRA PEFT, environment setup, dataset exploration, training, and inference. This sits next to the existing GRPO assets under usage-cookbook/Nemotron-3-Nano-Omni/ as the AutoModel-side counterpart. Source: docs/guides/vlm/nemotron-omni.md in NVIDIA-NeMo/Automodel. Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Nemotron Nano Omni V3 RL assets (#168) * adding omni-v3 RL assets * fix bash cells * remove bash magic * adding omni v3 sft cookbook (#171) * Adding Omni-3 recipes (#173) * refactor(recipes): move data curation under data tree Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * refactor(runspec): support archive container schemes Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): scaffold build command and app Keep the _smoke_test_stage placeholder in-tree for wave-1 CLI wiring smoke coverage; replace or remove it in the next wave when the real omni3 stages land. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(data/sdg): add long-document SDG pipeline scaffold Add the long-document SDG recipe directory with a full README and nine script stubs.\n\nThe upstream GitLab recipe source currently redirects to sign-in from this environment, so the script bodies remain explicit release-time TODO stubs for now. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add sft training path Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add sft model lifecycle commands Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add rl stages (mpo, text, vision) Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add eval stage Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * docs(omni3): add family docs + update cli/nemo-run references Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3): add pipe command Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * fix(omni3): expand $USER in container paths via OmegaConf Replaces 19 occurrences of the literal $USER in OCI-archive container paths with ${oc.env:USER}. OmegaConf does not expand shell variables, so the literal was being passed through to ensure_squashed_image and would fail to resolve the archive path on any real cluster. Spotted across: - stage0_sft configs (default, tiny, audio_text, image_text_sft, image_text_peft, peft_valor32k) and README.md - stage1_rl sub-stages (mpo, text, vision) configs + train.py runspec comments - stage2_eval default config No functional change for any config that was previously overriding the container field; only fixes the out-of-the-box default path resolution. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * fix(omni3): P0 stub allocation + multi-node torchrun rendezvous Two critical fixes before the branch can survive real cluster usage: 1. stage3_vision_rl/train.py runspec declared nodes=16, gpus_per_node=8 on a stub that raises NotImplementedError. A stray `nemotron omni3 rl vision --batch` would allocate 128 GPUs, wait, and then crash. Dropped to nodes=1, gpus_per_node=0; the real footprint gets restored when the upstream launcher lands alongside the real main() body. 2. stage0_sft/train.py invoked torchrun with only --nproc-per-node, so a multi-node run (`nodes=2` in the runspec) launched two independent torchrun processes that never rendezvoused — each spanning its own 8 GPUs, no cross-node collectives. Now reads SLURM_NNODES/SLURM_NODEID and resolves the first SLURM_STEP_NODELIST hostname into MASTER_ADDR, passing --nnodes / --node-rank / --rdzv-endpoint through to torchrun. Single-node runs fall back to the static default. Emits a warning when nnodes>1 but MASTER_ADDR can't be resolved. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * fix(omni3): pipe stub gating + data-prep config clobber and tempfile leak Three fixes that turn "works on a happy-path stable SSH session" into "survives real operational conditions": 1. omni3/pipe.py now auto-detects the vision stub via its degenerate resource footprint (gpus_per_node=0, set in the previous commit) and skips stage 4 by default, surfacing omni3-rl-text-model:latest as the final usable artifact. Users can pass `pipe.force_vision=true` via dotlist to override once the upstream launcher lands and the runspec footprint has been bumped back. Previously, `omni3 pipe --run` burned through stages 1-3 (days of compute) before crashing at stage 4's NotImplementedError. 2. omni3/data/prep/{sft,rl}.py used $CWD/config.yaml as a staging path for the resolved recipe config. That silently overwrote whatever the user happened to have at that path — a data-loss foot-gun for anyone iterating on config.yaml in the repo root. Now stages under a job-scoped filename (`.nemotron-data-prep-<job>.yaml`). The remote path inside the Ray workdir stays `config.yaml` so the `cmd` template is unchanged. 3. Same two files leaked one `NamedTemporaryFile(delete=False)` per invocation in /tmp for the Ray runtime_env yaml. Both that file and the new job-scoped config copy are now cleaned up in a try/finally around ray_job.start + tunnel.put. Test updates: - test_pipe_dry_run_succeeds now asserts the stub-skip path (rl text as final artifact). - test_pipe_dry_run_force_vision covers the override path. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * chore(gitignore): re-ignore __pycache__ under un-ignored data trees The broad `!src/nemotron/cli/commands/omni3/data/**` and `!src/nemotron/recipes/data/**` exceptions needed to defeat the top-level `data/` rule also un-masked `__pycache__/` directories inside those trees. Four `.pyc` files leaked into the previous commit. Re-ignore explicitly and remove the tracked artifacts. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * refactor(runspec): extract build-context precedence into shared helpers Six sites previously inlined the same `build_partition > run_partition > partition` / `build_time > time` / `build_image > default` precedence chain (flagged across the oracle review of commits 2, 3, 5, and 6 — the single largest duplication on the omni3 branch). Consolidate into four public helpers in `nemo_runspec.squash`: - resolve_build_partition(env) -> str | None - resolve_build_time(env, default) -> str - resolve_build_image(env, default) -> str - build_salloc_args(env, **opts) -> list[str] (salloc argv shape) Migrate call sites: - nemo_runspec.squash.ensure_squashed_image (was 20 lines, now 1) - nemotron.cli.kit.squash (was 20 lines, now 1) - nemotron.cli.commands.omni3.build (two display/executor sites, both now use the three resolve_* helpers directly since the SlurmExecutor kwargs differ from salloc argv) Behavior is unchanged: the partition/time precedence and defaults match what was written inline in each site. Adds 13 unit tests covering the helpers' precedence rules; existing 11 scheme-passthrough tests keep passing. Note: the data-prep Ray submissions (omni3/data/prep/{sft,rl}.py) and the model lifecycle helper (omni3/model/_base.py) use a different, mode-driven precedence (run_partition vs batch_partition depending on attached mode) and are intentionally not touched by this commit. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * docs(omni3): add Known Limitations sections to family docs Oracle review of commit 9 flagged that the omni3 family docs described the design-doc state of the branch rather than the shipped state — the vision RL stub, RL Dockerfile placeholder, SFT data-prep staging scaffold, SDG script stubs, text-only eval default, and upstream-private branch resolution were either glossed over or absent entirely. Fix: - README.md: prominent "Current Limitations" section listing each stub/placeholder with the exact path, the auto-detect/override mechanism where one exists, and a pointer to the per-stage guide. - sft.md: blockquote callout covering data-prep staging scaffold + the pending upstream dev/nomni branch. - rl.md: blockquote callout covering the vision stub auto-skip in pipe, the Dockerfile placeholder body, and the pending nano-v3-omni-recipes branch. - evaluate.md: blockquote callout explaining text-only default task list + how to override the default model artifact while vision is stubbed. No code changes. The rl.md vision-stub acknowledgment that already existed is preserved and expanded. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * refactor(omni3/sft): honor PEP 723 cmd, let nemo-run handle torchrun The previous shape had three problems the user surfaced: 1. `uv run` leaked into the PEP 723 cmd template — that's a Dockerfile implementation detail (we use uv to manage the venv), not part of the script's declared interface. 2. `train.py` hand-rolled a subprocess dance that reconstructed the torchrun command, duplicated SLURM_NNODES/SLURM_NODEID parsing, and flattened OmegaConf overrides back to argv — work that nemo-run's Torchrun launcher already does. 3. `train.py` ran in the base image's Python while the subprocess ran in the uv venv — a split that made imports fragile. Split concerns cleanly: **Dockerfile** (`stage0_sft/Dockerfile`): - `ENV PATH=/workspace/Megatron-Bridge/.venv/bin:$PATH` makes the uv-synced venv the default Python environment. Downstream tooling no longer needs to know `uv` was how it got built. - Drop `uv run` prefix from the sanity-check RUN lines. **Runspec** (`stage0_sft/train.py` PEP 723): - `launch = "torchrun"` (was `"direct"`) — nemo-run now wraps with torchrun and injects --nproc-per-node / --nnodes / --node-rank / the rendezvous flags from SLURM env vars automatically. - `cmd = "python {script} --recipe {recipe} --step_func {step_func} --config {config}"` — no `uv run`, matches nano3's declarative style. **train.py body** — collapsed to a ~15-line `os.execvp` forwarder that accepts the CLI flags and hands them to `/workspace/Megatron-Bridge/scripts/training/run_recipe.py`. Deletes `_build_command`, `_torchrun_rendezvous_args`, and `_flatten_overrides` (~90 lines). No nemotron.* imports so it runs cleanly in the uv venv. **CLI** (`commands/omni3/sft.py`): - Reads `recipe.name` / `recipe.step_func` from the compiled job config and passes them as `--recipe` / `--step_func` args alongside `--config REMOTE_CONFIG`. - `torchrun=(SPEC.run.launch == "torchrun")` — honors the runspec's launch mode for local execution too (was hardcoded `False`). Net effect: the hand-rolled multi-node rendezvous logic from the previous fix commit becomes unnecessary (nemo-run's Torchrun launcher does it); the `uv run` / base-venv-vs-uv-venv split disappears; the runspec is portable across dependency-management tools (swap uv for conda tomorrow: only the Dockerfile changes). Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(omni3/sft): open-dataset default via CORD-v2, Valor32k as opt-in The shipped default for Omni SFT pointed at Valor32k-AVQA, an NVIDIA-internal Energon dataset — so a fresh user running `nemotron omni3 sft --run <cluster>` out of the box hit `Energon dataset not found at /datasets/valor32k/energon` with no path to recovery. Flipping the default makes the recipe runnable from open data on a public clone. Config swaps (explicit git mv so history follows): - default.yaml → valor32k.yaml (internal Energon SFT, same contents; added header documenting OMNI3_VALOR32K_ENERGON_PATH) - image_text_sft.yaml → default.yaml (CORD-v2 via HF `vlm-hf` loader; train_iters bumped 20 → 1000; save path aligned to /nemo_run/omni3-sft-model; header documents the projector-only trade-off per QA guide §5.2.2) tiny.yaml repointed at CORD-v2 so `-c tiny` smoke tests work without internal dataset access. Same for data_prep/tiny.yaml. data_prep flow now supports both modes: - hf_dataset_id set → manifest-only (training container pulls from Hub) - dataset_path set → Energon validation (existing behavior) - both → ValueError - neither → ValueError pointing at the two config exemplars data_prep.py: - Omni3EnergonDataPrepConfig gains optional hf_dataset_id; dataset_path is now Optional[Path] so callers can choose the flow via which field they set. - run_data_prep dispatches to _run_hf_prep or _run_energon_prep; both emit a manifest.json carrying `source: huggingface|energon` for downstream artifact lineage. - PEP 723 runspec default flipped "valor32k" → "default" so `omni3 data prep sft` picks the HF config by default. New config: data_prep/default.yaml (CORD-v2 HF flow). Docs: sft.md and README.md Known Limitations updated to reflect the open default and the Valor32k opt-in. Stays honest about the projector-only constraint on single-node CORD-v2. Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Adding omni 3 recipes Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Update rl recipe and remove eval Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Fixing some docs issues Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Fixing some docs issues Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> --------- Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Update TRT-LLM setup for Nemotron-3-Nano-Omni (#175) * Add support for Lepton and DGX Cloud executors (#162) * Add Lepton and DGX Cloud executor support for nano3/super3 recipes Extends Nemotron executor layer to run on NVIDIA DGX Cloud Lepton and DGX Cloud (run:ai) alongside the existing Slurm path, including multi-node Ray-based RL (GRPO) via nemo-run's unified RayCluster + RayJob classes. Changes ------- * nemo-run pinned to the PR #480 build of rapaul-nv/Run (adds DGXCloudRayJob / LeptonRayCluster backends + client_credentials auth). * New nemo_runspec.execution: - execute_cloud : inline Script submission for non-Ray cloud jobs (data prep, torchrun-based SFT/pretrain). - execute_cloud_ray: RayCluster + RayJob path for launch="ray" recipes; used by nano3/super3 RL commands when on Lepton or DGX Cloud with nodes > 1. - _create_lepton_executor / _create_dgxcloud_executor factories, aligned with nemo-run 0.10 (client_id/client_secret, kube_apiserver_url, ray_version). - get_executor_type helper shared across all CLI command dispatchers. * CLI commands (nano3 + super3 sft/pretrain/rl + data/prep) route cloud jobs through the new execute_cloud / execute_cloud_ray paths. * env.toml profiles: [lepton], [lepton_gcp], [lepton_sft*], [lepton_rl], [dgxcloud] plus a shared [wandb] section. * Pipeline config accepts "dgxcloud"/"lepton" literals. * Tests: tests/nemo_runspec/test_execution.py covers executor creation (local/docker/slurm/lepton/dgxcloud), OmegaConf <-> plain-dict conversion, cloud workspace derivation, and git-mount command generation. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Ship local source to cloud pods without git install Airgap-friendly source transport for Lepton and DGX Cloud executors: local src/ is tarballed and delivered over the same Job API the launcher already uses, replacing the old `pip install nemotron @ git+…` step. - Lepton: chunk tarball across env vars (envp bypasses the 128 KiB argv MAX_ARG_STRLEN). NODE_RANK=0 extracts on shared NFS; other pods wait on a chunk-count-suffixed marker file to avoid the multi-pod rm-rf-then-tar race. - DGX Cloud (run:AI): drop tarball into job_dir as one `.tgz` file so move_data chunks only that file — torchrun_job.sh stays small. Env-var chunks would otherwise blow up the per-file deploy count. - Fallback (Slurm / others): nemo-run native packager extraction. Infrastructure: - New nemo_runspec.data_mover module with SourcePackager, Plan, plan_for. - run.py: patch_cloud_data_mover_skip_configs() excludes configs/ from nemo-run's inline-base64 tarball on both Lepton and DGXCloud, keeping the data-mover command under the kernel's MAX_ARG_STRLEN. - execute_cloud / execute_cloud_ray dispatch through a single plan_for() call; removed ~250 lines of branching packager/env-var/flag code. - env.toml knobs: `repo_root` (repo override), `dgxcloud_max_args_chars` (tune run:AI per-workload Args budget, default 9500). Tests: tests/nemo_runspec/test_data_mover.py covers include scoping, pycache filtering, all three transport branches, and the NODE_RANK gate. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Refactor Slurm executor construction and fix auto-mount behavior Slurm, Lepton, and DGX Cloud now share the same factory pattern. The previous ``create_executor`` had ~80 lines of inline Slurm setup while Lepton and DGX Cloud already used dedicated helpers. Five CLI commands also maintained near-identical copies of ``_build_slurm_executor``. Changes ------- * Extract ``create_slurm_executor`` as a public helper (with a ``launcher`` kwarg so Ray-based flows can pass ``None``). Also extract ``_create_local_executor`` and ``_create_docker_executor``. * Add ``_resolve_container_image`` and ``_resolve_nodes_gpus`` shared helpers that replace the three duplicated resolution blocks. * ``create_executor`` is now a small dispatch function. * Accept an ``identity`` field in env.toml so the SSH tunnel can be given an explicit private-key path when the agent isn't available. Bug fix for Slurm auto-mounts ----------------------------- On Slurm, ``${auto_mount:git+...}`` entries in a recipe config only register the requested repo into the ``get_git_mounts()`` registry at the moment OmegaConf resolves them. The previous code cloned the repos *before* the mounts list was accessed, so the registry was empty and the resulting sbatch script was missing the bind mounts. The container then silently fell back to the pre-baked Megatron-LM / Megatron-Bridge versions, causing training to run against the wrong code. ``_create_slurm_executor`` now reads ``env.mounts`` before cloning, which triggers resolution and populates the registry. CLI commands ------------ The following commands no longer carry their own Slurm executor setup and delegate to ``create_slurm_executor`` (with ``launcher=None`` for Ray flows): * ``nano3/data/prep/sft.py`` * ``nano3/data/prep/rl.py`` * ``nano3/data/prep/pretrain.py`` * ``nano3/rl.py`` * ``super3/rl/_base.py`` (persistent_cache and sif_dir extras applied on top of the returned executor) Misc ---- * Fix stale ``.gitignore`` whitelist: the CLI data directory moved from ``src/nemotron/cli/nano3/data/`` to ``src/nemotron/cli/commands/nano3/data/`` but the override entry was never updated, so the files were silently ignored. Verification ------------ * 131/131 unit tests pass. * Lint clean on all touched files. * End-to-end Slurm SFT data prep succeeds on CW DFW CS-001 (2 nodes, 10k samples, 47M tokens). Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Update dependencies and improve data mover functionality - Updated the `nemo-run` dependency to point to the main branch of the NVIDIA-NeMo repository. - Enhanced the `SourcePackager` class to streamline tarball creation for job directories. - Refactored the `plan_for` function to unify environment variable chunking for Lepton and DGX Cloud executors, improving data transport efficiency. - Introduced new patches to handle legacy keyword arguments and optimize environment variable handling in DGX Cloud. - Added a manifest resolver to facilitate configuration management for data preparation outputs. This commit ensures better compatibility with upstream changes and enhances the overall performance of the data mover functionality across different execution environments. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Enhance data preparation for cloud execution in super3 recipes - Added support for executing data preparation commands (pretrain, rl, sft) on DGX Cloud and Lepton using the new `execute_cloud` function. - Implemented logic to handle single-node CPU work for cloud environments, ensuring compatibility with the existing data preparation workflow. - Introduced functions to create train/val splits and update the manifest with per-split paths in the RL data preparation process. This update improves the flexibility and efficiency of data preparation across different execution environments. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * fix(nemo_runspec): address PR #162 review comments - docs: document required kube_apiserver_url; promote client_id/client_secret as primary auth fields with app_id/app_secret as legacy aliases - run.py: use argv list form for tar to avoid shell-metachar issues in nemo_run_dir paths - execution.py: honor explicit gpus_per_node=0 when picking nproc - execution.py: rename shadowed local launch -> launch_cmd - execution.py: narrow cluster.start except to "already exists" idempotency, re-raise auth/quota/image-pull failures Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Rakesh Paul <rapaul@nvidia.com> --------- Signed-off-by: Rakesh Paul <rapaul@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Nemotron 3 Nano Omni deployment cookbooks (#181) * Add Nemotron-3-Nano Omni deployment cookbooks Signed-off-by: Anusha Pant <apant@nvidia.com> * Update TRT-LLM setup for Nemotron-3-Nano-Omni Signed-off-by: Anusha Pant <apant@nvidia.com> * Update TRT-LLM setup for Nemotron-3-Nano-Omni Signed-off-by: Anusha Pant <apant@nvidia.com> * Update Nemotron-3-Nano-Omni deployment cookbooks Signed-off-by: Anusha Pant <apant@nvidia.com> --------- Signed-off-by: Anusha Pant <apant@nvidia.com> * First pass over nemotron skills Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * Splitting up skill Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> * feat(steps): add Nemotron steps for SFT, PEFT, RL, pretraining, SDG, and optimization This commit brings the following capabilities into the Nemotron steps: - SFT and PEFT workflows with AutoModel and Megatron Bridge runners. - RL workflows for DPO, RLHF, RLVR, and NeMo Gym-style GRPO execution. - Pretraining workflows with AutoModel and Megatron Bridge support. - Data preparation steps for SFT packing, pretraining, and RL data. - SDG workflows through the Data Designer step. - ModelOpt optimization steps for quantization, pruning, and distillation. - Evaluation, conversion, curation, and translation steps under the same discoverable execution model. - A common `nemotron step list/show/run` CLI for step discovery and launch. - Backend-aware execution for local, Slurm, and Lepton/cloud job submission. - Portable code packaging so remote jobs can run from packaged step sources. - Standardized `default` and `tiny` configs for fast bring-up and validation. - NVIDIA-backed tiny data blends for SFT and pretraining data prep. - Improved `nemo_runspec` execution and data movement support. - Updated docs, skills, and tests for extending and operating the step system. - Standalone data-prep steps now use the regular job backend path instead of assuming RayCluster provisioning. Validation: - SFT data prep successfully tested on Lepton. - SFT training submission successfully tested on Lepton. - DPO submission successfully tested on Lepton. - AutoModel SFT test successfully completed on Lepton. - Pretraining data prep successfully tested on Lepton. - Added/updated step test suite is passing with 100% pass rate. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * refactor(modelopt): standardize argument handling and improve configuration structure - Transitioning to a unified `args:` section in YAML configurations for upstream script arguments, enhancing clarity and usability. - Updating the Python scripts to accommodate the new argument structure, ensuring backward compatibility with legacy flat configurations. - Enhancing documentation to reflect the new configuration format and usage patterns. - Adding tests to validate the new argument mapping and ensure correct CLI flag translation. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(wandb): enhance W&B integration across various components - Added support for additional W&B environment variables such as WANDB_ENABLED, WANDB_GROUP, WANDB_JOB_TYPE, and WANDB_TAGS in the execution environment. - Updated the `init_wandb_from_env` function to utilize new W&B variables for improved logging and tracking. - Refactored YAML configuration files to pull W&B settings from environment variables, ensuring consistency and flexibility in project and entity naming. - Introduced a new function `init_prep_wandb` to initialize W&B with specific tags for data preparation steps, enhancing observability during the preparation phase. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(automodel): update configurations for Qwen3.6-35B-A3B model for demonstrating the support of non nemotron models using automodel - Enhanced step.toml and default.yaml files across PEFT, SFT, and pretraining workflows to include Qwen3.6-35B-A3B as the default model, optimizing for larger GPU setups. - Updated descriptions to clarify the use of LoRA adapters with qwen models for various training scenarios. - Ensured backward compatibility by maintaining existing model options while introducing the new larger model for improved performance. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(steps): add tiny configs for functional validation - Add low-scale tiny.yaml configs across prep, SFT, PEFT, pretrain, RL, synth, and ModelOpt steps with working values tested on A100 GPUs. - Align defaults with current launcher/runtime behavior, including NeMo-RL dataset schemas and policy defaults. - Update AutoModel configs for Qwen3 MoE settings. - Switch ModelOpt quantization configs to Nano-friendly recipes. - Use packaged SDG seed files and output paths. - Replace Megatron-Bridge packed-data paths with shared/env-driven locations. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * docs(steps): teach agents to use step metadata - Add step-level `SKILL.md` guidance across Nemotron stages - Instruct agents to read `step.toml` before changing configs or code - Document `step.toml` as the flow, artifact, parameter, strategy, failure-mode, and reference contract - Expand step and pattern docs for agentic navigation across prep, SFT, PEFT, pretrain, RL, synth, and ModelOpt workflows Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * `feat(runspec): harden Lepton and Ray step launches` - Add simple `defaults:` YAML inheritance for step configs - Preserve YAML and profile `env_vars` when merging env profiles - Support `NEMOTRON_ENV_FILE` for alternate env profile files - Redact sensitive config/env values in CLI display - Stage cloud source into unique per-run directories to avoid shared filesystem races - Keep `/nemo_run/code/src` valid when cloud source is staged elsewhere - Mask large transport env vars before Ray workers inherit them - Poll Ray job status instead of streaming logs for attached Ray launches - Add focused runspec tests for config inheritance, env override, transport, and Ray helpers Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * `feat(steps): harden functional step runtimes` - Add compatibility patches for AutoModel, Megatron-Bridge checkpointing, and NeMo-RL/NeMo-Gym API drift - Improve ModelOpt launch behavior for distributed runs, rank-0-only W&B logging, and safer optional checkout refresh - Emit pretrain prep `blend.json` split metadata for downstream pretraining steps - Update SFT, RL, pretrain, quantize, and Data Designer step entrypoints for current runtime behavior - Add RL prep and synthetic data seed files for tiny functional validation - Expand step tests for ModelOpt runner behavior, Data Designer schemas/results, and updated step contracts Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Add env profile generator and guidance Add env/env_toml for generating Lepton and Slurm profile examples as env.lepton.toml and env.slurm.toml, preserving any existing root env.toml. Include step-linked profiles for prep, pretrain, SFT, RL, and ModelOpt quantization, plus guidance for NEMOTRON_ENV_FILE, force=true, inheritance, runtime env vars, Ray placement, W&B, and site-private placeholders. Update step skills to make env profile setup the first check for non-local Lepton, Slurm, Ray, and batch runs. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Make data-prep env profiles CPU-only - Set Lepton and Slurm data prep profiles to request CPU resources. - Add Slurm CPU build/import settings for data-prep profiles. - Document the data-prep CPU-only convention in env skills. - Allow squash builds to opt into GPU allocations with build_include_gpus and build_gpus_per_node. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * fix: address PR review comments and harden staged steps - fix PR review findings around AutoModel model defaults, SFT/PEFT contracts, placeholder paths, and pretrain backend docs - update NeMo-RL DPO/RLVR/RLHF configs for v0.6 runtime and NeMo-Gym contracts - harden ModelOpt args precedence, W&B failure reporting, setup timeouts, and prune compatibility patching - bound cloud source-transport marker waits and fail fast on extraction failures - improve env profile merging, cloud script path handling, and Data Designer dataset creation - make Data Designer tool-chat generation parquet-safe - generate customer-support tool conversations as JSON text instead of nested structured columns - add tests for review fixes, RL configs, cloud paths, source transport, config coherence, and synth JSON handling Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * fix: address review comments for SDG Data Designer - rename the Data Designer step namespace from synth to sdg across steps, skills, docs, and tests - make customer-support tool-chat generation parquet-safe by avoiding escaped nested JSON in model output - serialize generated tool arguments and tool responses back to OpenAI-compatible string fields during projection - keep malformed generated rows skippable while failing when the full batch is invalid - add Data Designer tests for JSON parsing, tool payload normalization, and bad-row handling Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * refactor(skills, steps, patterns): one home per fact, sovereign-AI-focused catalog Make skills/nemotron-customize/ a thin orchestrator over src/nemotron/steps/. Realign the pattern catalog around the sovereign-AI customization journey (curate → SDG → prep → CPT/SFT/PEFT → RL → convert → eval). Strengthen every category and per-step SKILL with consistent structure. What changed - Customize SKILL.md rewritten around a numbered four-phase workflow (Orient → Plan → Act → Verify) with CLI-first discovery; act/PROJECT.md and act/STAGE.md tightened with concrete pointers into _runners/. - 10 raw upstream-doc dumps (~50K lines, regeneratable) deleted from context/. 5 authored extracts remain (≤250 lines each), grounded in actual step.toml + step.py + config defaults: automodel-pretrain, data-designer-sdg, modelopt-optimization, nemo-rl-alignment, nemotron-data-prep. - 11 step.toml files cleaned of dead `skills = [...]` lists and inline `skill = "..."` pointers to deleted packs. URLs in [reference] now carry the upstream pointers. - 5 missing per-step SKILL.md files added (eval/model_eval, curate/nemo_curator, translate/nemo_skills, plus a single category convert/SKILL.md covering the three thin I/O bridges). - prep/guide.md and sft/guide.md merged into their category SKILL.md. Pattern catalog - 14 patterns total (was 14). Refocused on sovereign-AI customization. - Removed: distill-after-structural-compression, representative-calibration-before-optimization, production-export-trt (compression / serving — not customization journey). - Added: cpt-data-blend-scoping, sft-data-blending, byob-benchmark-design. - Renamed for self-explanatory <topic>-<rule> form: eval-bookends → eval-before-and-after-training, pack-variable-length → sft-sequence-packing, small-dataset-lora → sft-small-dataset-prefer-lora, validate-rl-rewards-before-scale → rl-validate-rewards-before-scale, version-sdg-pipeline → sdg-pipeline-versioning, prepared-data-is-tokenizer-locked → prep-data-is-tokenizer-locked, checkpoint-before-convert → convert-checkpoint-safety, adapter-artifact-before-merge → peft-adapter-merge-discipline. - Quality-pass rewrites: convert-checkpoint-safety, pretrain-token-budget-before-scale, sdg-pipeline-versioning. - eval-before-and-after-training steps[] populated (was empty); every pattern now has populated cross-references. Category + per-step SKILLs - All 9 category SKILLs (prep, sft, peft, pretrain, rl, sdg, optimize + optimize/modelopt + rl/nemo_rl) follow the same structure: identity, steps table, decision tree, pipeline placement, pre-conditions, workflow, smoke commands, patterns to cite, guardrails. - Fixed peft/automodel/SKILL.md frontmatter typo ("suggest ---") that silently broke parsing. Verification - nv-base: 232 errors → 0 errors, 0 warnings. SCHEMA, SECURITY, PII, HYGIENE, UNICODE all pass. - discover_steps() = 23, discover_patterns() = 14, all step.toml valid. - All 32 SKILL.md files have valid YAML frontmatter. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * context(skills): add Megatron-Bridge SFT/PEFT pack, clarify AutoModel scope, fix Super3 model id - New pack mbridge-sft.txt covers sft/megatron_bridge + peft/megatron_bridge (runner flow, recipe selection, MoE knobs, packed-sequence rules, full vs LoRA). - AutoModel pack now explicitly states it's the path for non-Nemotron and non-MB-recipe models; MB stays for the curated recipe families. - Fix sft/megatron_bridge [[models]]: correct HF ids (NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, NVIDIA-Nemotron-3-Super-120B-A12B-BF16) and Super3 description (hybrid Mamba LatentMoE, not "dense 49B" — that conflated the older Llama-Nemotron-Super). - Wire both step.tomls + index.toml to the new pack. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * feat(execution): add cloud script path handling for pod-local source - Introduced `_cloud_script_path` function to convert script paths for cloud pod compatibility, ensuring that file-based commands reference the correct pod-local paths. - Updated `execute_cloud` and `execute_cloud_ray` functions to utilize the new path handling, enhancing support for concurrent jobs and avoiding shared mutable symlinks. - Added unit tests for `_cloud_script_path` to verify correct path transformations based on source layout. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> --------- Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: chrisalexiuk-nvidia <calexiuk@nvidia.com> Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Steve Han <sthan@nvidia.com> Signed-off-by: Doondi-Ashlesh <doondiashlesh@gmail.com> Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> Signed-off-by: Shashank Verma <shashankv@nvidia.com> Signed-off-by: Chia-Chih Chen <chiachihc@nvidia.com> Signed-off-by: Anusha Pant <apant@nvidia.com> Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> Signed-off-by: Rakesh Paul <rapaul@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Chris Alexiuk <161380339+chrisalexiuk-nvidia@users.noreply.github.com> Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com> Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Co-authored-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Steve Han <150830061+shan-nvidia@users.noreply.github.com> Co-authored-by: Doondi Ashlesh <126656890+Doondi-Ashlesh@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: hoangnguyenN94 <hoannguyen@nvidia.com> Co-authored-by: Shashank Verma <shashankv@nvidia.com> Co-authored-by: Chia-Chih Chen <chiachihc@nvidia.com> Co-authored-by: Anusha Pant <67914344+anushapant@users.noreply.github.com> Co-authored-by: Huiying <willwin.lee@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Vinh Nguyen <vinhn@nvidia.com> Co-authored-by: Marc Romeyn <marcromeyn@gmail.com> * remove personal paths * add instruction & example sections * Add airgap packaging flow for Nemotron Customizer - Add deploy-scoped airgap tooling for Nemotron Customizer steps under src/nemotron/steps. - Build a portable submitter image plus deduplicated task images for selected workflow targets. - Expand step dependencies and map selected steps to task image families through a single airgap.yaml. - Discover small task-image Python dependency gaps and bake pinned repo overlays required by step configs. - Models, datasets, checkpoints, and customer data to be kept in external persistent storage by user - Add resumable build state, image manifests with checksums, Dockerfiles, SFT overlay configs, README guidance, and focused tests. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Refine Nemotron Customizer airgap image flow - Rename airgap artifacts to use launcher and execution image terminology - Update runner stages, manifests, README, and config keys to match the new naming - Keep execution image generation scoped to selected Nemotron Customizer steps - Preserve external handling for models, datasets, checkpoints, and customer storage paths - Refresh SFT Megatron Bridge airgap overlay configs - Update tests for launcher/execution image behavior and staged runner flow Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Fix launcher image setup and Docker platform inspection - Install git and CA certificates in the launcher image before uv sync - Capture only docker inspect stdout while suppressing stderr during platform checks - Keep the airgap runner platform probe compatible with subprocess stderr handling Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Add configurable Data Designer providers Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Add Data Designer custom provider example Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Move Data Designer provider example into config comments Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Normalize step CLI under nemotron steps - Move generic step commands and backends from commands/step to commands/steps - Register only `nemotron steps`; remove the singular `nemotron step` alias - Expose `steps list`, `steps show`, `steps run`, and `steps translation` - Update imports, tests, docs, skills, and config examples to the plural CLI - Add coverage for plural command registration and singular alias rejection Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Airgap SKILL addition Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * bugs env resolve * Update RL step references from `rl/nemo_rl_grpo` to `rl/nemo_rl/rlvr` across documentation and codebase, reflecting the removal of the planned GRPO step. Clean up related files and ensure consistency in the Nano3 recipes and references, and synth cleanup Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * update skill: Prioritize existing repo code with YAML-only config generation for training, post-training, and training-data preparation workflows. Remove non-training workflow guidance while preserving SFT, PEFT/LoRA, RL alignment, continued pretraining, and post-training coverage. * Add lightweight Curator-backed step profiles and BYOB all-stage support - Add BYOB `stage=all` dispatch and config-driven family/stage defaults - Mount pinned Curator at `/opt/Curator` for BYOB and translate configs - Keep BYOB tiny smoke self-contained and avoid heavy semantic imports unless enabled - Simplify curate step for JSONL smoke runs with optional filters and Ray CPU env override - Add curate tiny config/data and refresh curate docs/metadata - Add Lepton/Slurm env profiles for BYOB, translate, curate, and SDG Data Designer - Prefer `tiny` configs in generated env examples where available - Fix env TOML rendering for empty lists Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Rename "prep" steps category to "data_prep" - Refactor step IDs, docs, configs, tests, and downstream references to use data_prep consistently across the Nemotron step library. - Addressed review comments - Removed benchmark folder Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Update translation QA runbook and dependencies Signed-off-by: rkalani <rkalani@nvidia.com> * Restructure QA runbook into test case flow Signed-off-by: rkalani <rkalani@nvidia.com> * Update and expand the env toml creation to all the steps Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Updated SKILL.md Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Update QA customization runbook Signed-off-by: rkalani <rkalani@nvidia.com> * Simplify Lepton training QA runbook * Update QA runbook for Lepton execution Signed-off-by: rkalani <rkalani@nvidia.com> * Update QA data prep validation Signed-off-by: rkalani <rkalani@nvidia.com> * Add translation skill guidance Signed-off-by: rkalani <rkalani@nvidia.com> * Improve Nemotron customize skill guidance Signed-off-by: rkalani <rkalani@nvidia.com> * Incorporate translation skill review guidance Signed-off-by: rkalani <rkalani@nvidia.com> * Add evaluator launcher step guidance Signed-off-by: rkalani <rkalani@nvidia.com> * faith generation config fix * remove test step * test scope doc update * undo guide changes * steps: normalise byob and translate step layout Every step now lives at `<category>/<implementation>/step.toml` and carries a two-segment `<category>/<implementation>` id. Two outliers were fixed: - `byob` -> `byob/mcq` (was a flat manifest directly under `byob/`, breaking the convention and leaving no room for future families). Only the manifest, entry script, and `config/` move into `byob/mcq/`; the `nemotron.steps.byob.*` Python package keeps its current import paths. - `translate/translation` -> `translate/curator`, naming the step after its backing engine to match `curate/nemo_curator` and to leave room for `translate/nmt` / `translate/google` / `translate/aws` siblings. Also: - Add the missing `env` category title in `CATEGORY_TITLES` (and drop the unused `benchmark` title that had no folder). - Regenerate `src/nemotron/steps/STEPS.md`. - Register one-release legacy aliases (`byob`, `translate/translation`) in `cli/commands/steps/_resolve.py` with a deprecation note, and start suggesting close matches on unknown ids. - Update every doc, skill pack, pattern manifest, tier-2 plan-graph case, and use-case example that referenced the old step ids or config paths. * steps: drop bespoke `steps translation` and top-level `nemotron byob` Both commands re-implemented config loading on top of the generic step dispatcher, forced ``mode == "local"``, rejected passthrough, and skipped the executor/backend selection that every other step uses. They existed solely because the underlying steps (`translate/translation`, `byob`) had irregular layout — fixed in the previous commit. Now every step is run through the same surface: nemotron steps run byob/mcq -c default -o stage=generate -o family=mcq nemotron steps run translate/curator -c default -o input_path=... This removes ~150 lines of duplicate config plumbing, restores one CLI contract for users and agents, and makes batch executors (Lepton, Slurm, DGXCloud) immediately available for translation and BYOB. The BYOB step's ``--list-families`` is still reachable through ``python -m nemotron.steps.byob.scripts.run --list-families`` and the choices are listed in ``nemotron steps show byob/mcq``. Updates docs, SKILL.md files, the use-case BYOB notebook, the QA scope checklist, and the translation CLI tests to assert the bespoke commands stay removed. * steps: list ergonomics (tag filter, tree view) Keep the override flow as bare key=value positionals — no -o flag. Verbose "-o key=value -o key=value" forms inflate every translation/BYOB example without adding meaningful safety over what split_unknown_args already does, so this commit drops the experimental -o/--override flag and rewrites the docs back to the concise positional form everyone was already using. Net additions on the catalog side: - `nemotron steps list --tag <tag>` filters by manifest tag, which makes the byob/translation overlap discoverable (`--tag translation`, `--tag mcq`, …). - `nemotron steps list --tree` groups discovered steps under their category title (driven by CATEGORY_TITLES) for a human-friendly view. No change to `steps run`'s public surface beyond what was already there: bare `key=value` positionals at the end of the command remain the override path. * steps: drop legacy id aliases for byob and translate Removes the temporary `byob` and `translate/translation` -> canonical-id redirects that the layout-normalisation commit had kept around for one release. With the rest of the repo (docs, skills, patterns, tier-2 plan-graph cases, use-case notebook, QA scope) already updated to the new ids in the earlier commits, the redirects only delayed the cutover and gave agents two valid spellings for the same step. Now: - `nemotron steps run byob` -> exit 1 with `Did you mean: byob/mcq?`. - `nemotron steps run translate/translation` -> exit 1 with `Did you mean: translate/curator?`. The fuzzy-match suggestion in `_resolve.py` still points users at the right id, so the legacy ids degrade gracefully instead of silently routing into a deprecated path. Tests swap the two `*_resolves_legacy_*_alias` cases for explicit rejection cases, and drops a stray `(legacy alias: byob)` mention from the byob-benchmark-curator-translation context pack. * Add generic Curator runtime bootstrap - Generate BYOB/translate/curate runtime requirements from pyproject/uv.lock at submission time. - Pass runtime metadata to remote jobs via env payloads instead of committed requirement files. - Run Curator-backed profiles through a shared curator_runtime entrypoint. - Update Lepton, Slurm, and DGX Cloud env templates for BYOB, translate, and curate. - Fix Slurm Curator jobs to use CodePackager and set PYTHONPATH for remote imports. - Add focused tests for runtime payloads, preflight behavior, BYOB config, and Slurm run_command handling. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Fix curator runtime profiles and evaluator hosting config * Remove QA runbook from CLI fixes branch * Sync CLI fixes with latest step layout Signed-off-by: rkalani <rkalani@nvidia.com> * Update SKILL.md to reflect new paths for nemo_curator step - Corrected references from `curator` to `nemo_curator` in the documentation. - Ensured consistency in the workflow instructions and load more section for clarity. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Enhance data mover and execution scripts with new features and optimizations - Expanded the list of excluded suffixes in data mover to include common data artifacts. - Introduced functions for tarball size warnings and formatted byte output. - Updated execution scripts to generate unique cloud config paths based on content digest. - Refactored symlink command for improved clarity and functionality. - Adjusted tests to validate new features and ensure proper functionality. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Update Megatron-Bridge configuration and documentation - Changed default configuration file from `nano3.yaml` to `default.yaml` in both the step implementation and documentation. - Updated recovery instructions in `step.toml` to reflect new dataset path specifications. - Removed obsolete `nano3.yaml` configuration file. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Add HF/Megatron conversion and LoRA merge steps - Add shared convert runner for HF->Megatron, Megatron->HF, and LoRA merge - Add default configs and metadata for convert steps - Add convert profiles to Lepton, Slurm, and DGX Cloud env configs - Add focused convert runner tests Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Refactor dtype handling in HF to Megatron conversion - Updated the conversion logic to prefer `torch_dtype` over the deprecated `dtype` alias. - Modified related documentation and configuration files to reflect the change from `dtype` to `torch_dtype`. - Added tests to ensure the new preference is correctly implemented and deprecated alias is handled. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Resolve high severity dependency CVEs Signed-off-by: rkalani <rkalani@nvidia.com> * Address VDR workflow issues Signed-off-by: rkalani <rkalani@nvidia.com> * Update configuration files for remote cluster compatibility - Modified various YAML configuration files to replace hardcoded paths with environment variable references, enhancing flexibility for remote cluster setups. - Updated comments in the `tiny.yaml` files to clarify usage for remote clusters. - Added new environment variable `RL_PREP_MANIFEST` to relevant configurations for better manifest management. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Reverted data designer tests Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Add Hugging Face access instructions to data preparation steps - Introduced Hugging Face access guidelines in `index.md`, `pretrain-prep.md`, `rl-prep.md`, and `sft-packing.md` to ensure users set the `HF_TOKEN` environment variable before running steps to avoid rate limits. - Updated `model-eval.md` to include Hugging Face access instructions for benchmark containers. Signed-off-by: Rakesh Paul <rapaul@nvidia.com> * Rename step docs to README Signed-off-by: rkalani <rkalani@nvidia.com> --------- Signed-off-by: Marc Romeyn <marcromeyn@gmail.com> Signed-off-by: rkalani <rkalani@nvidia.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: chrisalexiuk-nvidia <calexiuk@nvidia.com> Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com> Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com> Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Steve Han <sthan@nvidia.com> Signed-off-by: Doondi-Ashlesh <doondiashlesh@gmail.com> Signed-off-by: hoangnguyenN94 <hoannguyen@nvidia.com> Signed-off-by: Shashank Verma <shashankv@nvidia.com> Signed-off-by: Chia-Chih Chen <chiachihc@nvidia.com> Signed-off-by: Anusha Pant <apant@nvidia.com> Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Signed-off-by: Rakesh Paul <rapaul@nvidia.com> Co-authored-by: Marc Romeyn <marcromeyn@gmail.com> Co-authored-by: rapaul-nv <rapaul@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Chris Alexiuk <161380339+chrisalexiuk-nvidia@users.noreply.github.com> Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com> Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com> Co-authored-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Steve Han <150830061+shan-nvidia@users.noreply.github.com> Co-authored-by: Doondi Ashlesh <126656890+Doondi-Ashlesh@users.noreply.github.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: hoangnguyenN94 <hoannguyen@nvidia.com> Co-authored-by: Shashank Verma <shashankv@nvidia.com> Co-authored-by: Chia-Chih Chen <chiachihc@nvidia.com> Co-authored-by: Anusha Pant <67914344+anushapant@users.noreply.github.com> Co-authored-by: Huiying <willwin.lee@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Vinh Nguyen <vinhn@nvidia.com> Co-authored-by: hvnguyen <hvnguyen@nvidia.com> Co-authored-by: anushak <anushak@nvidia.com>

Update vLLM to pinned version 0.18.1

9718550

Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com>

chrisalexiuk-nvidia requested a review from shashank3959 April 9, 2026 18:47

claude Bot approved these changes Apr 9, 2026

View reviewed changes

chrisalexiuk-nvidia merged commit 16a983e into main Apr 9, 2026
4 checks passed

rapaul-nv pushed a commit to rkalaniNV/Nemotron that referenced this pull request Apr 30, 2026

Update vLLM to pinned version 0.18.1 (NVIDIA-NeMo#135)

ef05d06

Signed-off-by: Chris Alexiuk <calexiuk@nvidia.com>

Navigation Menu

Uh oh!

Update vLLM to pinned version 0.18.1 - #135

Update vLLM to pinned version 0.18.1#135
chrisalexiuk-nvidia merged 1 commit into
mainfrom
pin-vllm-version

chrisalexiuk-nvidia commented Apr 9, 2026

Uh oh!

chtruong814 commented Apr 9, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

chrisalexiuk-nvidia commented Apr 9, 2026

Uh oh!

chtruong814 commented Apr 9, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants