docs(eval-skill): add NVFP4 model-card sampling reference

Add references/nvfp4-modelcard-sampling.md: the published temperature,
top_p and max generation length for the 2026 NVFP4 checkpoints under
huggingface.co/nvidia that disclose them, grouped by model family, and
make Step 3 of the evaluation skill cross-check it before writing eval
params.

Config generation currently rederives sampling params per run from a
single model card, which is slow and silently wrong in two ways. First,
most NVIDIA NVFP4 cards paste a TensorRT-LLM quickstart containing
SamplingParams(temperature=0.8, top_p=0.95) -- byte-identical across
Llama-3.1-8B, Llama-3.3-70B, Llama-4-Scout, Phi-4, Qwen2.5-VL and six
Qwen3 repos -- which is boilerplate, not an eval setting, yet it is the
most prominent temperature= in the card. Second, when a card is silent
the skill falls back to a generic 65536/16384 default even though a
sibling in the same family publishes an exact value.

All 69 NVFP4 checkpoints in the org were read; the table lists the 28
published in 2026 whose cards disclose usable settings, so absence means
"read the card" rather than "not yet checked". Rows carry provenance --
23 "eval" where the card ties the values to its accuracy table, 5 "rec"
where it recommends them without that tie -- and the lookup procedure
resolves per field so a "rec" row or an unpublished field has defined
behavior. Per-task exceptions are captured (GLM-5.2 GPQA 100000 vs
64000; Qwen3.5-397B-V2 tau2 128000; Qwen3.6 SciCode temperature 0.6;
Kimi-K3 uncapped for Terminal-Bench).

Cross-check rule: exact model listed -> use its row; absent -> use the
nearest same-family rows; card conflicts with the family pattern -> card
wins, but surface the discrepancy. Generations do break the pattern, so
the priors are scoped to what the table actually shows -- DeepSeek is
explicitly carved out of the 1.0/0.95 house default because both its
rows use top_p=1.0.

The file ends with a refresh recipe covering the two extraction traps
found while building it: cards that publish the cap only as a footnote
under the accuracy table ("*Max OSL for evals can be as high as 64K"),
and DeepSeek stating sampling in its usage block rather than a
"Benchmarked with" sentence. Neither is reachable from the obvious grep.

Not changelog-worthy: agent-skill documentation, no user-facing API or
behavior change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>

Author: cjluo-nv

plugins/modelopt/skills/evaluation/SKILL.md

@@ -274,13 +274,16 @@ nemo_evaluator_config:
 
 Per-task `max_new_tokens` overrides are forbidden — set one top-level ceiling everywhere.
 
+**Cross-check `temperature` / `top_p` / `max_new_tokens` against `references/nvfp4-modelcard-sampling.md`** — the published settings for the 2026 NVFP4 checkpoints under `huggingface.co/nvidia` that disclose them (older releases and cards that publish nothing are absent — for those, read the card). Mandatory whenever the model is an NVFP4 checkpoint **or shares a family with one** (Qwen3.x, GLM-4.7/5.x, Kimi K2.x/K3, MiniMax M2.x/M3, DeepSeek V3.x/V4/R1, Gemma 4, Nemotron 3/3.5, Llama-Nemotron, Mistral Medium 3.5). See the rules in that file's "How to use it".
+
 #### `max_new_tokens` — mandatory model-card lookup
 
 1. **Fetch the HF model card before writing the value.** Not optional.
 2. Scan for any `max_tokens` / `max_new_tokens` / "output length" recommendation. Pick the **highest** value the card mentions (Qwen3.6: 32768 general + 81920 math-coding → use **81920**). Annotate with a citing comment.
-3. If the card is genuinely silent after a thorough read, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user.
-4. **Forbidden:** writing `max_new_tokens: <generic_default>` with a "card not yet checked" comment. Either fetch and apply, or fetch and confirm silence.
-5. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence.
+3. **Cross-check `references/nvfp4-modelcard-sampling.md`.** If the model is listed there, your value must match its row (or you explain why). If it is not listed, read the nearest same-family rows and use that pattern — it is a far better prior than the generic fallback below. The card wins on a conflict, but surface the discrepancy to the user rather than auto-correcting.
+4. If the card is genuinely silent after a thorough read **and** the family table offers no usable pattern, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user.
+5. **Forbidden:** writing `max_new_tokens: <generic_default>` with a "card not yet checked" comment. Either fetch and apply, or fetch and confirm silence.
+6. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence.
 
 #### Quantization-aware benchmark defaults
 

plugins/modelopt/skills/evaluation/references/model-card-research.md

@@ -3,6 +3,17 @@
 Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it carefully, the FULL text, the devil is in the details. Extract ALL relevant configurations:
 
 - Sampling params (`temperature`, `top_p`)
+  - **Only trust a sentence that ties the values to the benchmarks** ("Benchmarked
+    with…", "…were evaluated with…", "We evaluate the model using…") or a
+    "Recommended Sampling" row. The `SamplingParams(temperature=0.8, top_p=0.95)`
+    in a card's TensorRT-LLM/vLLM quickstart snippet is boilerplate copied
+    verbatim across unrelated models — **never** read eval settings out of it.
+  - **Then cross-check `nvfp4-modelcard-sampling.md`** — published
+    `temperature` / `top_p` / `max_num_tokens` for the 2026 NVFP4 checkpoints
+    under `huggingface.co/nvidia` that disclose them, grouped by family.
+    Required for any NVFP4 checkpoint or same-family sibling; it is also the
+    best source of a default when the card is silent. Pre-2026 releases are out
+    of scope there — read their cards.
 - Context length (`deployment.extra_args: "--max-model-len <value>"`)
 - **Output length (`max_new_tokens`) — mandatory extraction.** Scan the
   card for any `max_tokens` / `max_new_tokens` / "output length"
@@ -11,7 +22,9 @@ Use WebSearch to find the model card (HuggingFace, build.nvidia.com). Read it ca
   apply at the top level (no per-task overrides). If the card is genuinely
   silent on output length, note that explicitly and fall back to the
   generic default (64K reasoning / 16K non-reasoning) — never write a
-  config with "card not yet checked" + generic default. See SKILL.md
+  config with "card not yet checked" + generic default. Check
+  `nvfp4-modelcard-sampling.md` for the model or its family before falling
+  back; a same-family published cap beats the generic default. See SKILL.md
   Step 3 "`max_new_tokens` — pick a single top-level value" for the full
   rule.
 - TP/DP settings (to set them appropriately, AskUserQuestion on how many GPUs the model will be deployed)

plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md

@@ -0,0 +1,219 @@
+# NVFP4 Model-Card Sampling Reference
+
+Published sampling / generation settings for the **NVFP4 checkpoints under
+[huggingface.co/nvidia](https://huggingface.co/nvidia/models) that disclose
+them**, limited to checkpoints **published in 2026** — 28 rows, collected
+**2026-08-20**. The org listing was swept in full (not sampled); checkpoints
+older than 2026-01-01, and those whose cards publish no usable settings, are
+absent. A miss here means "read the card", not "not yet checked".
+
+## Why this file exists
+
+Model cards state the settings NVIDIA actually used to produce the accuracy
+table on the card. Reproducing a published NVFP4 number — or comparing a new
+quantized checkpoint against one — requires the *same* settings. This table is
+the fast lookup so you don't re-derive them per run, and the sanity check when a
+card is silent or ambiguous.
+
+**It does not replace reading the card** (see `model-card-research.md`). It is a
+cross-check on top of it.
+
+## How to use it (mandatory cross-check)
+
+When generating an eval config, **before** writing
+`nemo_evaluator_config.config.params`:
+
+1. **Look up the exact model** in the table, and resolve **per field**:
+   - `eval` provenance → use the value and cite the row.
+   - `rec` provenance → use the value, but note in the config comment that it is
+     the card's *recommended* sampling rather than a stated eval setting. If a
+     same-family `eval` row disagrees, surface both to the user.
+   - A field showing `—` → that field is unpublished even though the row exists;
+     resolve **that field only** via step 2.
+2. **If the exact model is absent** (new checkpoint, unreleased variant,
+   non-NVIDIA baseline) — **look up the nearest same-family rows** (same base
+   family and, where it exists, the adjacent version: Qwen3.5 → Qwen3.6,
+   GLM-5 → 5.1 → 5.2, Kimi-K2.6 → K2.7 → K3, MiniMax-M2.5 → M3,
+   DeepSeek-V4-Flash, Nemotron-3 Super / 3.5 Lightning). A pre-2026 model is
+   out of scope here — read its card. Use the family pattern as the expected
+   value.
+3. **Reconcile card vs. table:**
+   - Agree → proceed with confidence.
+   - Card is silent, family is consistent → adopt the family value and say so
+     in a comment on the line (e.g. `# family default per
+     references/nvfp4-modelcard-sampling.md (Qwen3.5/3.6 thinking)`). This beats
+     the generic 65536 / 16384 fallback in SKILL.md Step 3.
+   - **Card disagrees with the family pattern → the card wins**, but surface the
+     discrepancy to the user before running. Sampling defaults *do* change
+     between generations (Qwen3-Coder-480B-Instruct is greedy at
+     `temperature=0.0` while Qwen3.5 thinking is `0.6` and Qwen3.6 is `1.0`;
+     DeepSeek-V4-Flash uses `top_p=1.0` where the rest of the table uses
+     `0.95`), so a mismatch is a signal to re-read, not to auto-correct.
+4. **Baseline vs. candidate must share one setting.** When comparing an NVFP4
+   checkpoint to its BF16/FP8 baseline, use the NVFP4 card's row for *both* —
+   the cards report both precisions measured under the single setting listed.
+
+### Trap — do not read sampling params out of the quickstart code
+
+Most NVIDIA NVFP4 cards paste a TensorRT-LLM / vLLM snippet containing
+`SamplingParams(temperature=0.8, top_p=0.95)` or `max_tokens=32`. That is
+**boilerplate**, identical across unrelated models, and is **not** the eval
+setting. Only take values from a sentence that says *Benchmarked with…* /
+*evaluated with…* / *We evaluate the model using…*, or from a "Recommended
+Sampling" row. Everything below follows that rule; `provenance` records which.
+
+- `eval` (23 rows) — card states these were the benchmark/eval settings.
+  Authoritative.
+- `rec` (5 rows) — card recommends these for inference but does not tie them to
+  the accuracy table. Good prior, weaker evidence.
+
+A `—` in a value column means that card publishes the other fields but not this
+one (commonly no generation cap); fall back to the family pattern for the
+missing field, then to SKILL.md Step 3 defaults, and tell the user.
+
+`max_num_tokens` = maximum generation length (the card's "max num tokens" /
+`max_new_tokens` / `max_tokens`), which maps to
+`nemo_evaluator_config.config.params.max_new_tokens`.
+
+---
+
+## DeepSeek
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/DeepSeek-V4-Flash-NVFP4` | 1.0 | **1.0** | 384000 | eval | note the `top_p=1.0` — differs from the `0.95` every other row here uses |
+| `nvidia/DeepSeek-V4-Flash-nvfp4-DSpark` | 1.0 | **1.0** | 384000 | eval | same as the non-DSpark checkpoint |
+
+## Qwen
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/Qwen3.6-35B-A3B-NVFP4` | 1.0 | 0.95 | 131072 | eval | **SciCode used `temperature=0.6`** (same top_p / max) |
+| `nvidia/Qwen3.6-27B-NVFP4` | 1.0 | 0.95 | 81920 | eval | **SciCode `temperature=0.6`; τ²-Bench Telecom `temperature=0.0`, `top_p=1.0`** |
+| `nvidia/Qwen3.5-397B-A17B-NVFP4-V2` | 0.6 | 0.95 | 64000 | eval | **τ²-Bench Telecom used 128000** |
+| `nvidia/Qwen3.5-397B-A17B-NVFP4` | 0.6 | 0.95 | 64000 | eval | |
+| `nvidia/Qwen3.5-122B-A10B-NVFP4` | 0.6 | 0.95 | 64000 | eval | |
+| `nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP4` | **0.0** | **1.0e-05** | 16384 | eval | greedy — instruct variant |
+
+> **Qwen family pattern:** thinking/reasoning variants → `0.6 / 0.95` (Qwen3.5)
+> with a large cap (64000–131072, trending up by generation); instruct/coder
+> variants → near-greedy `0 / 1e-5` with `16384`. Qwen3.6 breaks the thinking
+> temperature to `1.0` — confirm against the card for any Qwen3.6+ checkpoint.
+
+## GLM
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/GLM-5.2-NVFP4` | 1.0 | 0.95 | 64000 | eval | **GPQA Diamond used `max_new_tokens=100000`** |
+| `nvidia/GLM-5.1-NVFP4` | 1.0 | 0.95 | 64000 | eval | benchmarked on `vllm/vllm-openai:v0.19.1` |
+| `nvidia/GLM-5-NVFP4` | 1.0 | 0.95 | 131072 | eval | |
+| `nvidia/GLM-4.7-NVFP4` | 1.0 | 0.95 | 131072 | eval | |
+
+> **GLM family pattern:** `1.0 / 0.95` throughout; the cap dropped from 131072
+> (4.7, 5) to 64000 (5.1, 5.2).
+
+## Kimi
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/Kimi-K3-NVFP4` | 1.0 | 0.95 | 65536 | eval | **uncapped generation for Terminal-Bench.** Card's recommended generation params: `temperature=1.0`, `n=1`, `presence_penalty=0`, `frequency_penalty=0`, `top_p=0.95` single-step / **`top_p=1.0` agentic** |
+| `nvidia/Kimi-K2.7-Code-NVFP4` | 1.0 | 0.95 | 64000 | eval | |
+| `nvidia/Kimi-K2.6-NVFP4` | 1.0 | 0.95 | 128000 | eval | |
+
+## MiniMax
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/MiniMax-M3-NVFP4` | 1.0 | 0.95 | 65536 | eval | baseline is MiniMax-M3 native MXFP8 |
+| `nvidia/MiniMax-M2.5-NVFP4` | 1.0 | 0.95 | 64000 | eval | |
+
+## Gemma
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/Gemma-4-31B-IT-NVFP4` | 1.0 | 0.95 | 131072 | eval | |
+| `nvidia/Gemma-4-26B-A4B-NVFP4` | 1.0 | 0.95 | 131072 | eval | |
+| `nvidia/diffusiongemma-26B-A4B-it-NVFP4` | upstream default | upstream default | — | eval | card defers to `google/diffusiongemma-26B-A4B-it`; diffusion decoding, serve with `--override-generation-config '{"max_new_tokens": null}'` |
+| `nvidia/Ising-Calibration-1.5-31B-NVFP4` | 0.2 | — | 8192 zero-shot / 32767 ICL | rec | domain model (Gemma-4-31B derivative); these are suggested inference settings, not an eval statement |
+
+## Mistral
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/Mistral-Medium-3.5-128B-NVFP4` | 0.7 | 0.95 | — | eval | benchmarked with `reasoning_effort="high"`; no generation cap stated |
+
+## Nemotron
+
+| Model card ID | temperature | top_p | max_num_tokens | provenance | notes |
+| --- | --- | --- | --- | --- | --- |
+| `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` | 1.0 | 0.95 | — | rec | "Recommended Sampling: Temperature 1.0, Top_P 0.95"; eval recipes published in NeMo Gym, cap not on the card (client examples use `max_tokens=16000`) |
+| `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash` | 1.0 | 0.95 | — | eval | |
+| `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark` | 1.0 | 0.95 | — | eval | |
+| `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` | 1.0 | 0.95 | — | rec | card: use `1.0 / 0.95` **across all tasks and serving backends** — reasoning, tool calling, chat alike |
+| `nvidia/NVIDIA-Nemotron-Labs-3-Elastic-30B-A3B-NVFP4` | 1.0 | 1.0 | — | rec | reasoning tasks |
+| `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4` | 0.6 thinking / 0.2 instruct | 0.95 thinking / — instruct | 20480 thinking / 1024 instruct | rec | thinking mode also sets `reasoning_budget=16384`, `grace_period=1024`; instruct mode sets `top_k=1` |
+
+---
+
+## Cross-family observations (use as priors, verify per card)
+
+- **`1.0 / 0.95` is the house default** for recent reasoning checkpoints (GLM
+  4.7–5.2, Kimi K2.6–K3, MiniMax M2.5–M3, Gemma 4, Nemotron 3/3.5, Qwen3.6).
+  When a recent card is silent, this is the best guess.
+- **DeepSeek is not covered by that prior.** Both DeepSeek rows here
+  (V4-Flash, V4-Flash-DSpark) use `top_p=1.0`, not `0.95`. Do not carry the
+  house default onto an unlisted DeepSeek variant — read its card.
+- **Qwen is the exception**: Qwen3.5 *thinking* uses `0.6 / 0.95`;
+  *instruct*/*coder* variants use near-greedy `0 / 1e-5`.
+- **Caps cluster at 64000 / 65536 / 81920 / 128000 / 131072.** 64000 is the most
+  common recent value. `16384` shows up only with greedy instruct Qwen.
+  DeepSeek-V4-Flash's `384000` is a long-context outlier.
+- **Per-task overrides are real but narrow** — they appear only for SciCode
+  (lower temperature), τ²-Bench Telecom (greedy or larger cap), GPQA Diamond
+  (larger cap), and Terminal-Bench (uncapped). SKILL.md Step 3 forbids per-task
+  `max_new_tokens` overrides in generated configs, so when a card lists two
+  caps, **take the maximum** as the single top-level value and note the card's
+  split in a comment.
+
+## Refreshing this table
+
+The list was built from the HuggingFace API rather than by scraping
+`https://huggingface.co/nvidia/models?p=N`, and verified to match it page for
+page (918 repos across `p=0..31`; the 68 NVFP4-named repos were identical in
+both). To refresh:
+
+```bash
+# 1. Enumerate the org (single page; 917+ repos, well under the 1000 limit)
+curl -s "https://huggingface.co/api/models?author=nvidia&limit=1000" -o all.json
+
+# 2. Filter NVFP4 by repo name OR tag (catches FP4-named NVFP4 checkpoints)
+python3 -c "
+import json
+ms = json.load(open('all.json'))
+for m in ms:
+    tags = [t.lower() for t in m.get('tags', [])]
+    if (m['id'].lower().count('nvfp4') or 'nvfp4' in tags or 'fp4' in tags) \\
+            and m.get('createdAt', '') >= '2026-01-01':   # this-year scope
+        print(m['id'])
+" > ids.txt
+
+# 3. Pull cards (add -H \"Authorization: Bearer \$(cat ~/.cache/huggingface/token)\" for gated repos)
+while read id; do curl -sfL "https://huggingface.co/$id/raw/main/README.md" \
+  -o "cards/${id//\//_}.md"; done < ids.txt
+
+# 4. Only these phrasings carry eval settings — never the quickstart snippets
+grep -ihnE "benchmark(ed|ing) (parameters|with)|were evaluated with|we evaluate the model using|evaluation settings" cards/*.md
+
+# 5. Caps also hide in footnotes under the accuracy table, and DeepSeek states
+#    its sampling in a usage block rather than a "Benchmarked with" sentence
+grep -ihnE "max OSL|for evals?|including benchmarking" cards/*.md
+```
+
+Step 5 is not optional. Some cards publish the cap **only** as a footnote under
+the accuracy table — e.g. *"\*Max OSL for evals can be as high as 64K"* — and
+DeepSeek states its sampling in the `## Input:` usage block rather than a
+*Benchmarked with* sentence. Neither is reachable from the step-4 greps.
+
+Step 3 returns 404 for repos that ship no `README.md` and 401 for gated repos
+without a token. Eagle3 draft-head repos surface via the `fp4` tag but are not
+NVFP4 target checkpoints — skip them.