Skip to content

docs: add DeepSeek V4 Flash performance verification - #5724

Merged
yaoyu-33 merged 4 commits into
mainfrom
yuya/mb-1361-dsv4-perf
Aug 22, 2026
Merged

docs: add DeepSeek V4 Flash performance verification#5724
yaoyu-33 merged 4 commits into
mainfrom
yuya/mb-1361-dsv4-perf

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the DeepSeek V4 Flash verification card and records verified pretraining-performance results from the canonical 128-GPU MXFP8 recipes.

  • GB300: 7,840.320 ms/step, 759.280 model TFLOPS/GPU, 8,358.842 tokens/s/GPU; loss 12.59616 -> 3.160061
  • GB200: 8,134.600 ms/step, 731.400 model TFLOPS/GPU, 8,056.450 tokens/s/GPU; loss 13.58750 -> 3.62683

The broader card content preserves the prior model-card work from #5360, with schema-required training throughput fields audited and completed.

Tracking: MB-1361
GB300 evidence: GitLab job 396937714

Testing

  • Card schema validator
  • 38 focused model-card validator tests
  • pre-commit run --all-files
  • git diff --check

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@claude

claude Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Light review: DeepSeek-V4-Flash verification card

New model verification card plus matching entries in the validator throughput-input test dict. The card is well-formed and the five throughput metrics in test_validate_card.py are consistent with the recorded step times. Two things worth addressing:

  1. Cluster codename leaks into the card (privacy): line 168 says '...during data-loading on Lyris.' The card skill forbids recording execution-environment / cluster names, and the note is fully portable without it. The bundled validator only greps the literal word cluster, so it passes validation, but this still breaks the privacy boundary. Inline suggestion left.

  2. SFT recipe name does not exist on main: lines 179 and 256 use deepseek_v4_flash_sft_openmath_thinking_packed_gb200_config, but the exported recipe is deepseek_v4_flash_sft_openmath_thinking_packed_config (no _gb200 suffix), defined at src/megatron/bridge/recipes/deepseek/deepseek_v4.py:88. A user running the sft / sft_long_context commands verbatim would hit an unknown-recipe error. Please confirm the intended name. The pretrain/perf recipes (64gpu_gb200_bf16, 128gpu_gb200_fp8mx, 128gpu_gb300_fp8mx) all do exist.

Minor: verification_index.training.GB200 (lines 38-40) has a stray blank line 39 between the verified: and unsupported: keys. Cosmetic only.

Suggested test cases

  • tests/unit_tests/skills/create_model_verification_card/test_validate_card.py::test_shipped_training_tps_matches_audited_token_slot_inputs (exercises the five new deepseek-v4-flash entries added to TRAINING_THROUGHPUT_INPUTS against the shipped card metrics)
  • Validate the new card file with skills/create-model-verification-card/scripts/validate_card.py (confirms it parses and passes structural/privacy gates)

No perf tests impacted: this PR touches only a model verification card and the validator unit-test data, not any scripts/performance/configs/ recipe.

0-31999 are within DSV4 vocab 0-129279; NullTokenizer is used). The run
completes 100 steps from random initialisation with finite loss, no skipped or
NaN iterations, and complete checkpoints at steps 50 and 100. Note: set
dataset.num_workers=0 to avoid NCCL pipeline timeout during data-loading on Lyris.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Privacy: "Lyris" reads as an execution-environment / cluster codename. The card skill explicitly forbids recording execution-environment names, and the point being made ("set dataset.num_workers=0 to avoid an NCCL pipeline timeout during data-loading") is portable — drop the cluster name. The validator doesn't catch this because it only matches the literal word cluster, but it still violates the skill's privacy boundary.

Suggested change
dataset.num_workers=0 to avoid NCCL pipeline timeout during data-loading on Lyris.
dataset.num_workers=0 to avoid NCCL pipeline timeout during data-loading.

context_parallel_size: 2
command: >
./scripts/training/train.sh --nodes 16 --gpus-per-node 4
--recipe deepseek_v4_flash_sft_openmath_thinking_packed_gb200_config

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The recipe deepseek_v4_flash_sft_openmath_thinking_packed_gb200_config does not exist on main. The registered SFT recipe is deepseek_v4_flash_sft_openmath_thinking_packed_config (no _gb200 suffix) — see src/megatron/bridge/recipes/deepseek/deepseek_v4.py:88. This same name is repeated at line 256 (sft_long_context). A user copy-pasting the command today would hit an unknown-recipe error. Please confirm the recipe name matches what's currently exported, or add the _gb200 variant.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

Addressed in 41e0c4b: removed the private cluster codename and corrected both SFT commands to the exported deepseek_v4_flash_sft_openmath_thinking_packed_config recipe. Card validation, 38 focused tests, pre-commit, and diff check all pass.

@yaoyu-33 yaoyu-33 added area:model Model implementations and HF bridge logic docs Documentation-only updates or documentation debt docs-only With great power comes great responsibility. needs-review PR is ready for code review and waiting on a reviewer labels Aug 20, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

Updated in 91ce65d after auditing current main:\n\n- CPU import/export are now unverified: the generic CPU backend exists, and the observed ~570 GB+ host-memory requirement is a resource constraint rather than evidence of product-level non-support.\n- Inference is now unverified: optimized KV-cache inference remains unavailable, but the maintained legacy full-prefix path intentionally supplies inference_context=None, so it is a plausible compatibility path pending an exact DSV4 run.\n- PEFT is now unverified under GB200: no exact recipe is exported, but recipe absence does not establish architectural non-support.\n\nThe verification index, detailed leaves, summary, immutable revision pins, and candidate public commands were updated together. Card validation, all 38 focused card tests, pre-commit, and git diff --check pass.

@yaoyu-33 yaoyu-33 removed the docs-only With great power comes great responsibility. label Aug 22, 2026
@yaoyu-33
yaoyu-33 merged commit cc59a63 into main Aug 22, 2026
4 checks passed
@yaoyu-33
yaoyu-33 deleted the yuya/mb-1361-dsv4-perf branch August 22, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic docs Documentation-only updates or documentation debt needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant