[CICD] Add train image pipeline and real TE-FL smoke testCi/ascend train te smoke by AlexMa616 · Pull Request #1250 · flagos-ai/FlagScale

AlexMa616 · 2026-07-24T14:21:32Z

Description

Add an Ascend training image pipeline and enable the Ascend CI suite with a
real TransformerEngine-FL training smoke test.

The change builds the training runtime from Ascend-specific Docker and installer
files, validates Megatron-LM-FL and TE-FL NPU initialization, and runs a
two-device Qwen3 0.6B smoke test through the TE-FL reference backend.

Chip-specific setup remains outside reusable common workflows. Ascend behavior
is defined through the platform configuration, Ascend installer and launcher,
Dockerfile, and dedicated image-build workflow. No FlagScale core source is
changed.

Prerequisite: TransformerEngine-FL must include 771a7488
(fix(ascend): include NPU backend in Python package) or an equivalent upstream
change before rebuilding the image from the official TE-FL repository.

Type of change

Infra/Build change (changes to CI/CD workflows or build scripts)
Bug fix
Code refactoring
New feature (non-breaking change which adds functionality)
Documentation change
Breaking change

Changes

Added a trusted Ascend training image build and Harbor publishing workflow.
Added the Ascend training Dockerfile and source dependency installation for
Megatron-LM-FL and TransformerEngine-FL.
Added an Ascend launcher that restores TE-FL NPU compatibility after Megatron
platform initialization.
Added a two-device Qwen3 0.6B mock-data smoke test using
transformer_impl: transformer_engine and TE-FL reference.torch operators.
Added platform-neutral finite-loss smoke validation and parser tests.
Enabled Ascend through the standard all_tests workflow and selected the
validated Harbor training image.
Aligned torchvision with the validated torch 2.7.1 and
torch_npu 2.7.1.post2 runtime.
Improved installer cache path handling, resumable sccache downloads, and
relative requirements include processing.

Validation

Repository-built training image smoke:
- Completed 3 training iterations on two Ascend NPUs.
- Dispatched TE-FL GEMM, RMSNorm forward/backward, and optimizer operations
  through reference.torch.
- No skipped or NaN iterations.
- 1 passed
- FLAGSCALE_ASCEND_REPO_IMAGE=PASS
Validated Harbor image:
- harbor.baai.ac.cn/flagos-dev/flagscale:manual-20260724-ascend-train-426ad116-te771a7488
- Digest: sha256:2fe15cf8b339b6828395fc3ce56e67b004b7c5f251d9e6673f20579819fe2c16
GitHub Actions:
- https://github.com/BrianPei/FlagScale/actions/runs/30083843003
- CLI, unit, train, inference, serve, and aggregate jobs passed.
- Benchmark and hetero-train remained skipped because they are not enabled
  in the Ascend matrix.
Local focused checks:
- Shell, Python, YAML, and Git diff validation passed.
- Ascend training matrix parsing passed.
- Result parser tests: 2 passed.

Checklist

I have read and followed the contributing guidelines
The functionality is complete for the Ascend training pipeline and smoke scope
I have commented my code, particularly in CI workflow setup steps
My changes generate no new warnings
I have tested the training workflow on the Ascend platform

CLAassistant · 2026-07-24T14:38:12Z

All committers have signed the CLA.

Darryl233 · 2026-08-05T14:10:05Z

Next time we try to build a build_image_common.yml and use vendor.yml to control the build workflow

We’ll reuse a common image-build workflow with vendor-specific configuration for future chips.

Darryl233 · 2026-08-05T14:11:30Z

  envs:
    ASCEND_VISIBLE_DEVICES: "0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15"
    HCCL_WHITELIST_DISABLE: 1
+    CC: ./tools/install/ascend/triton_gxx_wrapper.sh


what is this？

This wrapper serializes concurrent Triton compilation to prevent .gch file conflicts. I’ve added a comment to clarify it.

Darryl233 · 2026-08-05T14:13:59Z

+
+data:
+  # Keep the accelerator smoke test independent of runner-local .idx/.bin files.
+  # MockGPTDataset still exercises tokenization, forward/backward, HCCL, and the optimizer.


If use the MockGPTDataset，we can‘t verify the accuracy with golden value

I replaced the mock dataset with the existing real dataset and added golden loss validation.

Darryl233 · 2026-08-05T14:17:52Z

We write a test script to test another test script?

The extra parser logic and its unit test have been removed.

hooray03 · 2026-08-10T07:45:13Z

Following the review suggestion, image building has been split from this PR. This PR now contains only the standard Ascend workflow and test validation using prebuilt images; it no longer changes Dockerfiles, image-build workflows, requirements, or Ascend image installers. The image-build work is preserved separately on BrianPei:ci/ascend-image-build and will be submitted after this workflow PR is merged.

Darryl233 · 2026-08-11T07:08:08Z

check if necessary

Darryl233 · 2026-08-11T07:08:47Z

check if necessary

…d-train-te-smoke # Conflicts: # .github/workflows/all_tests.yml # tests/test_utils/runners/run_functional_tests.sh

Darryl233 · 2026-08-12T12:34:00Z

-ci_image: harbor.baai.ac.cn/flagscale/vllm-plugin-fl:v0.1.0-ascend-ci
-ci_train_image: harbor.baai.ac.cn/flagscale/vllm-plugin-fl:v0.1.0-ascend-ci
-ci_inference_image: harbor.baai.ac.cn/flagscale/vllm-plugin-fl:v0.1.0-ascend-ci
+ci_image: harbor.baai.ac.cn/flagos-dev/flagscale@sha256:e5b8e3b2c37584f980dffa1c08649b44e9edd2415522e123502e8d82a7854ef1


The Image tag seems strange to me？

Darryl233 · 2026-08-12T12:39:35Z

  envs:
    HYDRA_FULL_ERROR: 1
-    ASCEND_VISIBLE_DEVICES: "0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15"
+    ASCEND_VISIBLE_DEVICES: "2,3"


Are there any reasons why we set ASCEND_VISIBLE_DEVICES: "2,3"and “3, 4” in serve？ It seems strange.

This ensures that the inference and serving CI pipelines use separate NPUs when running in parallel, thereby avoiding device contention, VRAM conflicts, and mutual interference between tasks.

Does YuZhou agree with this ideas?

Darryl233 · 2026-08-12T12:42:40Z

Remove platform-related configs in a follow-up change

xmhubj · 2026-08-13T11:07:01Z

-            || { echo "❌ vllm-plugin-FL install failed"; exit 1; }
-          echo "✅ vllm-plugin-FL installed successfully"
-
          # Install FlagScale and dependencies


Is this necessary? Install FlagScale CLI in line 188 & in line 25, we installed FlagScale CLI again.

xmhubj · 2026-08-13T11:08:49Z

 #### 1. Launch Container
 ```bash
-docker pull quay.io/ascend/vllm-ascend:v0.13.0rc1-a3
+ASCEND_IMAGE=harbor.baai.ac.cn/flagos-dev/flagscale@sha256:e5b8e3b2c37584f980dffa1c08649b44e9edd2415522e123502e8d82a7854ef1


do we need to update the image tag?

xmhubj · 2026-08-13T11:19:43Z

+  - data
+
+data:
+  data_path: /home/gitlab-runner/data/pile_wikipedia_demo/pile_wikipedia_demo


is this the right data path?

Confirmed, yes.

xmhubj · 2026-08-13T11:35:08Z

+# after the Ascend inference and serve matrices pass.
+FLAGGEMS_REF="${FLAGSCALE_FLAGGEMS_REF:-61f3ff2773bc9c8e86b97489775ef9668a96a33c}"
+VLLM_PLUGIN_REPO="${FLAGSCALE_VLLM_PLUGIN_REPO:-https://github.com/flagos-ai/vllm-plugin-FL.git}"
+VLLM_PLUGIN_REF="${FLAGSCALE_VLLM_PLUGIN_REF:-43edeb601f4b8f616f56109de64836529e758deb}"


why we use this commit tag

AlexMa616 requested a review from aoyulong as a code owner July 24, 2026 14:21

AlexMa616 force-pushed the ci/ascend-train-te-smoke branch 2 times, most recently from be82b5a to c2a56d4 Compare July 24, 2026 14:44

tengqm force-pushed the main branch from ff2dae3 to 54957df Compare July 25, 2026 11:22

tengqm requested review from ceci3, cyber-pioneer, heavyrain-lzy and zhaoyinglia as code owners July 25, 2026 11:22

AlexMa616 force-pushed the ci/ascend-train-te-smoke branch from c2a56d4 to 3d7ea91 Compare July 26, 2026 14:03

Darryl233 reviewed Aug 5, 2026

View reviewed changes

hooray03 force-pushed the ci/ascend-train-te-smoke branch 2 times, most recently from 3eec7aa to 64843ae Compare August 10, 2026 07:44

hooray03 force-pushed the ci/ascend-train-te-smoke branch 4 times, most recently from 5abe025 to b0de48f Compare August 10, 2026 11:34

Darryl233 reviewed Aug 11, 2026

View reviewed changes

ci(ascend): enable workflow validation

b3072f1

hooray03 force-pushed the ci/ascend-train-te-smoke branch from b0de48f to b3072f1 Compare August 11, 2026 07:56

AlexMa616 added 5 commits August 11, 2026 20:49

ci(ascend): stabilize functional test launch

02e2569

Merge commit '067efe6f1b00bc9416e22cfe46c52990e10bf045' into ci/ascen…

ccb7f8a

…d-train-te-smoke # Conflicts: # .github/workflows/all_tests.yml # tests/test_utils/runners/run_functional_tests.sh

ci(ascend): use validated runtime images

dc0c34c

ci(ascend): restore available train image

fdcf3b7

ci(ascend): require standard training runtime

53fa6be

Darryl233 reviewed Aug 12, 2026

View reviewed changes

ci(ascend): use validated runtime images

3098e58

xmhubj reviewed Aug 13, 2026

View reviewed changes

hooray03 added 2 commits August 13, 2026 19:17

ci(ascend): add reproducible image contracts

d9a22d3

ci(metax): add image contracts and benchmark gate

34f0ac8

xmhubj reviewed Aug 13, 2026

View reviewed changes

hooray03 added 4 commits August 13, 2026 19:27

fix(metax): pin training runtime in image

58a23ca

fix(ascend): preserve vendor torch runtime

d4e91bd

fix(ascend): preserve vendor torch runtime

3ba5b8d

fix(ascend): validate inference with mounted driver

7c86400

xmhubj reviewed Aug 13, 2026

View reviewed changes

AlexMa616 and others added 7 commits August 13, 2026 19:51

ci(ascend): deduplicate CLI installation

47f4420

fix(ascend): preserve inference base contract

857b11e

ci: scope initial platform image contracts

d81bebc

fix(metax): initialize pinned training runtime

0a6cc4b

ci(metax): pin validated train image

29d322d

docs(ascend): use readable inference image tag

e4f6229

fix(ascend): restore inference installer mode

1143d4c

Uh oh!

Conversation

AlexMa616 commented Jul 24, 2026

Description

Type of change

Changes

Validation

Checklist

Uh oh!

CLAassistant commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hooray03 commented Aug 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CLAassistant commented Jul 24, 2026 •

edited

Loading