Refactor install scripts and add dockerfiles (flagos-ai#1100)

### PR Category
<!-- One of [ Train | Inference | Compress | Serve | RL | Core |
Hardware | CICD | Tools | Others ] -->
CICD

### PR Types
<!-- One of [ User Experience | New Features | Bug Fixes | Improvements
| Performance | Breaking Change| Deprecations | Test Case | Docs |
Others ] -->
Improvemetns

### PR Description
<!-- Describe what you’ve done -->
This pull request updates the CI/CD configuration to support a unified
and flexible package manager/environment setup for functional tests,
with a focus on future compatibility (e.g., for switching to `uv` or
`pip`), and temporarily disables some functional test jobs. The changes
introduce new configuration options, refactor workflow inputs/outputs,
and update environment activation logic to be package-manager agnostic.

Key changes:

**Package Manager and Environment Configuration**
- Added unified package manager configuration (`pkg_mgr`, `env_path`,
`env_names`) to `.github/configs/cuda.yml`, supporting `pip`, `uv`, and
`conda`, with clear documentation for future transitions.
- Updated workflow inputs/outputs in
`.github/workflows/all_tests_common.yml` to use generic `pkg_mgr`,
`env_name`, and `env_path` variables instead of hardcoded `conda_env`
and `conda_path`, allowing for flexible environment handling.

**Functional Test Job Refactoring**
- Updated `functional_tests_hetero_train.yml` to use the new `pkg_mgr`,
`env_name`, and `env_path` inputs, and refactored the environment
activation logic to support `conda`, `uv`, and `pip` using a new utility
script (`pyenv_utils.sh`).

**Temporary Disabling of Some Functional Tests**
- Commented out (disabled) the `functional_tests_inference`,
`functional_tests_serve`, and `functional_tests_rl` jobs in
`all_tests_common.yml`, and removed their checks from the
`all_tests_complete` job, with explanatory comments for future
re-enabling.

**Cosmetic and Documentation Updates**
- Renamed workflow in `.github/workflows/all_tests_cuda.yml` from "CUDA
Tests Validation" to "cuda_tests" and added a comment about reading
environment settings from the config file.

These changes make the CI/CD system more modular, easier to maintain,
and ready for future upgrades in package management.

---------

Co-authored-by: MC952-arch <MC952-arch@qq.com>

Author: aoyulong

.github/configs/cuda.yml

@@ -26,3 +26,32 @@ container_volumes:
 
 # Container options (hardware-specific settings)
 container_options: "--gpus all --shm-size=500g --hostname flagscale_cicd --user root --ulimit nofile=65535:65535"
+
+# =============================================================================
+# Package Manager Configuration
+# =============================================================================
+# Supported package managers: pip, uv, conda
+#   - pip:   Use pip directly (standard Python)
+#   - uv:    Use uv pip (fast, modern package manager)
+#   - conda: Use conda environment with pip for PyPI packages
+#
+# Unified environment parameters:
+#   - env_name: Conda environment name (for conda only)
+#   - env_path: Environment path (venv path for uv, conda installation path for conda)
+#
+# To transition to uv in the future:
+#   1. Change pkg_mgr to "uv"
+#   2. Ensure uv is installed in the Docker image
+#   3. Set env_path to the virtual environment path (e.g., "/opt/venv")
+#
+pkg_mgr: "conda"  # Current: conda for CI/CD compatibility
+
+# Environment path (venv path for uv, conda installation path for conda)
+env_path: "/root/miniconda3"
+
+# Conda environment name (for conda only)
+env_names:
+  train: "flagscale-train"
+  hetero_train: "flagscale-train"
+  inference: "flagscale-inference"
+  rl: "flagscale-rl"

.github/workflows/all_tests_common.yml

@@ -25,6 +25,12 @@ jobs:
       inference_test_matrix: ${{ steps.config.outputs.inference_test_matrix }}
       serve_test_matrix: ${{ steps.config.outputs.serve_test_matrix }}
       rl_test_matrix: ${{ steps.config.outputs.rl_test_matrix }}
+      pkg_mgr: ${{ steps.config.outputs.pkg_mgr }}
+      env_path: ${{ steps.config.outputs.env_path }}
+      env_name_train: ${{ steps.config.outputs.env_name_train }}
+      env_name_inference: ${{ steps.config.outputs.env_name_inference }}
+      env_name_serve: ${{ steps.config.outputs.env_name_serve }}
+      env_name_rl: ${{ steps.config.outputs.env_name_rl }}
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
@@ -107,8 +113,9 @@ jobs:
       container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
       container_options: ${{ needs.checkout_and_config.outputs.container_options }}
       source_artifact: flagscale-source-${{ github.sha }}
-      conda_env: flagscale-train  # Optional: can be empty for non-conda environments
-      conda_path: "/root/miniconda3"  # Optional: specify custom conda path, empty for auto-detection
+      pkg_mgr: ${{ needs.checkout_and_config.outputs.pkg_mgr }}
+      env_name: ${{ needs.checkout_and_config.outputs.env_name_train }}
+      env_path: ${{ needs.checkout_and_config.outputs.env_path }}
 
   functional_tests_train:
     needs:
@@ -124,8 +131,9 @@ jobs:
       container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
       container_options: ${{ needs.checkout_and_config.outputs.container_options }}
       source_artifact: flagscale-source-${{ github.sha }}
-      conda_env: flagscale-train  # Optional: can be empty for non-conda environments
-      conda_path: "/root/miniconda3"  # Optional: specify custom conda path, empty for auto-detection
+      pkg_mgr: ${{ needs.checkout_and_config.outputs.pkg_mgr }}
+      env_name: ${{ needs.checkout_and_config.outputs.env_name_train }}
+      env_path: ${{ needs.checkout_and_config.outputs.env_path }}
 
   functional_tests_hetero_train:
     needs:
@@ -141,59 +149,64 @@ jobs:
       container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
       container_options: ${{ needs.checkout_and_config.outputs.container_options }}
       source_artifact: flagscale-source-${{ github.sha }}
-      conda_env: flagscale-train  # Optional: can be empty for non-conda environments
-      conda_path: "/root/miniconda3"  # Optional: specify custom conda path, empty for auto-detection
+      pkg_mgr: ${{ needs.checkout_and_config.outputs.pkg_mgr }}
+      env_name: ${{ needs.checkout_and_config.outputs.env_name_train }}
+      env_path: ${{ needs.checkout_and_config.outputs.env_path }}
 
-  functional_tests_inference:
-    needs:
-      - checkout_and_config
-      - unit_tests
-    if: fromJson(needs.checkout_and_config.outputs.inference_test_matrix)[0] != null
-    uses: ./.github/workflows/functional_tests_inference.yml
-    with:
-      platform: ${{ inputs.platform }}
-      test_matrix: ${{ needs.checkout_and_config.outputs.inference_test_matrix }}
-      image: ${{ needs.checkout_and_config.outputs.ci_image }}
-      runs_on: ${{ needs.checkout_and_config.outputs.runs_on }}
-      container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
-      container_options: ${{ needs.checkout_and_config.outputs.container_options }}
-      source_artifact: flagscale-source-${{ github.sha }}
-      conda_env: flagscale-inference  # Optional: can be empty for non-conda environments
-      conda_path: "/root/miniconda3"  # Optional: specify custom conda path, empty for auto-detection
+  # NOTE: Inference, serve, and rl functional tests are temporarily disabled
+  # functional_tests_inference:
+  #   needs:
+  #     - checkout_and_config
+  #     - unit_tests
+  #   if: fromJson(needs.checkout_and_config.outputs.inference_test_matrix)[0] != null
+  #   uses: ./.github/workflows/functional_tests_inference.yml
+  #   with:
+  #     platform: ${{ inputs.platform }}
+  #     test_matrix: ${{ needs.checkout_and_config.outputs.inference_test_matrix }}
+  #     image: ${{ needs.checkout_and_config.outputs.ci_image }}
+  #     runs_on: ${{ needs.checkout_and_config.outputs.runs_on }}
+  #     container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
+  #     container_options: ${{ needs.checkout_and_config.outputs.container_options }}
+  #     source_artifact: flagscale-source-${{ github.sha }}
+  #     pkg_mgr: ${{ needs.checkout_and_config.outputs.pkg_mgr }}
+  #     env_name: ${{ needs.checkout_and_config.outputs.env_name_inference }}
+  #     env_path: ${{ needs.checkout_and_config.outputs.env_path }}
 
-  functional_tests_serve:
-    needs:
-      - checkout_and_config
-      - unit_tests
-    if: fromJson(needs.checkout_and_config.outputs.serve_test_matrix)[0] != null
-    uses: ./.github/workflows/functional_tests_serve.yml
-    with:
-      platform: ${{ inputs.platform }}
-      test_matrix: ${{ needs.checkout_and_config.outputs.serve_test_matrix }}
-      image: ${{ needs.checkout_and_config.outputs.ci_image }}
-      runs_on: ${{ needs.checkout_and_config.outputs.runs_on }}
-      container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
-      container_options: ${{ needs.checkout_and_config.outputs.container_options }}
-      source_artifact: flagscale-source-${{ github.sha }}
-      conda_env: flagscale-inference  # Optional: can be empty for non-conda environments
-      conda_path: "/root/miniconda3"  # Optional: specify custom conda path, empty for auto-detection
+  # functional_tests_serve:
+  #   needs:
+  #     - checkout_and_config
+  #     - unit_tests
+  #   if: fromJson(needs.checkout_and_config.outputs.serve_test_matrix)[0] != null
+  #   uses: ./.github/workflows/functional_tests_serve.yml
+  #   with:
+  #     platform: ${{ inputs.platform }}
+  #     test_matrix: ${{ needs.checkout_and_config.outputs.serve_test_matrix }}
+  #     image: ${{ needs.checkout_and_config.outputs.ci_image }}
+  #     runs_on: ${{ needs.checkout_and_config.outputs.runs_on }}
+  #     container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
+  #     container_options: ${{ needs.checkout_and_config.outputs.container_options }}
+  #     source_artifact: flagscale-source-${{ github.sha }}
+  #     pkg_mgr: ${{ needs.checkout_and_config.outputs.pkg_mgr }}
+  #     env_name: ${{ needs.checkout_and_config.outputs.env_name_serve }}
+  #     env_path: ${{ needs.checkout_and_config.outputs.env_path }}
 
-  functional_tests_rl:
-    needs:
-      - checkout_and_config
-      - unit_tests
-    if: fromJson(needs.checkout_and_config.outputs.rl_test_matrix)[0] != null
-    uses: ./.github/workflows/functional_tests_rl.yml
-    with:
-      platform: ${{ inputs.platform }}
-      test_matrix: ${{ needs.checkout_and_config.outputs.rl_test_matrix }}
-      image: ${{ needs.checkout_and_config.outputs.ci_image }}
-      runs_on: ${{ needs.checkout_and_config.outputs.runs_on }}
-      container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
-      container_options: ${{ needs.checkout_and_config.outputs.container_options }}
-      source_artifact: flagscale-source-${{ github.sha }}
-      conda_env: flagscale-RL  # Optional: can be empty for non-conda environments
-      conda_path: "/root/miniconda3"  # Optional: specify custom conda path, empty for auto-detection
+  # functional_tests_rl:
+  #   needs:
+  #     - checkout_and_config
+  #     - unit_tests
+  #   if: fromJson(needs.checkout_and_config.outputs.rl_test_matrix)[0] != null
+  #   uses: ./.github/workflows/functional_tests_rl.yml
+  #   with:
+  #     platform: ${{ inputs.platform }}
+  #     test_matrix: ${{ needs.checkout_and_config.outputs.rl_test_matrix }}
+  #     image: ${{ needs.checkout_and_config.outputs.ci_image }}
+  #     runs_on: ${{ needs.checkout_and_config.outputs.runs_on }}
+  #     container_volumes: ${{ needs.checkout_and_config.outputs.container_volumes }}
+  #     container_options: ${{ needs.checkout_and_config.outputs.container_options }}
+  #     source_artifact: flagscale-source-${{ github.sha }}
+  #     pkg_mgr: ${{ needs.checkout_and_config.outputs.pkg_mgr }}
+  #     env_name: ${{ needs.checkout_and_config.outputs.env_name_rl }}
+  #     env_path: ${{ needs.checkout_and_config.outputs.env_path }}
 
   all_tests_complete:
     defaults:
@@ -204,9 +217,10 @@ jobs:
       - unit_tests
       - functional_tests_train
       - functional_tests_hetero_train
-      - functional_tests_inference
-      - functional_tests_serve
-      - functional_tests_rl
+      # NOTE: Disabled tests removed from needs
+      # - functional_tests_inference
+      # - functional_tests_serve
+      # - functional_tests_rl
     runs-on: ubuntu-latest
     if: always()
     steps:
@@ -233,23 +247,24 @@ jobs:
             failed=true
           fi
 
-          if [ "${{ needs.functional_tests_inference.result }}" != "success" ] && \
-             [ "${{ needs.functional_tests_inference.result }}" != "skipped" ]; then
-            echo "❌ Inference functional tests failed"
-            failed=true
-          fi
+          # NOTE: Inference, serve, and rl checks disabled
+          # if [ "${{ needs.functional_tests_inference.result }}" != "success" ] && \
+          #    [ "${{ needs.functional_tests_inference.result }}" != "skipped" ]; then
+          #   echo "❌ Inference functional tests failed"
+          #   failed=true
+          # fi
 
-          if [ "${{ needs.functional_tests_serve.result }}" != "success" ] && \
-             [ "${{ needs.functional_tests_serve.result }}" != "skipped" ]; then
-            echo "❌ RL functional tests failed"
-            failed=true
-          fi
+          # if [ "${{ needs.functional_tests_serve.result }}" != "success" ] && \
+          #    [ "${{ needs.functional_tests_serve.result }}" != "skipped" ]; then
+          #   echo "❌ Serve functional tests failed"
+          #   failed=true
+          # fi
 
-          if [ "${{ needs.functional_tests_rl.result }}" != "success" ] && \
-             [ "${{ needs.functional_tests_rl.result }}" != "skipped" ]; then
-            echo "❌ RL functional tests failed"
-            failed=true
-          fi
+          # if [ "${{ needs.functional_tests_rl.result }}" != "success" ] && \
+          #    [ "${{ needs.functional_tests_rl.result }}" != "skipped" ]; then
+          #   echo "❌ RL functional tests failed"
+          #   failed=true
+          # fi
 
           if [ "$failed" = "true" ]; then
             exit 1

.github/workflows/all_tests_cuda.yml

@@ -1,4 +1,4 @@
-name: CUDA Tests Validation
+name: cuda_tests
 
 on:
   push:
@@ -12,6 +12,7 @@ concurrency:
 
 jobs:
   run_tests:
+    # Package manager and environment settings are read from .github/configs/cuda.yml
     uses: ./.github/workflows/all_tests_common.yml
     with:
       platform: cuda