[CI] retire run-local venv across CI; install via image-baked stack (tile-ai#1606)

## What

Full RFC §4 cleanup on top of the ephemeral containerized-runner
migration. The runner image bakes tilelang + the runtime/dev stack, so
CI stops building/copying per-run venvs and installs only tileops
(`--no-deps`) via `scripts/ci/install_tileops.sh`.

- **gpu-smoke**: drop `Set up Python`, `Resolve runtime state` (venv
hash/copy/mtime-sync/divergence guard), `Cleanup isolated fork state`;
install via `install_tileops.sh`; run with the image's `python3`;
trust-route PRs by collaborator permission (write/maintain/admin →
resident `nightly` pool; everyone else + lookup failure → `fork` pool,
fail-closed); reclaim the `/ci-cache` layout (no wheels, no tool-cache
prune); per-test `--timeout` + a `timeout-minutes` backstop so a wedged
kernel cannot hold the single runner.
- **nightly**: `setup_nightly_venv.sh` → `install_tileops.sh`; system
`python3`; cache env `/data7` → `/ci-cache`; per-test timeout on the
pytest runs.
- **runner-maintenance**: drop the retired `venv` runs-on label (→
`nightly`).
- **preflight**: pin CPU installs with `constraints.txt`.
- **pyproject / constraints**: tilelang as a compatibility range
(constraints pins its deps, not tilelang itself); pin `pytest-timeout`.
- **reclaim action + verify script**: `/ci-cache` layout, drop
`WHEEL_DIR`; trim stamp on the persistent cache.
- **delete**: `setup_nightly_venv.sh`, `ci_venv_hash.py` and their
obsolete tests.

## In scope vs follow-up

To keep CI dispatching, installing and importing cleanly on the new
stack, this PR DOES adapt the import surface: `gqa_fwd_fp8` guards its
`from tvm import tir` with a sentinel that raises a targeted error only
when the kernel is built, and the fp8-GQA / topk-selector
kernel-building smoke cases are skipped via a `tvm.tir` availability
gate (with a focused test for the gate). So **CI is green at import**,
not red.

**Out of scope (follow-up):** the actual kernel migration off `tvm.tir`
(`tir.call_extern` → `tilelang.language` `T.*`) and any other new-stack
kernel regressions surfaced by gpu-smoke (e.g. a build hang in
`FP8LightingIndexerKernel`). Those are tracked separately; until they
land, the corresponding smoke cases are skipped or fail on the per-test
timeout rather than wedging the runner.

---------

Co-authored-by: Ibuki 🍃 — a wind born from GPTs <Ibuki-wind@users.noreply.github.com>

Author: lcy-seso

.github/actions/reclaim-runner-disk/action.yml

@@ -49,16 +49,15 @@ inputs:
     description: >
       Newline-separated list of cache directory roots whose files older than
       cache-age-days should be trimmed at file granularity. Defaults target
-      the shared /data7-backed runner cache layout.
+      the bind-mounted /ci-cache runner cache layout.
       Do NOT include atomic cache roots here (see atomic-cache-dirs); file-
       level trim on an atomic root can produce a half-dead "directory exists
       but sentinel missing" state that crashes downstream consumers.
     required: false
     default: |
-      /data7/shared/ci-cache/triton
-      /data7/shared/ci-cache/pip
-      /data7/shared/ci-cache/wheels
-      /data7/shared/ci-cache/tilelang/tmp
+      /ci-cache/triton
+      /ci-cache/pip
+      /ci-cache/tilelang/tmp
   atomic-cache-dirs:
     description: >
       Newline-separated list of cache directory roots whose first-level
@@ -75,7 +74,7 @@ inputs:
       tuning artefacts alongside a best_config.json sentinel.
     required: false
     default: |
-      /data7/shared/ci-cache/tilelang/autotuner
+      /ci-cache/tilelang/autotuner
   prune-tool-cache:
     description: >
       Whether to prune old tileops_ci_venv_* directories under runner.tool_cache.
@@ -195,7 +194,9 @@ runs:
         AVAIL_KB=$(df -Pk "${RUNNER_TEMP}" | awk 'NR==2 {print $4}')
         MIN_KB=$(( MIN_AVAIL_GIB * 1024 * 1024 ))
         RECLAIM_BELOW_KB=$(( RECLAIM_BELOW_GIB * 1024 * 1024 ))
-        CACHE_TRIM_STAMP="/home/ci-runner/.ci-maintenance/reclaim-runner-disk-cache-trim.stamp"
+        # Stamp lives on the persistent bind-mounted cache (not the ephemeral container fs under
+        # /home/ci-runner), so the cache-trim cooldown survives across ephemeral runner jobs.
+        CACHE_TRIM_STAMP="/ci-cache/.ci-maintenance/reclaim-runner-disk-cache-trim.stamp"
         DID_RECLAIM="false"
         SHOULD_RECLAIM="false"
 

.github/actions/reclaim-runner-disk/reclaim_cache.sh

@@ -11,7 +11,7 @@
 # validate behaviour against a tmp_path fixture.
 #
 # The core safety invariant this script enforces is that some cache
-# roots (notably /data7/shared/ci-cache/tilelang/autotuner) store
+# roots (notably /ci-cache/tilelang/autotuner) store
 # *atomic* first-level subdirectories: consumers assume "directory
 # exists => contents complete" and the directory is only valid if the
 # sentinel file (best_config.json) is present. File-level `-mtime`
@@ -37,8 +37,7 @@
 #   trim-files <age-days> <root> [<root>...]
 #       File-level `-mtime +N -delete` trim for non-atomic cache
 #       roots, followed by empty-directory cleanup. This is the legacy
-#       behaviour used for /data7/shared/ci-cache/triton,
-#       /data7/shared/ci-cache/pip, /data7/shared/ci-cache/wheels, etc.
+#       behaviour used for /ci-cache/triton, /ci-cache/pip, etc.
 #
 # All subcommands are idempotent, tolerate missing roots (skipped
 # silently), and never exit non-zero for per-directory errors so a

.github/workflows/gpu-smoke.yml

@@ -18,11 +18,10 @@ concurrency:
 # ---------------------------------------------------------------------------
 
 env:
-  TILELANG_CACHE_DIR: /data7/shared/ci-cache/tilelang
-  TILELANG_TMP_DIR: /data7/shared/ci-cache/tilelang/tmp
-  TRITON_CACHE_DIR: /data7/shared/ci-cache/triton
-  PIP_CACHE_DIR: /data7/shared/ci-cache/pip
-  WHEEL_DIR: /data7/shared/ci-cache/wheels
+  TILELANG_CACHE_DIR: /ci-cache/tilelang
+  TILELANG_TMP_DIR: /ci-cache/tilelang/tmp
+  TRITON_CACHE_DIR: /ci-cache/triton
+  PIP_CACHE_DIR: /ci-cache/pip
   MAX_JOBS: "64"
 
 jobs:
@@ -43,31 +42,17 @@ jobs:
         run: scripts/ci/verify_nightly_runner.sh
         shell: bash
 
-      - name: Set up run-local venv
-        run: scripts/ci/setup_nightly_venv.sh
+      - name: Install TileOPs (image-baked stack, --no-deps)
+        run: bash scripts/ci/install_tileops.sh
         shell: bash
 
       - name: Warm kernel compilation cache
         continue-on-error: true
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
-          source "${VENV_PATH}/bin/activate"
-
-          PIP_NO_BUILD_ISOLATION=1 pip install -e ".[dev]"
-
           export PYTHONPATH="${GITHUB_WORKSPACE}${PYTHONPATH:+:$PYTHONPATH}"
           echo "Starting kernel cache warmup..."
-          python scripts/warmup_kernel_cache.py --max-workers 64 -n 16
-        shell: bash
-
-      - name: Cleanup run-local venv
-        if: ${{ always() }}
-        run: |
-          set -euo pipefail
-          if [[ -n "${RUNTIME_ROOT:-}" && "${RUNTIME_ROOT}" == "${RUNNER_TEMP}/nightly-"* ]]; then
-            rm -rf "${RUNTIME_ROOT}"
-          fi
+          python3 scripts/warmup_kernel_cache.py --max-workers 64 -n 16
         shell: bash
 
   # =========================================================================
@@ -92,17 +77,15 @@ jobs:
         run: scripts/ci/verify_nightly_runner.sh
         shell: bash
 
-      - name: Set up run-local venv
-        run: scripts/ci/setup_nightly_venv.sh
+      - name: Install TileOPs (image-baked stack, --no-deps)
+        run: bash scripts/ci/install_tileops.sh
         shell: bash
 
       - name: Run benchmark ops
         id: benchmark_tests
         continue-on-error: true
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
-          source "${VENV_PATH}/bin/activate"
 
           # Validate GPU frequency
           TARGET_CLOCK_MHZ=1500
@@ -118,16 +101,15 @@ jobs:
             sleep 2
           done
 
-          PIP_NO_BUILD_ISOLATION=1 pip install -e ".[dev]"
-
           export PYTHONPATH="${GITHUB_WORKSPACE}${PYTHONPATH:+:$PYTHONPATH}"
           echo "Runtime cache env:"
           echo "TILELANG_CACHE_DIR=${TILELANG_CACHE_DIR:-}"
           echo "TILELANG_TMP_DIR=${TILELANG_TMP_DIR:-}"
           echo "TRITON_CACHE_DIR=${TRITON_CACHE_DIR:-}"
 
           set -o pipefail
-          python -m pytest -q benchmarks/ops --junit-xml=bench_results.xml | tee tileops_benchmarks.log
+          python3 -m pip install -q -c constraints.txt pytest-timeout
+          python3 -m pytest -q benchmarks/ops --timeout=900 --timeout-method=thread --junit-xml=bench_results.xml | tee tileops_benchmarks.log
 
           if [ -f profile_run.log ]; then
             { echo; echo "===== profile_run.log summary ====="; cat profile_run.log; } >> tileops_benchmarks.log
@@ -152,15 +134,6 @@ jobs:
         run: exit 1
         shell: bash
 
-      - name: Cleanup run-local venv
-        if: ${{ always() }}
-        run: |
-          set -euo pipefail
-          if [[ -n "${RUNTIME_ROOT:-}" && "${RUNTIME_ROOT}" == "${RUNNER_TEMP}/nightly-"* ]]; then
-            rm -rf "${RUNTIME_ROOT}"
-          fi
-        shell: bash
-
   # =========================================================================
   # Phase 3 — Op correctness tests (serial after benchmark for GPU exclusivity)
   # =========================================================================
@@ -179,19 +152,15 @@ jobs:
         run: scripts/ci/verify_nightly_runner.sh
         shell: bash
 
-      - name: Set up run-local venv
-        run: scripts/ci/setup_nightly_venv.sh
+      - name: Install TileOPs (image-baked stack, --no-deps)
+        run: bash scripts/ci/install_tileops.sh
         shell: bash
 
       - name: Run full op tests
         id: op_tests
         continue-on-error: true
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
-          source "${VENV_PATH}/bin/activate"
-
-          PIP_NO_BUILD_ISOLATION=1 pip install -e ".[dev]"
 
           export PYTHONPATH="${GITHUB_WORKSPACE}${PYTHONPATH:+:$PYTHONPATH}"
           echo "Runtime cache env:"
@@ -200,7 +169,8 @@ jobs:
           echo "TRITON_CACHE_DIR=${TRITON_CACHE_DIR:-}"
 
           set -o pipefail
-          python -m pytest tests/ -v --tb=short --junit-xml=test_results.xml | tee tileops_op_test.log
+          python3 -m pip install -q -c constraints.txt pytest-timeout
+          python3 -m pytest tests/ -v --tb=short --timeout=900 --timeout-method=thread --junit-xml=test_results.xml | tee tileops_op_test.log
         shell: bash
 
       - name: Download benchmark artifacts
@@ -277,18 +247,15 @@ jobs:
         if: ${{ always() }}
         run: |
           set -euo pipefail
-          # shellcheck source=/dev/null
-          source "${VENV_PATH}/bin/activate"
 
-          pip install --no-deps -e . 2>/dev/null
           export PYTHONPATH="${GITHUB_WORKSPACE}${PYTHONPATH:+:$PYTHONPATH}"
 
           HISTORY_ARGS=()
           if [ -f .perf_history/perf_history.json ]; then
             HISTORY_ARGS=(--history .perf_history/perf_history.json)
           fi
 
-          python scripts/nightly_report.py \
+          python3 scripts/nightly_report.py \
             --test-xml test_results.xml \
             --bench-xml bench_results.xml \
             "${HISTORY_ARGS[@]}" \
@@ -328,15 +295,6 @@ jobs:
         run: exit 1
         shell: bash
 
-      - name: Cleanup run-local venv
-        if: ${{ always() }}
-        run: |
-          set -euo pipefail
-          if [[ -n "${RUNTIME_ROOT:-}" && "${RUNTIME_ROOT}" == "${RUNNER_TEMP}/nightly-"* ]]; then
-            rm -rf "${RUNTIME_ROOT}"
-          fi
-        shell: bash
-
   # =========================================================================
   # Phase 3b — Publish benchmark data to the nightly-bench orphan branch
   # Consumed by tile-ai/TileOPs.github.io to render the docs Benchmarks page.
@@ -423,16 +381,14 @@ jobs:
         run: scripts/ci/verify_nightly_runner.sh
         shell: bash
 
-      - name: Set up run-local venv
-        run: scripts/ci/setup_nightly_venv.sh
+      - name: Install TileOPs (image-baked stack, --no-deps)
+        run: bash scripts/ci/install_tileops.sh
         shell: bash
 
       - name: Build and test wheel package
         run: |
           set -euo pipefail
           exec > >(tee packaging.log) 2>&1
-          # shellcheck source=/dev/null
-          source "${VENV_PATH}/bin/activate"
 
           # Validate GPU frequency
           TARGET_CLOCK_MHZ=1500
@@ -448,17 +404,17 @@ jobs:
             sleep 2
           done
 
-          pip install build pytest
+          python3 -m pip install -c constraints.txt build pytest pytest-timeout
 
           echo "=== Build wheel package ==="
-          python -m build --wheel
+          python3 -m build --wheel
           echo "=== Build artifacts ==="
           ls -lh dist/*.whl
 
           echo "=== Install built wheel ==="
-          pip install dist/*.whl
+          python3 -m pip install dist/*.whl
 
-          TMP_TEST_DIR="$(mktemp -d "${RUNTIME_ROOT}/packaging-test.XXXXXX")"
+          TMP_TEST_DIR="$(mktemp -d "${RUNNER_TEMP}/packaging-test.XXXXXX")"
           cp -r tests "${TMP_TEST_DIR}/tests"
           cp pyproject.toml "${TMP_TEST_DIR}/"
           cd "${TMP_TEST_DIR}"
@@ -469,7 +425,8 @@ jobs:
           echo "TRITON_CACHE_DIR=${TRITON_CACHE_DIR:-}"
 
           echo "=== Run pytest packaging ==="
-          python -m pytest -q tests/ops -m "packaging" \
+          python3 -m pytest -q tests/ops -m "packaging" \
+            --timeout=900 --timeout-method=thread \
             --junit-xml="${GITHUB_WORKSPACE}/packaging_smoke.xml"
         shell: bash
 
@@ -499,12 +456,3 @@ jobs:
         with:
           files: dist/*.whl
           fail_on_unmatched_files: true
-
-      - name: Cleanup run-local venv
-        if: ${{ always() }}
-        run: |
-          set -euo pipefail
-          if [[ -n "${RUNTIME_ROOT:-}" && "${RUNTIME_ROOT}" == "${RUNNER_TEMP}/nightly-"* ]]; then
-            rm -rf "${RUNTIME_ROOT}"
-          fi
-        shell: bash

.github/workflows/nightly.yml

@@ -247,7 +247,7 @@ jobs:
           python-version: "3.10"
 
       - name: Install TileOPs and validator dependencies
-        run: pip install -e .
+        run: pip install -e . -c constraints.txt
 
       - name: Validate ops manifest
         run: python scripts/validate_manifest.py --levels schema,signature,shape,dtype,bench --strict
@@ -299,7 +299,7 @@ jobs:
           python-version: "3.10"
 
       - name: Install TileOPs (with dev extras)
-        run: pip install -e ".[dev]"
+        run: pip install -e ".[dev]" -c constraints.txt
 
       - name: Run benchmark contract tests
         run: python -m pytest -q benchmarks/tests

.github/workflows/preflight.yml

@@ -7,8 +7,8 @@ name: Runner Maintenance
 # cooldown; this scheduled job forces the full cleanup pass on days with
 # little PR traffic.
 #
-# GPU smoke and nightly share the persistent cache tree under
-# `/data7/shared/ci-cache/*`.
+# GPU smoke and nightly share the persistent cache tree bind-mounted at
+# `/ci-cache/*` inside the runner container.
 
 on:
   schedule:
@@ -25,7 +25,9 @@ concurrency:
 jobs:
   reclaim-disk:
     if: ${{ github.repository == 'tile-ai/TileOPs' }}
-    runs-on: [self-hosted, tile-ops, venv]
+    # `nightly` (not `fork`) keeps maintenance on the resident shared-cache runners; the
+    # destructive autotuner age-trim must never run against a fork pool's overlay cache.
+    runs-on: [self-hosted, tile-ops, nightly]
     timeout-minutes: 60
     steps:
       # Invoke the action directly via owner/repo/path@ref so this workflow

.github/workflows/runner-maintenance.yml

@@ -21,3 +21,6 @@ typing_extensions==4.15.0
 Cython==3.2.4
 z3-solver==4.15.4.0
 torch_c_dlpack_ext==0.1.5
+# CI test plugin: per-test hang guard for the GPU jobs. Pinned here so the runtime install is
+# deterministic; the runner image should bake it (follow-up) and drop the per-job install.
+pytest-timeout==2.3.1

constraints.txt

@@ -6,11 +6,13 @@ readme = "README.md"
 authors = [{ name="Tile-AI"}]
 dependencies = [
     "torch>=2.1.0,<2.11.0",
-    "tilelang==0.1.9",
-    # FIXME(apache-tvm-ffi-pin): Remove once tilelang carries its own
-    # apache-tvm-ffi<0.1.10 pin (already on tilelang main, not yet released).
-    # Tracking: https://github.com/tile-ai/TileOPs/issues/969
-    "apache-tvm-ffi>=0.1.2,<0.1.10",
+    # tilelang as a compatibility range, not a source or exact pin. CI does not resolve tilelang
+    # from this range: the runner image bakes a specific tilelang build and CI installs tileops
+    # with --no-deps against it. constraints.txt pins tilelang's runtime deps (torch, triton,
+    # apache-tvm-ffi, ...) for reproducibility but intentionally does NOT pin tilelang itself.
+    # Downstream `pip install tileops` resolves a compatible tilelang release; apache-tvm-ffi
+    # comes in transitively via tilelang's own metadata — no manual cap here.
+    "tilelang>=0.1.9,<0.2.0",
     "einops",
     "pyyaml>=6.0",
 ]