Skip to content

[CI] [Feature] Avoid redundant Git checkout inside Modal test containers #1609

Description

@SolitaryThinker

Problem

Buildkite performs its normal source checkout before executing .buildkite/scripts/pr_test.sh, but each Modal CI function then clones BUILDKITE_REPO into /FastVideo and checks out the PR/commit again in fastvideo/tests/modal/pr_test.py::run_test_command.

This duplicates Git network work for every parallel GPU lane and increases cold-start time.

Proposed investigation

Evaluate attaching the Buildkite worker's already-checked-out repository to Modal with modal.Image.add_local_dir(..., copy=False), then using a per-container writable workspace:

Buildkite checkout -> /src/FastVideo (runtime source attachment) -> copy -> /workspace/FastVideo (ephemeral writable workspace)

Run uv pip install -e, kernel build, and pytest from /workspace/FastVideo.

Do not use one shared writable Modal Volume as a Git checkout/cache: concurrent git fetch, checkout, and build-artifact writes can race.

Scope

  • Update fastvideo/tests/modal/pr_test.py image/function setup and run_test_command.
  • Remove the remote git clone, PR-ref fetch, checkout, and submodule update from the Modal runtime command.
  • Preserve exact Buildkite revision semantics and required submodules/assets.
  • Keep test outputs, editable-install metadata, and kernel build artifacts in per-container writable storage.
  • Retain separate shared caches only for appropriate immutable/cache-like data (for example, HF model weights).

Acceptance criteria

  • A representative Modal CI lane runs against the same commit as the Buildkite worker checkout.
  • Parallel Modal functions cannot modify one another's source/workspace.
  • CI no longer performs git clone inside the Modal test container.
  • The migration does not require rebuilding the base Modal image for every source change.
  • Startup-time comparison is recorded for before/after validation.

Notes

add_local_dir(..., copy=False) makes source available at container startup, not during image build. Any code-dependent setup must therefore run at container runtime in the writable workspace

Metadata

Metadata

Assignees

No one assigned

    Labels

    installationInstallation and setup issuesscope: attentionAttention backends (VSA, STA, Flash, etc.)scope: docsDocumentationscope: kernelCUDA kernels, fastvideo-kerneltype: perfPerformance improvement

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions