Proposal: Lightweight cross-provider GPU support

[hjames9]

GPU Support Proposal for kind

Context

This proposal addresses issue #3164 (Support GPUs) by suggesting a lightweight implementation approach that learns from previous rejected PRs.

History of GPU Support Attempts

Several PRs have attempted to add GPU support but were not merged:

• PR add gpus support in docker provider #1886 (2021): Added Docker-specific --gpus flag. Rejected because it exposed runtime implementation details and wasn't portable across providers.
• PR Add GPU support #3257 (2023): Similar Docker-focused approach. Closed by author after feedback recommending CDI instead.
• PR Add API for CDI --devices flag in Docker and Podman for mapping GPUs #3290 (2023-2025): Implemented CDI-based device mapping. Had serious discussion for 2+ years but was ultimately closed due to:
  • Heavy dependency tree (CDI library + transitive deps)
  • Concerns about maintaining kind's "minimal dependencies" philosophy
  • Merge conflicts after extended discussion period

Key Maintainer Concerns from Previous PRs

1. Runtime implementation leakage: Exposing that kind uses docker run with specific flags
2. Provider portability: Solutions must work across Docker, Podman, and nerdctl
3. Dependency minimalism: Avoid adding heavy dependency trees
4. CRI alignment: Preference for approaches aligned with Kubernetes CRI patterns

Proposed Solution

Add a simple, provider-agnostic gpus field to the node configuration that maps to provider-specific runtime flags internally.

User Configuration (Provider-Agnostic)

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: worker
    gpus: "all"  # Start with "all", extensible to specific GPUs later

Internal Provider Mapping

The gpus field would be translated to provider-specific flags:

• Docker: --gpus all
• Podman: --device nvidia.com/gpu=all (CDI syntax, no library dependency)
• nerdctl: --gpus all

Implementation Scope

1. Add GPUs string field to Node struct in config API
2. Validate that only "all" is supported initially
3. Implement flag generation in all three providers (docker/podman/nerdctl)
4. Run make generate for deepcopy methods
5. Add unit tests for validation and flag generation

Estimated changes: ~100 lines across 6 files, no new dependencies.

Why This Approach

Advantages

• Cross-provider: Works on Docker, Podman, and nerdctl from day one
• Zero dependencies: Simple string matching, no CDI library needed
• Follows existing patterns: Mirrors extraMounts and extraPortMappings
• Extensible: String field allows future support like "0,1" for specific GPUs
• Minimal code: Small, maintainable change

Acknowledged Limitations

• Still exposes runtime details: Like extraPortMappings, this reveals we use container runtime CLI
• Not fully CRI-aligned: Kubernetes uses device plugins, not runtime flags
• Runtime-specific behavior: Each provider may handle GPUs slightly differently

The Core Question for Maintainers

Is a cross-provider, zero-dependency GPU field acceptable despite exposing runtime implementation details?

The alternative (full CDI library integration) was discussed in PR #3290 but raised dependency concerns. Given that:

• Issue Support GPUs #3164 shows GPU support is desired (milestone v0.20.0)
• The "pure" approach (CDI library) adds unwanted dependencies
• Users are currently blocked on GPU workloads in kind

Would you accept a pragmatic solution that follows the extraMounts/extraPortMappings pattern, or would you prefer to wait for a more architecturally pure approach even if it means heavier dependencies?

Open Questions

1. Is the proposed API (gpus: "all") acceptable, or would you prefer a different structure?
2. Should this be per-node or cluster-wide configuration?
3. Are there concerns with the provider-specific flag mapping approach?
4. Would you want runtime version detection to provide better error messages?

Alternative Approaches Considered

1. Full CDI library integration (PR Add API for CDI --devices flag in Docker and Podman for mapping GPUs #3290): Rejected due to dependencies
2. Generic extraArgs field: Would expose too many runtime internals
3. Cluster-wide GPU setting: Less flexible than per-node configuration
4. DeviceRequests API: Would require runtime-specific API calls, not exec-based

---

I'd appreciate feedback on whether this approach addresses your concerns from the previous GPU PRs, or if there are architectural issues that would prevent merging regardless of implementation quality.

References:

• Issue Support GPUs #3164
• PR add gpus support in docker provider #1886 (closed - Docker-only)
• PR Add GPU support #3257 (closed - Docker-only)
• PR Add API for CDI --devices flag in Docker and Podman for mapping GPUs #3290 (closed - CDI dependencies)

@BenTheElder @aojea

[BenTheElder]

Still exposes runtime details: Like extraPortMappings, this reveals we use container runtime CLI

extraPortMappings is a complete abstraction that can be backed by the APIs and not merely the CLI, the port mapping config type is actually based on CRI

Validate that only "all" is supported initially

I don't think most users want "all", so we should probably not add a feature without a credible path to selecting less than all?

Would you accept a pragmatic solution that follows the extraMounts/extraPortMappings pattern, or would you prefer to wait for a more architecturally pure approach even if it means heavier dependencies?

I think this is a false dichotomy. We should be able to pass CDI config without heavy dependencies.

But that's not the actual problem anyhow, see the discussions on the CDI PR. Roughly: because --privileged is not fully available with granular alternatives, we have to use that (or API equivalent) to create the nodes, and that flag includes mounting all devices already.

The tricky part is how to communicate this and then find and remove unwanted devices from the node since all devices will be made available already simply by being a privileged container.

---

Users are currently blocked on GPU workloads in kind

This looks like it's still specific to nvidia GPUs, in which case you might consider the nvkind project from nvidia (which is based on kind) to immediately unblock. That takes additional liberties related to only targeting nvidia but gives you more capability than the proposed feature.

[BenTheElder]

nvkind depends on nvidia libraries to enumerate GPUs and nvidia specific config to specify which GPUs are desired, then removes the undesired GPUs early after the container starts, this requires heavy vendor specific dependencies which also have to be hooked into the container entrypoint and is not portable. Something comparable is probably possible with CDI but I think most users have just gone with nvkind for now and we haven't seen much follow-up since.

They also support go text/template in the config, which I would very much prefer not to for all of the footguns that creates (it's pretty useful for simple internal templates but makes for a rather fragile API and one that is prone to injection attacks) but I understand why that project does this for the specific use case.

I think unforking in a way that isn't at-odds with key project goals here will be tricky but at the same time I think most people that would use an "give me all nvidia GPUs" option are better served by nvkind instead.

If we can find a way to give a similarly powerful experience without forcing all the existing users that don't care about nvidia to take in the dependency / binary bloat / startup penalty and avoid introducing vendor-specific configuration in what is otherwise a vendor-neutral upstream project (see also how core Kubernetes has DRA rather than nvidia config fields), then that is quite desirable. But I think it's a hard problem with limited demand 😅

As an aside: We develop DRA with kind, including nvidia engineers as far as I know, it's just usually developed with a mock driver in the Kubernetes repo which again isn't tied to any accelerator vendor. For developing nvidia applications that of course doesn't work, but our first and founding goal is developing Kubernetes itself.