Add feature to install llm-d infra using kustomize for teams preferring native Kubernetes tooling#248
Conversation
Signed-off-by: Avinash Singh <avinashsingh.rcoem@gmail.com>
avinashsingh77
requested review from
Gregory-Pereira,
nerdalert,
robertgshaw2-redhat and
tlrmchlsmth
as code owners
Gregory-Pereira
left a comment
Gregory-Pereira
left a comment
There was a problem hiding this comment.
Excellent job, especially with little guidance from me as I have been busy. Review the suggestions and architecture changes I discuss below - but based on them I think examples and overlays is a bit redundant, so I think we could consider removing examples but I am willing to hear arguments to the contrary. I think the Gateway API itself and various implementations on top of it are well enough known and popular that people could look at a central readme as a doc on how to extend it to their use-case rather than giving them explicit examples
There was a problem hiding this comment.
I think we could drop this manifest entirely, this was added early in the process and kind of defeats the purpose of the inference gateway as a component. It was kept on in the edge case where people want an inference gateway but could only use clusterIP as the service type, so it gave them a simple way to expose in hit the gateway - but definitely not production ready.
There was a problem hiding this comment.
Same as above, we can remove the whole directory
There was a problem hiding this comment.
This directory I think we can remove. It was an "environment" but from the infra perspective which will eventually map into just a gateway directory in the regular examples, the TPU pieces are irrelevant. That implementation would correspond to your PR in modelservice and be
So to clarify our pattern were aiming for, for deploying a specialized hardware type with a specific gateway (EX: GKE + TPU):
tree
.
├── gateway
│ ├── base
│ └── gke
└── modelserver
├── base
└── tpu
This is what the final setup in the llm-d guides will look like
| - op: add | ||
| path: /metadata/annotations | ||
| value: | ||
| cloud.google.com/neg: '{"exposed_ports": {"80":{}}}' |
There was a problem hiding this comment.
@liu-cong is this best practice for GKE? Is this something we should be doing
There was a problem hiding this comment.
Useful docs, while the directory should be deleted based on the above comment, this should find a home somewhere in the main repo
There was a problem hiding this comment.
One comment - this falls outside of the scope of this PR but I think its important to mention. Our Kgatway support is 2nd class, we need to find a declaritive (manifest) way to increase the number of concurrent connections that Kgateway supports from the default 1024. I know its possible with an API call or going through the Kgateway control plan UI. We should have some documentation on it at the very last, but its unrelated to the content of the PR so I can follow on with later.
There was a problem hiding this comment.
I think this is missing an https (443) listener in the gateway, which AFAIK is required, I dont think it works even if one were to configure a port 80 listener with HTTPs.
There was a problem hiding this comment.
Just a high level comment - this spec works for kgateway 2.1.X, on previous versions (like v2.0.4 which we used to use) need floatingUserId to allow the envoy container to spawn with a environment derived variable user, which is a requirement to meet OCP restricted-v2 SCCs.
This implementation will eventually land in the main repo, this pr is for gathering feedback, so this concern will go away but I think its important to call this out.
|
This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the |
Co-authored-by: Greg Pereira <grpereir@redhat.com>
|
Unsigned commits detected! Please sign your commits. For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation. |
2 participants
This PR adds Kustomize-based deployment support for llm-d-infra, providing an alternative to Helm charts for gateway infrastructure deployment.
The implementation supports all gateway providers (Istio, Kgateway, AgentGateway, GKE) with optional features as reusable components.
Kustomize structure
Gateway Providers (4 overlays):
Optional Features (2 components):
Production-Ready Examples(aligning with well-lit paths)
istio-minimal/: Basic Istio gateway deploymentistio-with-ingress/: Production Istio with external Ingress and custom resource limitsistio-with-tls/: Istio gateway with TLS/HTTPS supportkgateway-production/: Production-ready Kgateway deploymentgke-minimal/: Basic GKE gatewaygke-tpu/: GKE gateway optimized for TPU workloads with nodeSelector annotationsMigration Path
Existing Helm users are not affected - Helm charts remain fully supported. Kustomize provides an alternative deployment method for teams that prefer native Kubernetes tooling or require GitOps-friendly configuration management.
The
kustomize/MIGRATION.mdguide helps teams choose between Helm and Kustomize based on their specific needs.Testing
All overlays and examples have been tested to ensure:
kubectl kustomize