Skip to content

standup.md

Latest commit

 

History

History
329 lines (268 loc) · 30.8 KB

standup.md

File metadata and controls

329 lines (268 loc) · 30.8 KB
Raw

Concept

llm-d-benchmark provides its own automated framework for the standup of stacks serving large language models in a Kubernetes cluster.

Motivation

In order to allow reproducible and flexible experiments, and taking into account that the configuration paramaters have significant impact on the overall performance, it is necessary to provide the user with the ability to standup and teardown stacks.

Methods

Currently, two main standup methods are supported a) "Standalone", with multiple VLLM pods controlled by a deployment behind a single service b) "llm-d", which leverages a combination of llm-d-infra and llm-d-modelservice to deploy a full-fledged llm-d stack

Scenarios

All the information required for the standup of a stack is contained on a "scenario file". This information is encoded in the form of environment variables, with default values defined in config/defaults.yaml which can be then overriden inside a scenario file (YAML-based) or via specification templates (Jinja2 .yaml.j2 files).

Multi-Stack Scenarios

A scenario may define more than one stack in its scenario: list. Standup iterates every per-stack step across all stacks (in parallel, bounded by --parallel), so you can stand up N models behind one gateway in a single llmdbenchmark standup invocation. Scenario-wide config (gateway class, WVA controller, shared HTTPRoute, chart versions) lives in an optional top-level shared: block that's merged into every stack before per-stack overrides.

Cluster-scoped infrastructure that would race with itself across N parallel standup executions is deduplicated at render time - only the first stack emits the istio control-plane helmfile and the infra-llmdbench Helm release; subsequent stacks render empty files for those templates. WVA controller installation is deduplicated at the step level (one per wva.namespace).

Currently shipped multi-stack example:

  • examples/multi-model-wva - two models (Qwen3-0.6B + Meta-Llama-3.1-8B), each with its own EPP + InferencePool + VariantAutoscaling + HPA, one shared WVA controller, one HTTPRoute with two backendRefs routing by path prefix (/qwen3-06b/* -> Qwen pool, /llama-31-8b/* -> Llama pool).

See config/README.md for the shared: merge semantics and the developer guide's Multi-Stack Scenarios section for the render-engine details.

--stack NAME[,NAME...] (also LLMDBENCH_STACK=NAME) restricts standup to a subset of rendered stacks - handy for re-deploying a single pool after a scenario edit without tearing down siblings. Global steps (cluster admin prereqs, shared-infra helmfile, WVA controller install, scenario-wide PVCs) still run as usual; only per-stack steps (06+ for standup) are filtered. Unknown names fail loudly with a list of valid ones.

# One stack:
llmdbenchmark --spec examples/multi-model-wva standup -p my-namespace --stack qwen3-06b

# Multiple named stacks (comma-separated):
llmdbenchmark --spec examples/multi-model-wva standup -p my-namespace --stack qwen3-06b,llama-31-8b

The same flag works on smoketest, run, and teardown with identical semantics, so you can scope every lifecycle phase to the same subset.

Multiple steps

The full standup of a stack is a multi-step process. The lifecycle document go into more details explaning the meaning of each different individual step.

Use

A scenario file has to be manually crafted as a YAML file. Once crafted, it can be used by llmdbenchmark standup, llmdbenchmark run or llmdbenchmark teardown commands. Its access is controlled by the following parameters.

Note

llmdbenchmark experiment is a command that combines llmdbenchmark standup, llmdbenchmark run and llmdbenchmark teardown into a single operation. Therefore, the command line parameters supported by the former is a combination of the latter three.

The scenario parameters can be roughly categorized in four groups:

  • Target-specific (Cluster API access, authentication tokens, standup methods and models)
Variable Meaning Note
LLMDBENCH_CLUSTER_URL URL to API access to Kubernetes cluster "auto" means "current" (e.g. ~/.kube/config) is used
LLMDBENCH_CLUSTER_TOKEN Used to authenticate to the cluster Ignored for LLMDBENCH_CLUSTER_URL="auto"
LLMDBENCH_HF_TOKEN Hugging face token Required for gated models; optional for public models (auto-detected)
LLMDBENCH_DEPLOY_SCENARIO File containing multiple environment variables which will override defaults If not specified, defaults to (empty) none.yaml. Can be overriden with CLI parameter -c/--scenario
LLMDBENCH_DEPLOY_MODEL_LIST List (comma-separated values) of models to be run against Default=meta-llama/Llama-3.2-1B-Instruct. Can be overriden with CLI parameter -m/--models
LLMDBENCH_DEPLOY_METHODS List (comma-separated values) of standup methods Default=modelservice. Can be overriden with CLI parameter -t/--methods

Tip

In case the full path is ommited for the scenario file (either by setting LLMDBENCH_DEPLOY_SCENARIO or CLI parameter -c/--scenario, it is assumed that the scenario exists inside the config/scenarios folder

  • "Common" VLLM parameters, applicable to any standup method
Variable Meaning Note
LLMDBENCH_VLLM_COMMON_NAMESPACE Namespace where stack gets stood up Default=llmdbench. Can be overriden with CLI parameter -p/--namespace
LLMDBENCH_IGNORE_FAILED_VALIDATION Ignore failed sanity checks and continue to deployment Default=True. Capacity Planner will perform a sanity check on vLLM parameters such as valid TP, max-model-len, KV cache availability.
LLMDBENCH_VLLM_COMMON_ACCELERATOR_MEMORY GPU memory for LLMDBENCH_VLLM_COMMON_ACCELERATOR_RESOURCE (e.g. 80) Default=auto, will try to guess GPU memory from LLMDBENCH_VLLM_COMMON_ACCELERATOR_RESOURCE
LLMDBENCH_VLLM_COMMON_SERVICE_ACCOUNT Service Account for stack
LLMDBENCH_VLLM_COMMON_ACCELERATOR_RESOURCE Accelerator type (e.g., nvidia.com/gpu) "auto" means, will query the cluster to discover
LLMDBENCH_VLLM_COMMON_NETWORK_RESOURCE Network type (e.g., rdma/roce_gdr)
LLMDBENCH_VLLM_COMMON_VLLM_ALLOW_LONG_MAX_MODEL_LEN
LLMDBENCH_VLLM_COMMON_VLLM_SERVER_DEV_MODE e.g., 0, 1
LLMDBENCH_VLLM_COMMON_VLLM_LOAD_FORMAT e.g., safetensors, tensorizer, runai_streamer, fastsafetensors
LLMDBENCH_VLLM_COMMON_VLLM_LOGGING_LEVEL e.g., DEBUG, INFO, WARNING
LLMDBENCH_VLLM_COMMON_ENABLE_SLEEP_MODE e.g., true, false
LLMDBENCH_VLLM_COMMON_NETWORK_NR
LLMDBENCH_VLLM_COMMON_AFFINITY
LLMDBENCH_VLLM_COMMON_REPLICAS
LLMDBENCH_VLLM_COMMON_TENSOR_PARALLELISM
LLMDBENCH_VLLM_COMMON_DATA_PARALLELISM
LLMDBENCH_VLLM_COMMON_ACCELERATOR_NR
LLMDBENCH_VLLM_COMMON_ACCELERATOR_MEM_UTIL
LLMDBENCH_VLLM_COMMON_CPU_NR
LLMDBENCH_VLLM_COMMON_CPU_MEM
LLMDBENCH_VLLM_COMMON_MAX_MODEL_LEN
LLMDBENCH_VLLM_COMMON_BLOCK_SIZE
LLMDBENCH_VLLM_COMMON_MAX_NUM_BATCHED_TOKENS
LLMDBENCH_VLLM_COMMON_PVC_NAME
LLMDBENCH_VLLM_COMMON_PVC_STORAGE_CLASS
LLMDBENCH_VLLM_COMMON_PVC_MODEL_CACHE_SIZE
LLMDBENCH_VLLM_COMMON_PVC_DOWNLOAD_TIMEOUT
LLMDBENCH_VLLM_COMMON_HF_TOKEN_KEY
LLMDBENCH_VLLM_COMMON_HF_TOKEN_NAME
LLMDBENCH_VLLM_COMMON_INFERENCE_PORT
LLMDBENCH_VLLM_COMMON_FQDN
LLMDBENCH_VLLM_COMMON_TIMEOUT
LLMDBENCH_VLLM_COMMON_ANNOTATIONS
LLMDBENCH_VLLM_COMMON_ENVVARS_TO_YAML
LLMDBENCH_VLLM_COMMON_INITIAL_DELAY_PROBE
LLMDBENCH_VLLM_COMMON_POD_SCHEDULER
  • "Standalone"-specific VLLM parameters
Variable Meaning Note
LLMDBENCH_VLLM_COMMON_MODEL_LOADER_EXTRA_CONFIG
LLMDBENCH_VLLM_STANDALONE_PVC_MOUNTPOINT
LLMDBENCH_VLLM_STANDALONE_PREPROCESS e.g., source /setup/preprocess/standalone-preprocess.sh ; /setup/preprocess/standalone-preprocess.py
LLMDBENCH_VLLM_STANDALONE_ROUTE
LLMDBENCH_VLLM_STANDALONE_HTTPROUTE
LLMDBENCH_VLLM_STANDALONE_ARGS
LLMDBENCH_VLLM_STANDALONE_EPHEMERAL_STORAGE
  • Gateway provider
Variable Meaning Note
LLMDBENCH_VLLM_MODELSERVICE_GATEWAY_CLASS_NAME Gateway implementation used for the inference gateway Default=istio. Supported: istio, agentgateway, gke, data-science-gateway-class, epponly.

Gateway class options (set via gateway.className in the scenario YAML):

className What it deploys Use when
istio (default) istio-base + istiod control plane, a Gateway + HTTPRoute, the llm-d-router-gateway-dev chart Default; most flexible / production deployments
agentgateway agentgateway-crds + agentgateway controller, a Gateway + HTTPRoute, the llm-d-router-gateway-dev chart Want agentgateway's data plane instead of Envoy/Istio
gke Uses GKE-managed Gateway controller; same llm-d-router-gateway-dev chart Running on GKE
data-science-gateway-class OpenDataHub / OpenShift AI managed Gateway Running on OpenShift AI
epponly No Kubernetes Gateway, no HTTPRoute, the llm-d-router-standalone-dev chart (EPP with an Envoy sidecar serving HTTP) You want llm-d's standalone router topology without any gateway

Overriding gateway.className from the CLI

Every subcommand that renders templates (plan, standup, experiment, and the run/smoketest/teardown paths that re-render for setup overrides) accepts a --gateway-class flag that overrides the scenario's gateway.className for that invocation. The same value can be supplied via the LLMDBENCH_GATEWAY_CLASS environment variable.

# Scenario default is epponly -- flip to istio without editing YAML
llmdbenchmark --spec guides/optimized-baseline standup -p my-ns --gateway-class istio

# env-var form (matches the LLMDBENCH_* convention)
LLMDBENCH_GATEWAY_CLASS=agentgateway \
  llmdbenchmark --spec guides/optimized-baseline standup -p my-ns

Precedence (highest wins): --gateway-class CLI flag → scenario gateway.className → defaults.yaml (istio).

Method-aware validation

gateway.className only affects rendering when the active deploy method is modelservice. For kustomize, standalone, and fma the gateway block is ignored by every rendered template, so the CLI accepts any value (including sentinels like none or n/a that CI scripts often pass uniformly across deploy methods). The banner shows it as Gateway: <value> (ignored -- modelservice is not the active deploy method).

When modelservice is the active method, the override is checked against the whitelist of supported values above and a typo fails fast at plan time:

ValueError: --gateway-class='isto' is not a supported value for the
modelservice deploy method. Choose one of: epponly, istio, agentgateway,
gke, data-science-gateway-class.

This lets CI workflows pass --gateway-class=$SOME_VAR for every deploy method without per-method special-casing, while still catching typos when the value actually matters.

Switching from Istio to agentgateway

By default, llm-d-benchmark deploys Istio as the gateway provider for the modelservice deployment method. To use agentgateway instead, add a gateway block to your scenario YAML:

scenario:
  - name: "my-stack"
    gateway:
      className: agentgateway       # default is "istio"

    modelservice:
      enabled: true
    # ... rest of scenario config

That single change is all that's needed. The benchmark tool handles everything else automatically:

  1. Installs agentgateway -- the controller and CRDs are installed via helmfile during step 02 (admin prerequisites), the same way Istio is installed
  2. Configures the Gateway resource -- the llm-d-infra Helm chart creates a Gateway with gatewayClassName: agentgateway
  3. OpenShift SCC -- on OpenShift clusters, a minimal custom SCC (llmdbench-agentgateway) is automatically created and granted to the gateway service account, allowing the proxy to run as UID 10101 with NET_BIND_SERVICE

Differences from Istio

Aspect Istio agentgateway
Gateway pod creation Created by the llm-d-infra Helm chart directly Created dynamically by the agentgateway controller
gatewayParameters Uses ConfigMap-based parametersRef Not used -- agentgateway manages its own AgentgatewayParameters CRD
OpenShift compatibility Built-in via floatingUserId (uses namespace UID range) Requires custom SCC (auto-created by the tool)
Service name infra-{release}-inference-gateway-istio infra-{release}-inference-gateway

Example scenarios using agentgateway

EPP-only (llm-d standalone router) mode (gateway.className: epponly)

epponly mirrors the Standalone Mode documented in llm-d (guides/recipes/router/README.md) and used by every well-lit-path guide (e.g. optimized-baseline). The EPP is deployed via the llm-d-owned oci://ghcr.io/llm-d/charts/llm-d-router-standalone-dev chart (migrated from the upstream GAIE-published standalone chart), which adds an Envoy sidecar to the EPP pod so HTTP traffic can hit the EPP service directly -- no Kubernetes Gateway, no HTTPRoute, no llm-d-infra Helm release.

scenario:
  - name: "my-stack"
    gateway:
      className: epponly      # default is "istio"

    modelservice:
      enabled: true
    # ... rest of scenario config

When epponly is selected, standup automatically:

  1. Skips the gateway provider install -- step 02 does not install istio or agentgateway CRDs / controllers.
  2. Skips the llm-d-infra Helm release -- no Gateway resource is created.
  3. Skips HTTPRoute rendering -- nothing references a Gateway.
  4. Swaps the router chart to the llm-d-router-standalone-dev chart, which bundles the EPP + Envoy sidecar in a single pod.
  5. Adds a port 80 -> targetPort 8081 extraServicePort to the EPP service so HTTP requests reach the Envoy sidecar.
  6. Points endpoint discovery at {model_id_label}-router-epp:80 -- the smoketest and run phase resolve to the EPP service directly instead of a Gateway IP.

Restrictions

  • Single-stack only. epponly cannot multiplex multiple models since it has no shared Gateway / HTTPRoute. Multi-stack scenarios fail at render time with a clear error.
  • httpRoute.mode: shared is rejected -- shared HTTPRoute requires a Gateway that epponly does not deploy.
  • Only the modelservice deploy method. Combining epponly with standalone.enabled: true, fma.enabled: true, or kustomize.enabled: true is rejected at render time.

Differences from a gateway-based deployment

Aspect Gateway-based (istio / agentgateway / gke) epponly
Router Helm chart llm-d-router-gateway-dev llm-d-router-standalone-dev
llm-d-infra release Installed (creates Gateway) Skipped (no Gateway needed)
HTTPRoute Rendered Not rendered
Provider control plane istio / agentgateway controller installed via helmfile Not installed
Endpoint Gateway resource IP {model_id_label}-router-epp Service ClusterIP, port 80
Number of EPP replicas Configurable 1 (matches default router.epp.replicas: 1)
Multi-stack support Yes No (single-stack only)

Example scenario using epponly

  • config/scenarios/guides/optimized-baseline.yaml -- ships with gateway.className: epponly as the default. Flip it to istio/agentgateway/gke/data-science-gateway-class to switch topology without touching anything else in the scenario.

  • "llm-d"-specific VLLM paramaters

Variable Meaning Note
LLMDBENCH_VLLM_INFRA_CHART_NAME
LLMDBENCH_VLLM_INFRA_CHART_VERSION
LLMDBENCH_VLLM_INFRA_GATEWAY_CPU_REQUEST Gateway CPU request Default=4
LLMDBENCH_VLLM_INFRA_GATEWAY_CPU_LIMIT Gateway CPU limit Default=16
LLMDBENCH_VLLM_INFRA_GATEWAY_MEMORY_REQUEST Gateway memory request Default=4Gi
LLMDBENCH_VLLM_INFRA_GATEWAY_MEMORY_LIMIT Gateway memory limit Default=16Gi
LLMDBENCH_VLLM_GAIE_CHART_NAME
LLMDBENCH_VLLM_GAIE_CHART_VERSION
LLMDBENCH_VLLM_MODELSERVICE_RELEASE
LLMDBENCH_VLLM_MODELSERVICE_VALUES_FILE
LLMDBENCH_VLLM_MODELSERVICE_ADDITIONAL_SETS
LLMDBENCH_VLLM_MODELSERVICE_CHART_VERSION
LLMDBENCH_VLLM_MODELSERVICE_CHART_NAME
LLMDBENCH_VLLM_MODELSERVICE_HELM_REPOSITORY
LLMDBENCH_VLLM_MODELSERVICE_HELM_REPOSITORY_URL
LLMDBENCH_VLLM_MODELSERVICE_URI_PROTOCOL
LLMDBENCH_VLLM_MODELSERVICE_DECODE_INFERENCE_PORT
LLMDBENCH_VLLM_MODELSERVICE_GATEWAY_CLASS_NAME
LLMDBENCH_VLLM_MODELSERVICE_ROUTE
LLMDBENCH_VLLM_MODELSERVICE_EPP
LLMDBENCH_VLLM_MODELSERVICE_INFERENCE_MODEL
LLMDBENCH_VLLM_MODELSERVICE_INFERENCE_POOL
LLMDBENCH_VLLM_MODELSERVICE_GAIE_PLUGINS_CONFIGFILE
LLMDBENCH_VLLM_MODELSERVICE_GAIE_MONITORING_PROMETHEUS_ENABLED Enable Prometheus ServiceMonitor for GAIE EPP component metrics true (default) or false false