Benchmarking on existing endpoint - run only model - fails during report conversion

[Sneha-at]

Component

Run

Describe the bug

I am running a new benchmark with inference-perf harness using run only mode. I have model server and llm-d components deployed and reachable. The benchmark run completes the run and logs performance numbers but later fails during conversion from inference-perf report to llm-d-benchmark report with below error

Command

export IP=xx.xx.xx
MODEL=Qwen/Qwen3-32B
NAMESPACE=llm-d-storage
WORKLOAD=code_generation.yaml.in # I have added this file in my local


llmdbenchmark --spec gpu run   --endpoint-url http://$IP   --model $MODEL   --namespace $NAMESPACE   --harness inference-perf   --workload $WORKLOAD

Steps to reproduce

1. Make sure model server pods are up and accessible
2. Run benchmark with this command

export IP=10.128.0.22 
MODEL=Qwen/Qwen3-32B
NAMESPACE=llm-d-storage
WORKLOAD=code_generation.yaml.in # I have added this file in my local

llmdbenchmark --spec gpu run   --endpoint-url http://$IP   --model $MODEL   --namespace $NAMESPACE   --harness inference-perf   --workload $WORKLOAD

Additional context or screenshots

No response

[Vezio]

Can you supply the local file named code_generation.yaml.in I will try to reproduce.

[Sneha-at]

Sure, here's details for code_generation.yaml.in

load:
  type: concurrent
  stages:
  - num_requests: 120
    concurrency_level: 4
  - num_requests: 120
    concurrency_level: 8
  num_workers: 1
api:
  type: completion
  streaming: true
server:
  type: vllm
  model_name: REPLACE_ENV_LLMDBENCH_DEPLOY_CURRENT_MODEL
  base_url: REPLACE_ENV_LLMDBENCH_HARNESS_STACK_ENDPOINT_URL
data:
  type: conversation_replay
  conversation_replay:
    seed: 42
    num_conversations: 20
    shared_system_prompt_len: 3000
    dynamic_system_prompt_len:
      type: uniform
      min: 15000
      max: 120000
      mean: 55000
      std_dev: 25000
    turns_per_conversation:
      type: normal
      min: 2
      max: 20
      mean: 6
      std_dev: 3.5
    input_tokens_per_turn:
      type: lognormal
      min: 100
      max: 10000
      mean: 1500
      std_dev: 1200
    output_tokens_per_turn:
      type: lognormal
      min: 50
      max: 4000
      mean: 800
      std_dev: 600
    tool_call_latency_sec:
      type: lognormal
      min: 15
      max: 120
      mean: 45
      std_dev: 25

metrics:
  type: prometheus
  prometheus:
    google_managed: true
    scrape_interval: 15

report:
  request_lifecycle:
    summary: true
    per_stage: true
    per_request: false
  prometheus:
    summary: true
    per_stage: true

[Vezio]

I just tried to reproduce and I was not able to hit your error.

I will say, I did have to remove the prometheus section:

metrics:
  type: prometheus
  prometheus:
    google_managed: true
    scrape_interval: 15

Since we don't use google_managed: true.

@Sneha-at I'd love if you could send me a full set of logs - i.e., all of the terminal output, and everything captured in the workspace.

That would help me debug this further, feel free to dm on slack as we have been conversing there.

[Sneha-at]

llmdbenchmark   --spec gpu run   --endpoint-url http://$IP   --model $MODEL   --namespace $NAMESPACE   --harness inference-perf   --workload $WORKLOAD
2026-06-23 22:53:03,308 - INFO    - Specification resolved: gpu to /usr/local/google/home/snehaaradhey/go/src/llm-d/llm-d-benchmark/config/specification/examples/gpu.yaml.j2
2026-06-23 22:53:03,308 - INFO    - Using Package: "llmdbenchmark:0.6.0" found at https://github.com/llm-d/llm-d-benchmark
2026-06-23 22:53:03,308 - INFO    - ✅ Created Workspace: "/usr/local/google/home/snehaaradhey/data/gpu"
2026-06-23 22:53:03,308 - INFO    - ✅ Created llmdbenchmark instance in workspace: "/usr/local/google/home/snehaaradhey/data/gpu/snehaaradhey-20260623-225303-306"


2026-06-23 22:53:03,313 - INFO    - ✅ Specification file rendered and validated successfully.
2026-06-23 22:53:03,397 - INFO    - Processing scenario with 1 stack(s)...


2026-06-23 22:53:03,424 - INFO    - Loaded 29 template(s) from /usr/local/google/home/snehaaradhey/go/src/llm-d/llm-d-benchmark/config/templates/jinja
2026-06-23 22:53:03,424 - INFO    - [1/1] Processing stack: gpu-example
2026-06-23 22:53:03,426 - INFO    - Image override: benchmark pinned to ghcr.io/llm-d/llm-d-benchmark:v0.6.7
2026-06-23 22:53:03,426 - INFO    - Image override: vllm pinned to docker.io/vllm/vllm-openai:v0.20.2
2026-06-23 22:53:03,426 - INFO    - Image override: vllmOpenai pinned to docker.io/vllm/vllm-openai:v0.20.2
2026-06-23 22:53:03,426 - INFO    - Image override: routerEndpointPicker pinned to ghcr.io/llm-d/llm-d-router-endpoint-picker-dev:main
2026-06-23 22:53:03,426 - INFO    - Image override: inferenceScheduler pinned to ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0
2026-06-23 22:53:03,426 - INFO    - Image override: routingSidecar pinned to ghcr.io/llm-d/llm-d-routing-sidecar:v0.8.0
2026-06-23 22:53:03,426 - INFO    - Image override: udsTokenizer pinned to ghcr.io/llm-d/llm-d-uds-tokenizer:v0.8.0
2026-06-23 22:53:03,426 - INFO    - Image override: python pinned to python:3.10
2026-06-23 22:53:03,427 - INFO    - 🔍 Resolving image tag for: docker.io/vllm/vllm-openai
2026-06-23 22:53:04,327 - INFO    - 📦 Resolved docker.io/vllm/vllm-openai to vllm-minimax-m3-perf-x86_64-13.0.1-7a67223 (via crane)
2026-06-23 22:53:04,327 - INFO    - 🔍 Resolving image tag for: ghcr.io/llm-d/llm-d-workload-variant-autoscaler
2026-06-23 22:53:04,765 - INFO    - 📦 Resolved ghcr.io/llm-d/llm-d-workload-variant-autoscaler to ref-da0002bd (via crane)
2026-06-23 22:53:04,766 - INFO    - 🔍 Resolving chart version for: llmDInfra
2026-06-23 22:53:06,369 - INFO    - 📦 Resolved chart llmDInfra to v1.4.0 (via repo URL)
2026-06-23 22:53:06,369 - INFO    - 🔍 Resolving chart version for: llmDModelservice
2026-06-23 22:53:07,211 - INFO    - 📦 Resolved chart llmDModelservice to v0.4.15 (via repo URL)
2026-06-23 22:53:07,213 - INFO    - Namespace from CLI: deploy=llm-d-storage, harness=llm-d-storage, wva=llm-d-storage
2026-06-23 22:53:07,214 - INFO    - ℹ️ No HuggingFace token found -- HF secret will not be created. Public models will work; gated models will fail at standup.
2026-06-23 22:53:07,298 - INFO    - ✅ Rendered: 01_pvc_workload-pvc.yaml
2026-06-23 22:53:07,377 - INFO    - ✅ Rendered: 02_pvc_model-pvc.yaml
2026-06-23 22:53:07,452 - INFO    - ✅ Rendered: 03_cluster-monitoring-config.yaml
2026-06-23 22:53:07,532 - INFO    - ✅ Rendered: 04_download_job.yaml
2026-06-23 22:53:07,613 - INFO    - ✅ Rendered: 05_namespace_sa_rbac_secret.yaml
2026-06-23 22:53:07,688 - INFO    - ✅ Rendered: 05a_agentgateway_scc.yaml
2026-06-23 22:53:07,767 - INFO    - ✅ Rendered: 06_pod_access_to_harness_data.yaml
2026-06-23 22:53:07,843 - INFO    - ✅ Rendered: 07_service_access_to_harness_data.yaml
2026-06-23 22:53:07,928 - INFO    - ✅ Rendered: 08_httproute.yaml
2026-06-23 22:53:08,011 - INFO    - ✅ Rendered: 09_helmfile-gateway-provider.yaml
2026-06-23 22:53:08,089 - INFO    - ✅ Rendered: 10_helmfile-main.yaml
2026-06-23 22:53:08,165 - INFO    - ✅ Rendered: 11_infra.yaml
2026-06-23 22:53:08,258 - INFO    - ✅ Rendered: 12_router-values.yaml
2026-06-23 22:53:08,514 - INFO    - ✅ Rendered: 13_ms-values.yaml
2026-06-23 22:53:08,693 - INFO    - ✅ Rendered: 14_standalone-deployment_yaml.yaml
2026-06-23 22:53:08,773 - INFO    - ✅ Rendered: 15_standalone-service_yaml.yaml
2026-06-23 22:53:08,849 - INFO    - ✅ Rendered: 16_pvc_extra-pvc.yaml
2026-06-23 22:53:08,922 - INFO    - ✅ Rendered: 17_standalone-podmonitor.yaml
2026-06-23 22:53:08,996 - INFO    - ✅ Rendered: 18_podmonitor.yaml
2026-06-23 22:53:09,073 - INFO    - ✅ Rendered: 19_wva-values.yaml
2026-06-23 22:53:09,159 - INFO    - ✅ Rendered: 20_harness_pod.yaml
2026-06-23 22:53:09,229 - INFO    - ✅ Rendered: 21_prometheus-adapter-values.yaml
2026-06-23 22:53:09,301 - INFO    - ✅ Rendered: 22_prometheus-rbac.yaml
2026-06-23 22:53:09,372 - INFO    - ✅ Rendered: 23_wva-namespace.yaml
2026-06-23 22:53:09,458 - INFO    - ✅ Rendered: 24_fma-deployment.yaml
2026-06-23 22:53:09,529 - INFO    - ✅ Rendered: 25_fma-clusterrole.yaml
2026-06-23 22:53:09,603 - INFO    - ✅ Rendered: 26_helmfile-fma-controllers.yaml
2026-06-23 22:53:09,681 - INFO    - ✅ Rendered: 27_wva-variantautoscaling.yaml
2026-06-23 22:53:09,756 - INFO    - ✅ Rendered: 28_wva-hpa.yaml
2026-06-23 22:53:09,909 - INFO    - Output: /usr/local/google/home/snehaaradhey/data/gpu/snehaaradhey-20260623-225303-306/plan/gpu-example
2026-06-23 22:53:09,909 - INFO    - Success: 29, Errors: 0


2026-06-23 22:53:09,909 - INFO    - Scenario rendering complete! Output in: /usr/local/google/home/snehaaradhey/data/gpu/snehaaradhey-20260623-225303-306/plan
2026-06-23 22:53:12,385 - INFO    - 📄 Pre-rendered modelservice Helm manifests (365 lines) → gpu-example/helm/modelservice.yaml
2026-06-23 22:53:12,444 - INFO    - Auto-detected deploy method(s) from plan: modelservice
2026-06-23 22:53:14,212 - INFO    - Extracted current kubeconfig context context.ctx
2026-06-23 22:53:14,212 - INFO    - Using kubeconfig: /usr/local/google/home/snehaaradhey/data/gpu/snehaaradhey-20260623-225303-306/environment/context.ctx


2026-06-23 22:53:14,282 - INFO    - 📋 Executing 3 global step(s)...


2026-06-23 22:53:14,282 - INFO    - >> [00] Validate cluster and run-phase prerequisites


2026-06-23 22:53:14,282 - INFO    -     | Starting benchmark run phase
2026-06-23 22:53:14,282 - INFO    -     | Output destination: local (/usr/local/google/home/snehaaradhey/data/gpu/snehaaradhey-20260623-225303-306/results)
2026-06-23 22:53:14,282 - INFO    - ✅ [00] Completed: run_preflight
2026-06-23 22:53:14,283 - INFO    - >> [01] Clean up leftover harness pods from previous run
2026-06-23 22:53:14,340 - INFO    -     | Cleaning up previous harness pods (label=llmdbench-harness-launcher, ns=llm-d-storage)...
2026-06-23 22:53:14,697 - INFO    - ✅ [01] Completed: run_cleanup_previous
2026-06-23 22:53:14,697 - INFO    - >> [02] Prepare harness namespace (PVC, data access pod)
2026-06-23 22:53:15,333 - INFO    -     | HF token not configured -- skipping secret creation
2026-06-23 22:53:15,688 - INFO    -     | PVC 'workload-pvc' already exists with correct size (20Gi) ✓
2026-06-23 22:53:15,689 - INFO    -     | Using existing workload PVC 'workload-pvc'
2026-06-23 22:53:16,381 - INFO    -     | ⏭️  workload PVC "workload-pvc": StorageClass uses WaitForFirstConsumer -- PVC will bind when its consumer pod schedules; skipping bind wait.
2026-06-23 22:53:17,613 - INFO    -     | 🚚 Creating configmap with preprocess scripts...
2026-06-23 22:53:18,415 - INFO    -     | 📦 ConfigMap "llm-d-benchmark-preprocesses" created in ns/llm-d-storage with 7 file(s):
2026-06-23 22:53:18,416 - INFO    -     |     │ benchmark_nixl.py (24.7 KB)
2026-06-23 22:53:18,416 - INFO    -     |     │ gemini-arp-fix.sh (1.5 KB)
2026-06-23 22:53:18,416 - INFO    -     |     │ install_nsys.sh (0.3 KB)
2026-06-23 22:53:18,416 - INFO    -     |     │ set_llmdbench_environment.py (30.5 KB)
2026-06-23 22:53:18,416 - INFO    -     |     │ show_gids.sh (2.1 KB)
2026-06-23 22:53:18,416 - INFO    -     |     │ standalone-preprocess.py (7.1 KB)
2026-06-23 22:53:18,416 - INFO    -     |     │ standalone-preprocess.sh (3.3 KB)
2026-06-23 22:54:12,414 - INFO    -     | ✅ harness data-access pod: 1/1 Ready (00:53)
2026-06-23 22:54:12,414 - INFO    - ✅ [02] Completed: harness_namespace


2026-06-23 22:54:12,414 - INFO    - 📋 Executing 7 per-stack step(s) across 1 stack(s) (max parallel: 1)...


2026-06-23 22:54:12,414 - INFO    - Stack 'gpu-example': starting 7 step(s)...
2026-06-23 22:54:12,414 - INFO    - >> [03] Stack 'gpu-example': Detect model serving endpoint
2026-06-23 22:54:12,415 - INFO    -     | Using explicit endpoint URL: http://10.128.0.22 (stack_type=llm-d)
2026-06-23 22:54:12,415 - INFO    - ✅ [03] Stack 'gpu-example': Completed: detect_endpoint
2026-06-23 22:54:12,415 - INFO    - >> [04] Stack 'gpu-example': Verify model is served at endpoint
2026-06-23 22:54:12,475 - INFO    -     | Verifying model 'Qwen/Qwen3-32B' at http://10.128.0.22...
2026-06-23 22:54:23,198 - INFO    -     | Model 'Qwen/Qwen3-32B' verified at http://10.128.0.22
2026-06-23 22:54:23,198 - INFO    - ✅ [04] Stack 'gpu-example': Completed: verify_model
2026-06-23 22:54:23,198 - INFO    - >> [05] Stack 'gpu-example': Render workload profile templates
2026-06-23 22:54:23,265 - INFO    -     | Resolved workload profiles source: ~/go/src/llm-d/llm-d-benchmark/workload/profiles/inference-perf
2026-06-23 22:54:23,268 - INFO    -     | Rendered profile: code_generation.yaml
2026-06-23 22:54:23,268 - INFO    -     | Profiles rendered to /usr/local/google/home/snehaaradhey/data/gpu/snehaaradhey-20260623-225303-306/workload/profiles/inference-perf
2026-06-23 22:54:23,269 - INFO    - ✅ [05] Stack 'gpu-example': Completed: render_profiles
2026-06-23 22:54:23,269 - INFO    - >> [06] Stack 'gpu-example': Create profile and harness-scripts ConfigMaps
2026-06-23 22:54:23,329 - INFO    -     | Creating ConfigMap 'inference-perf-profiles' with 1 profile(s) in ns=llm-d-storage...
2026-06-23 22:54:24,046 - INFO    -     | ConfigMap 'inference-perf-profiles' created with 1 profile(s)
2026-06-23 22:54:24,047 - INFO    -     | Creating ConfigMap 'llmdbench-harness-scripts' with 11 harness script(s) in ns=llm-d-storage...
2026-06-23 22:54:25,056 - INFO    -     | ConfigMap 'llmdbench-harness-scripts' created with 11 script(s)
2026-06-23 22:54:25,056 - INFO    - ✅ [06] Stack 'gpu-example': Completed: create_profile_configmap
2026-06-23 22:54:25,056 - INFO    - >> [07] Stack 'gpu-example': Deploy harness pod(s) for benchmark execution
2026-06-23 22:54:25,117 - INFO    -     | Running 1 treatment(s) x 1 parallel pod(s) for 'inference-perf' (sequential per treatment)...
2026-06-23 22:54:25,117 - INFO    - 🚀     | [1/1] Treatment 'default': deploying 1 pod(s)...
2026-06-23 22:54:25,882 - INFO    -     | Deployed pod 'inference-perf-j8j026bk' (experiment=inference-perf-1782255265-6jbwtp, parallel=1/1)
2026-06-23 22:54:25,882 - INFO    -     | Waiting for pods (label=app=llmdbench-harness-launcher) to start (timeout=3600s)...
2026-06-23 22:54:27,272 - INFO    -     | All pods are running
2026-06-23 22:54:27,272 - INFO    -     | Waiting for pods (label=app=llmdbench-harness-launcher) to complete (timeout=3600s)...
2026-06-23 23:14:35,784 - INFO    -     | Collecting results for 1 dir(s): inference-perf-1782255265-6jbwtp_1
2026-06-23 23:14:36,512 - INFO    -     | Collected 15 file(s) for inference-perf-1782255265-6jbwtp_1
2026-06-23 23:14:37,025 - INFO    -     | Deleted pod 'inference-perf-j8j026bk'
2026-06-23 23:14:37,025 - INFO    - ✅     | [1/1] Treatment 'default' complete (1211s)
2026-06-23 23:14:37,028 - ERROR   - ❌ [07] Stack 'gpu-example': FAILED: [07] deploy_harness (gpu-example): FAILED - Some treatments had errors


2026-06-23 23:14:37,028 - INFO    - 📋 Executing 3 global step(s)...


2026-06-23 23:14:37,029 - INFO    - -- [10] Skipping: upload_results (Upload results to cloud storage)
2026-06-23 23:14:37,029 - INFO    - >> [11] Clean up harness pods and ConfigMaps
2026-06-23 23:14:37,099 - INFO    -     | Cleaning up harness resources in ns=llm-d-storage...
2026-06-23 23:14:37,455 - INFO    -     | Harness pods deleted
2026-06-23 23:14:37,888 - INFO    -     | ConfigMap 'inference-perf-profiles' deleted
2026-06-23 23:14:38,338 - INFO    -     | ConfigMap 'llmdbench-harness-scripts' deleted
2026-06-23 23:14:38,338 - INFO    - ✅ [11] Completed: run_cleanup_post
2026-06-23 23:14:38,338 - INFO    - -- [12] Skipping: analyze_results (Run local analysis on collected results)
2026-06-23 23:14:38,340 - ERROR   - ❌ Run failed:
Phase: run
Stack 'gpu-example' failures:
  - [07] deploy_harness (gpu-example): FAILED - Some treatments had errors
    * Error: Found pods in error state. Run: kubectl --namespace llm-d-storage get pods -l app=llmdbench-harness-launcher

[Sneha-at]

This works if Google metrics aren't set

metrics:
  type: prometheus
  prometheus:
    google_managed: true
    scrape_interval: 15

@Vezio would be adding more details on this issue