Benchmark image ships guidellm v0.6.0 which lacks replay profile support; error message is misleading

[asm582]

Summary

Two related problems with replay profile support in the benchmark harness:

1. The benchmark image ships guidellm v0.6.0, which does not include the replay profile. Using profile: replay in a workload YAML fails at runtime.
2. The resulting error message is misleading — instead of reporting an unsupported profile, guidellm v0.6.0 emits a Pydantic validation error (--target: Field required) that obscures the real cause.

Details

guidellm version

replay is implemented in vllm-project/guidellm@main (e.g. commit 73d91f311bd9) but has not been released to PyPI as of guidellm v0.6.0. Both the v0.6.2 and v0.7.0 benchmark images install guidellm v0.6.0.

Misleading error

When profile: replay is passed to guidellm v0.6.0, the actual failure is:

Invalid value for --profile: Input should be 'synchronous', 'concurrent', 'throughput', 'constant' or 'poisson'

However, this error is swallowed during argument parsing and surfaces as a different Pydantic error:

Error: Missing field '--target': Field required

This causes significant confusion — operators debug the --target argument rather than the unsupported profile.

Proposed Fixes

1. Image: Once a guidellm release with replay support is available on PyPI, update the benchmark image's dependency pin.
2. Error message: Add a validation step in the harness script (or guidellm wrapper) that checks whether the requested profile is supported before invoking guidellm, and emits a clear error if not:

   # example guard in guidellm-llm-d-benchmark.sh
   _profile=$(yq -r .profile "$workload_file")
   if [[ "$_profile" == "replay" ]]; then
     python3 -c "from guidellm.benchmark.profiles import replay" 2>/dev/null \
       || { echo "ERROR: profile 'replay' requires guidellm >= <version>; installed version lacks support."; exit 1; }
   fi

Workaround

Until the image is updated, install guidellm from the main branch inside the harness script:

pip install --upgrade "git+https://github.com/vllm-project/guidellm.git@73d91f311bd9"