TurboQuant for ORT WebGPU along with model_bechmark updates to measure GPU memory

[sushraja-msft]

Add support for kv cache quantization of existing models.

To enable this for webgpu ep provider, make the following change to genai_config.json.

       "session_options": {
            "log_id": "onnxruntime-genai",
            "provider_options": [
                {
                    "webgpu": {
                        "kvCacheQuantizationBits": "4"
                    }
                }
            ]
        },

ORT side changes microsoft/onnxruntime#28059

This change also updates the model_benchmark tool to print GPU memory usage numbers and allow profiling.

[Copilot]

Pull request overview

This PR adds initial (“WIP”) plumbing and tooling to support TurboQuant KV-cache compression for ORT WebGPU by allowing GenAI to use a reduced KV head dimension instead of the model’s original head_size.

Changes:

• Add kv_cache_head_size to genai_config.json parsing and use it when allocating the default KV cache.
• Add a new tools/prepare_turbo_quant.py helper to update ONNX KV tensor shapes (to a dynamic dim) and update genai_config.json with TurboQuant WebGPU provider options.
• Add new C++ attention-quality test executables (NIAH + RULER-inspired) and make small build/benchmark updates (ORT_HOME test env handling, WebGPU provider validation).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tools/python/util/dependency_resolver.py	Avoid copying directories when copying dependency artifacts.
tools/prepare_turbo_quant.py	New script to rewrite ONNX KV cache shapes and update GenAI config for TurboQuant.
src/models/kv_cache.cpp	Use kv_cache_head_size (if provided) when sizing the default KV cache allocation.
src/config.h	Add optional kv_cache_head_size to decoder config.
src/config.cpp	Parse kv_cache_head_size from genai_config.json.
examples/c/src/niah_test.cpp	New NIAH attention-quality test program.
examples/c/src/ruler_test.cpp	New RULER-inspired attention-quality benchmark program.
examples/c/CMakeLists.txt	Add build options/targets for the new NIAH/RULER test executables.
build.py	Update Windows generator default/list, improve env/PATH handling for ORT_HOME during tests, adjust examples build cleanup.
benchmark/c/options.cpp	Allow webgpu as a benchmark execution provider option.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

benchmark/c/main.cpp:321

• GPU memory is sampled after the iterations loop, but when reuse_generator is false each iteration's new_gen is destroyed at the end of the loop body. That means by the time GetGpuMemoryUsage() runs, the KV cache for the final iteration may already be freed, producing misleadingly low numbers (despite the comment saying KV cache is still allocated). Keeping the last new_gen alive until after sampling fixes this for the default reuse_generator=false path.

  for (size_t i = 0; i < opts.num_iterations; ++i) {
    std::unique_ptr<OgaGenerator> new_gen;
    if (opts.reuse_generator) {
      generator->RewindTo(0);
    } else {

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.