Skip to content

commands.md

Latest commit

 

History

History
139 lines (103 loc) · 2.84 KB

commands.md

File metadata and controls

139 lines (103 loc) · 2.84 KB
Raw

Commands

CLI commands for LLM Track evaluation.

Basic Usage

Single Operator Test

python scripts/generate_kernel_and_verify.py \
    --op-name aten::add \
    --single-test \
    --server-type openai \
    --model-name gpt-4o \
    --max-rounds 3

Full Benchmark

python scripts/generate_kernel_and_verify.py \
    --server-type openai \
    --model-name gpt-4o \
    --max-rounds 10

Dataset Selection

Full Dataset (NVIDIA)

python scripts/generate_kernel_and_verify.py \
    --dataset KernelGenBench \
    --server-type openai \
    --model-name gpt-4o

ATen Only (All Platforms)

python scripts/generate_kernel_and_verify.py \
    --dataset KernelGenBench-aten \
    --server-type openai \
    --model-name gpt-4o

Specific Operator Sources

# vLLM operators only
python scripts/generate_kernel_and_verify.py \
    --dataset KernelGenBench-vllm \
    --server-type openai

# cuBLAS operators only
python scripts/generate_kernel_and_verify.py \
    --dataset KernelGenBench-cublas \
    --server-type openai

API Formats

OpenAI

python scripts/generate_kernel_and_verify.py \
    --api-format openai \
    --model-name gpt-4o

Anthropic

python scripts/generate_kernel_and_verify.py \
    --api-format anthropic \
    --model-name claude-opus-4-6

Compatible Endpoints

No provider registration is required. Select the endpoint's wire protocol and pass its URL, key, and model directly:

python scripts/generate_kernel_and_verify.py \
    --api-format <openai|anthropic> \
    --model-name <model-name> \
    --base-url <api-endpoint> \
    --api-key <your-api-key>

The same values can be provided with OPENAI_BASE_URL / OPENAI_API_KEY or ANTHROPIC_BASE_URL / ANTHROPIC_API_KEY.

Advanced Options

Enable Reflection

Enable feedback from previous rounds:

python scripts/generate_kernel_and_verify.py \
    --server-type openai \
    --model-name gpt-4o \
    --reflection

Resume from Checkpoint

python scripts/generate_kernel_and_verify.py \
    --resume-from output/pass_at_k/previous_run/

Debug Mode

Test with only 8 operators:

python scripts/generate_kernel_and_verify.py \
    --debug \
    --server-type openai