inference-perf creates the KAI Queue namespaced, but Queue is cluster-scoped

[yuanchen8911]

Summary

inference-perf creates its KAI Queue as a namespaced resource, but Queue is cluster-scoped. The request 404s on every cluster, always. Because the call is best-effort and logs at INFO, the failure is invisible and the validator's intended queue configuration is silently never applied.

Evidence

Queue is cluster-scoped in kai-scheduler v0.14.1 (the pinned version):

$ kubectl api-resources --api-group=scheduling.run.ai
NAME         APIVERSION                   NAMESPACED   KIND
bindrequests scheduling.run.ai/v1alpha2   true         BindRequest
podgroups    scheduling.run.ai/v2alpha2   true         PodGroup
queues       scheduling.run.ai/v2         false        Queue

$ kubectl get crd queues.scheduling.run.ai -o jsonpath='{.spec.scope}'
Cluster

Both the template and the call site treat it as namespaced:

# validators/performance/testdata/inference/queue.yaml
metadata:
  name: ${QUEUE_NAME}
  namespace: ${NAMESPACE}      # <- namespace on a cluster-scoped kind

// validators/performance/inference_perf_constraint.go:1666-1673
// Apply KAI Queue (best-effort; KAI scheduler may not be installed).
if err := createOrUpdateFromTemplate(ctx, kaiQueueGVR,
    config.namespace, queuePath, templateData, nil); err != nil {
    slog.Info("Failed to apply KAI Queue (scheduler may not be installed)", "error", err)
}

This POSTs to /apis/scheduling.run.ai/v2/namespaces/<ns>/queues, which cannot exist for a cluster-scoped resource. Observed on every GB300 run:

level=INFO msg="Failed to apply KAI Queue (scheduler may not be installed)"
  error="[INTERNAL] failed to create resource: the server could not find the requested resource"

The log message misattributes the cause — kai-scheduler is installed and healthy; the URL shape is wrong.

The cleanup path has the matching problem and is also a no-op:

// inference_perf_constraint.go:2486
Delete(cleanupCtx, inferenceQueueName, metav1.DeleteOptions{})

Impact

The queue's intended settings never take effect:

spec:
  parentQueue: default-parent-queue
  resources:
    gpu: { limit: -1, overQuotaWeight: 1, quota: 0 }

The benchmark instead runs under whatever queue the workload's PodGroup references — in practice dynamo (parent dynamo-default), created by the dynamo-platform chart. So inference-perf has been benchmarking under dynamo-platform's queue settings rather than AICR's on every Dynamo leaf, on every platform, for as long as this code has existed.

Not a correctness failure of the benchmark itself, but the validator believes it is configuring something it never configures.

Explicitly NOT the cause of the GB300 scheduling failure

Worth stating up front so nobody fixes this, sees the other failure persist, and loses time.

During #2382, a GB300 Dynamo run had kai-scheduler report 0 PodGroupInfos while the workload sat Pending for 18 minutes with 8 GPUs idle. This Queue bug was investigated as the cause and ruled out:

• The workload's PodGroup referenced queue=dynamo, which exists — not the validator's aicr-inference-perf queue.
• ${QUEUE_NAME} / inferenceQueueName appears only in queue.yaml and its own create/delete calls. The Dynamo workload template contains no queue reference.
• The identical 404 occurred in several earlier runs that then scheduled all five workload pods successfully and produced 80-85k tok/s.

That incident is being tracked separately; current evidence points at the scheduler's PodGroup watch, not queue resolution.

Suggested fix

1. Drop namespace: ${NAMESPACE} from queue.yaml.
2. Call createOrUpdateFromTemplate / Delete cluster-scoped (empty namespace) for kaiQueueGVR.
3. Reconsider the best-effort handling. If the queue is meant to configure the benchmark, a failure to apply it should at minimum log at WARN naming the degraded mode, rather than INFO with a message that misattributes the cause. A malformed request that can never succeed is not the same condition as "scheduler not installed", and the current wording sent this investigation down the wrong path more than once.

Add a test asserting the Queue is created cluster-scoped, so a future CRD scope change is caught rather than silently swallowed.

Found during #2382 (GB300 EKS overlays).