AKS H100 inference TTFT regression from nodewright isolcpus tuning

[yuanchen8911]

Problem

azure-h100 UAT runs consistently fail the inference-perf TTFT gate on AKS Standard_ND96isr_H100_v5 (8x H100 SXM5), while the same validation passes on EKS/GKE H100 and passed on AKS before 2026-07-13. Benchmark: inference-perf validator — Dynamo 1.2.1 + vLLM serving Qwen/Qwen3-8B, 1 frontend (DYN_ROUTER_MODE=least-loaded) + 8x 1-GPU decode workers pinned to a single GPU node, AIPerf at concurrency 2048 (256/GPU), 16,384 requests. Gates: throughput >= 50,000 tok/s, TTFT p99 <= 2,000 ms (10% tolerance each).

Run	Throughput	TTFT p99	Verdict
UAT 29376768155	54,060 tok/s	4,485 ms	FAIL (TTFT)
UAT 29381050842	49,248 tok/s	5,125 ms	FAIL (TTFT)
UAT 29433572875	46,718 tok/s	5,717 ms	FAIL (TTFT)
Manual re-run on the 29433572875 cluster (same binary/images/recipe)	50,404 tok/s	6,034 ms	FAIL (TTFT)

Reproducible on demand — not a flake. EKS/GKE reference for the same benchmark: ~108k tok/s, passing TTFT.

Root cause

#1698 (merged 2026-07-13) replaced the AKS ib-node-config DaemonSet with the nodewright tuning chain, applying the nvidia-tuned aks-h100-inference profile to AKS GPU nodes for the first time. That profile's bootloader section renders isolcpus=0,1,48,49 into the kernel command line:

[bootloader]
# Isolate CPUs for inference processes, 2 per socket
cmdline_isolcpus=isolcpus=${f:cpulist_invert:${f:calc_isolated_cores:2}}

calc_isolated_cores:2 computes "all cores except 2 housekeeping cores per socket"; cpulist_invert inverts it, so the rendered value isolates the housekeeping cores (the first two of each socket, including CPU0) — the opposite of the comment's stated intent — and nothing is ever affined to them. CPU0 (default target for much of the IRQ routing, timers, and kernel housekeeping) is removed from the scheduler; under 2048-way concurrent HTTP serving, network/softirq processing serializes on cores the scheduler cannot balance, requests queue before their first token, and TTFT p99 blows out ~10x while throughput drops ~3x. The 4 isolated cores sit idle throughout.

Investigation

1. Code drift ruled out by cluster A/B. The same binary (2e0bc38), validator images (uat-29433572875), and recipe were run against aicr-test5 — identical VM type, node image (AKSUbuntu-2404gen2containerd-202606.19.0), kernel (6.8.0-1059-azure), driver (580.126.20), K8s v1.35.5, but deployed pre-feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning #1698 (no tuning CR, stock boot line). It passes: 146,452 tok/s / TTFT p99 547 ms. The regression is cluster-side.
2. Config comparison. RDMA host setup files (sysctl.d, modules-load.d, containerd/kubelet LimitMEMLOCK drop-ins) are byte-identical between the failing and passing clusters — the old DaemonSet had configured the same things. The only material difference is the Skyhook tuning CR (present and complete on failing clusters, absent on the passing one) and its boot-level effects, confirmed by diffing /proc/cmdline on GPU nodes of both clusters:
   • Tuned (failing) adds: init_on_alloc=0 pci=disable_acs_redir=pci:0:0 iommu=pt pci=realloc=off isolcpus=0,1,48,49 hugepagesz=2M hugepages=8192
   • Resulting state: isolated CPUs 0-1,48-49; 16 GiB of 2Mi hugepages reserved and 100% unused throughout the benchmark; vm.swappiness 1 vs 60. Governor performance on both.
3. Failure signature (throughput at the gate, TTFT p99 2–3x over) indicated request queueing rather than uniform slowness, pointing at isolcpus starving IRQ/softirq processing.

Validation (single-variable bisect, suggested by the Skyhook team)

On the failing cluster's GPU node: removed only isolcpus=0,1,48,49 from /etc/tuned/bootcmdline (hugepages, iommu=pt, pci=realloc=off, init_on_alloc=0, ACS disable all retained), update-grub, rebooted, verified /proc/cmdline, cordoned the other (still fully tuned) GPU node, and re-ran the identical validation:

Configuration (same node, same benchmark)	Throughput	TTFT p99	Verdict
Full aks-h100-inference profile (with isolcpus)	50,404 tok/s	6,034 ms	FAIL
Same profile, only isolcpus removed, reboot	148,344 tok/s	614 ms	PASS
Untuned stock AKS node (control cluster)	146,452 tok/s	547 ms	PASS

Removing isolcpus alone recovers ~3x throughput and ~10x TTFT, slightly better than the fully untuned control — isolcpus is single-handedly the entire regression; the remaining profile settings are innocent.

Suggested fix

Upstream fix belongs in nodewright-packages (nvidia-tuned inference profiles): drop the isolcpus line for inference intent, or fix the formula to match the stated intent and pair it with explicit IRQ/workload affinity. AICR then picks up the fixed nvidia-tuned pin in recipes/components/nodewright-customizations/manifests/tuning.yaml. Until then, azure-h100 UAT will fail the TTFT gate on every run.

Secondary finding for the same fix cycle: the tuned GRUB hook (/etc/default/grub.d/99_tuned.cfg) sets GRUB_CMDLINE_LINUX_DEFAULT=" ${TUNED_BOOT_CMDLINE}", replacing the distro default and silently dropping AKS Ubuntu 24.04's lsm=bpf,lockdown,capability,landlock,yama,apparmor — a security-posture change unrelated to performance.

Note: the 16Gi 2Mi-hugepage reservation is unused by this workload (HugePages_Free == HugePages_Total throughout) and only reduces allocatable node memory; not the cause of the regression, but worth revisiting for inference profiles.

Environment

• AKS v1.35.5, node image AKSUbuntu-2404gen2containerd-202606.19.0, Ubuntu 24.04.4, kernel 6.8.0-1059-azure, containerd 2.3.2-1
• Standard_ND96isr_H100_v5 (2-socket, 96 vCPU, 8x H100-80GB), driver 580.126.20, CUDA 13.0
• nvidia-tuned 0.3.1, nvidia-setup 0.5.0 (pins from recipes/components/nodewright-customizations/manifests/tuning.yaml)
• Workload: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1, Qwen/Qwen3-8B

References

• feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning #1698 — introduced nodewright tuning on AKS (2026-07-13); AKS validation passed 2026-07-09 pre-feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning #1698, fails on every post-feat(recipes): replace AKS ib-node-config DaemonSet with nodewright tuning #1698 cluster
• Failing UAT runs: 29376768155, 29381050842, 29433572875

[yuanchen8911]

Upstream fix tracker in the tuned profile: NVIDIA/nodewright-packages#95. This issue tracks the AICR side (picking up the fixed nvidia-tuned pin in nodewright-customizations and unblocking the azure-h100 UAT gate).