BENCHMARKS.md

Benchmarks — measured numbers, by model

This file is the consolidated cross-rig table for every compose variant we ship, with measured numbers (not derived estimates). It's intentionally append-friendly — every row carries an explicit Rig cell so multiple contributors can publish numbers for the same compose without rewriting the file.

Rows land here:

• when a contributor opens a PR adding a new compose variant, OR
• when a contributor supplies canonical bench output via the Numbers from your rig issue template.

Per-model qualitative findings, framework comparisons, and "why we picked this quant" rationale live in models/<model>/INTERNALS.md (or the local learnings/ tree). This file is just the numbers, anchored to (rig, date).

---

Canonical bench

All Narr / Code TPS rows come from bash scripts/bench.sh, which runs:

Narrative: "Write a detailed 800-word essay explaining transformer attention." (max_tokens=1000)

Code: "Write a Python implementation of quicksort with comments explaining each step." (max_tokens=800)

Sampling: temperature=0.6, top_p=0.95, top_k=20, presence_penalty=0.0, enable_thinking=false. Three warmups + five measured runs per prompt. Mean wall TPS reported.

Quality (8-pack) sampling: behavioral packs run at temperature 0 (greedy, reproducible) by default — the canonical bar for cross-config / cross-model ranking. To score a model at its recommended temp instead, the composes ship per-model sampling defaults (Qwen3.6 0.6 / Qwopus3.6 0.8 / Gemma 1.0, env-overridable) and the eval inherits them via quality-test.sh --sampling-from-server (tagged non-canonical). See QUALITY_TEST.md → Sampling & temperature.

PP tok/s is prompt-processing throughput. For vLLM rows, bench.sh scrapes the most recent Avg prompt throughput lines from container logs. For llama.cpp or engines without that log shape, run PP=1 bash scripts/bench.sh to add a single long-prompt fallback probe that computes prompt tokens over TTFT. Existing rows show — until re-benched with v0.7.1+ tooling.

Cross-rig numbers are comparable because the prompt + sampling are pinned. Variations against your rig usually trace back to power caps, PCIe lane counts, or pin (vLLM image SHA / Genesis commit) — see scripts/report.sh which captures all three.

How to add a row for your rig

1. Run bash scripts/report.sh --full > my-rig.md — captures hardware (incl. power caps + PCIe lanes), stack version (vLLM image SHA, Genesis commit), verify-full + verify-stress + SOAK_MODE=continuous + canonical bench numbers in one ~35-min pass. (Or --bench for the fast subset; soak-continuous catches Cliff 2b which the others don't.)
2. Open the Numbers from your rig issue template, paste the report, mention which compose variant you ran.
3. We'll append your numbers as a row in the appropriate table here, with Rig cell formatted @your-handle (rig-shape) — e.g. @whamp (4× 3090 PCIe x4/x8/x16/x16, 300 W).

If the same compose has multiple rig rows showing different numbers, that's a feature — it tells future readers what's portable vs rig-specific.

Notes column convention — soak verdict

Every row's Notes cell should start (or include) an explicit soak verdict so readers can grep at a glance:

• Soak: ✓ PASS — clean (no errors, 0 silent-empty turns, <200 MiB VRAM growth across 5×5 continuous sessions)
• Soak: ⚠ borderline — within thresholds but worth flagging (e.g. 240 MiB growth, slow turns >30s on x4 PCIe, etc.)
• Soak: ✗ FAIL — Cliff 2b suspect, silent-empty turns, mid-soak errors, or growth-threshold overshoot
• Soak: — — not run (acceptable but discouraged; row is "bench-only" until a follow-up brings the soak verdict)

Soak-continuous is the only signal that catches Cliff 2b. If your row says Soak: —, future readers will assume the worst; if your row says Soak: ✓ PASS, that's the 30-second answer to "is this rig+compose stable for hermes/openhands traffic." Use bash scripts/soak-test.sh --continuous (~25 min, auto-detects endpoint + container) to add the verdict if you initially submitted bench-only.

---

Qwen3.6-27B

Primary serving model. Hybrid Qwen3-Next architecture (DeltaNet GDN + standard attention). Quants used: AutoRound INT4 (vLLM), Unsloth Q5_K_XL GGUF (llama.cpp).

Single-card (1× RTX 3090) — vLLM

⚠️ Cliff 2b open on long-text* / long-vision (2026-05-05) — Genesis v7.72.2's PN59 streaming-GDN orchestrator doesn't engage on the chunked-prefill path 24 GB single-card configs are forced to take. Single-prompt prefill at >~50K may OOM. Filed at Sandermage/genesis-vllm-patches#22. Safe single-card paths: llamacpp/default (no Cliff 2b) or single-prompt context capped at <50K. TP=2 paths escape the cliff entirely (see Dual-card section).

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
minimal.yml (mem-util 0.95 max-model-len 65536)	@noonghunna (1× 3090, x16, 350 W)	TQ3	64K	~32 / ~33	—	~22.4 GB	2026-05-03	no MTP. stiggy2k16 cross-rig data point — short-prompt vLLM-safe path when llama.cpp is too slow.
long-vision.yml	@noonghunna (1× 3090)	TQ3	145K	50 / 66	—	~23.0 GB	2026-04-30	vision + tools + thinking. mem-util 0.95.
long-text.yml ⭐	@noonghunna (1× 3090)	TQ3	180K	50 / 67	—	~22.3 GB	2026-04-30	text-only (vision tower dropped). MTP n=3. mem-util 0.93. Default for RAG / IDE agents below 25K accumulated ctx.
long-text.yml	@laurimyllari (1× 4090, AMD Ryzen 7 7800X3D, 230W cap)	TQ3	90K (forced by KV-pool fit on 4090 — see Notes)	102.96 / 103.09	—	~23.7 GB	2026-05-05	First 4090 single-card vLLM bench on club-3090. Required max-model-len drop from 180K→90K at default mem-util 0.92 (KV cache budget on his 24 GB 4090 is tighter than the 3090s the compose was calibrated against — likely 4090 driver/desktop overhead consumes more idle VRAM). MTP n=3 active, AL 3.34-3.45 narr / per-pos accept 92-95% / 79-84% / 62-67%. CV 2.2%/2.2%. Verify-stress hit Cliff 2b OOM at long-vision 50 MiB (sidesteps via long-text). Issue #71 + disc #62.
long-text-no-mtp.yml	@noonghunna (1× 3090)	TQ3	200K	TBD	—	~21.0 GB	—	max-context single-shot, no MTP. Slow decode but biggest ctx window.
bounded-thinking.yml	@noonghunna (1× 3090)	TQ3	180K	50 / 66	—	~21.7 GB	2026-05-04	structured-CoT FSM in reasoning channel; recommended grammar: DeepSeek scratchpad (PLAN/NOTE×0-15/VERDICT). Phase 3 final: 93.9% HE+ / 66.0% LCB v6 (87.4% combined, +1 net vs the andthattoo G/A/E baseline). Andthattoo G/A/E grammar also works (94.5% HE+ / 62.0% LCB / 86.9% combined, ~4× tighter think budget — pass via extra_body). See STRUCTURED_COT.md.
tools-text.yml	@noonghunna (1× 3090)	fp8	75K	TBD	—	TBD	—	IDE-agent path that escapes the long-text Cliff 1 mech B leak (see #16).
dual-dflash.yml-shape forced TP=1 (DFlash N=5, fp8 KV, mem-util 0.96, custom_all_reduce disabled)	@efschu (1× RTX 5090 32 GB, AMD Ryzen 9 5950X, Debian trixie, PCIe x8, 575 W cap)	fp8	49K (KV-fit at 0.96 mem-util)	126.53 / 200.11 (decode 127.98 / 204.80)	—	31.5 GB	2026-05-07	First single-5090 DFlash data point on club-3090. AutoRound INT4 weights + DFlash N=5 draft. CV 3.0%/2.0%. Code TPS 200 is the highest single-card number measured on the matrix — beats single-3090 (50/67 long-text) by ~3× on code, single-4090 (102/103 at 90K) by ~2× code. Trade is ctx ceiling: 49K vs 90K-180K on 24 GB cards, due to KV-pool fit at fp8 + 32 GB total VRAM. vLLM nightly-01d4d1ad3 (post-v7.72.2 uplift). Issue #93.
vllm/default (single, MAX_MODEL_LEN=48000, mem-util 0.92, MTP n=3)	@ygafarov (1× 3090 via oculink eGPU on PCIe x4, AMD Ryzen AI MAX+ 395 / Strix Halo miniPC, CachyOS, 124 GB RAM, 290W cap)	TQ3	48K	68.86 / 91.70 (decode 69.27 / 92.76)	—	23.6 GB	2026-05-09	Soak: ⚠ borderline (VRAM grew 240 MiB > 200 MiB threshold, 3 turns >30s; 100% TPS retention + 0 errors + 0 silent-empty turns — x4-PCIe accretion + bus-latency under prefill, not a leak. Threshold may need an "eGPU bus class" allowance.) First Strix-Halo-miniPC + oculink-eGPU bench on club-3090. Single 3090 over PCIe x4 (oculink) instead of x8/x16 internal. CV 1.6%/2.7%. MTP AL 3.31, accept 76.9% (per-pos 0.918/0.772/0.616). scheduler_reserve_full_isl=False. Driver 595.71.05 (very new, CUDA 13.2). vLLM nightly-01d4d1ad3. Issue #113.

Single-card (1× RTX 3090) — llama.cpp

ℹ️ 4090 ctx-derate. A 24 GB 4090 carries more idle desktop + driver VRAM than a headless 3090, so single-card context ceilings land ~15–20% lower. Start below the 3090 number and confirm headroom with verify-stress.sh (watch its ceiling VRAM-margin line). Observed across engines: ik two-stage 200K→160K, long-text 180K→90K, dual-dflash-noviz 200K→180K.

Compose	Rig	Quant	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
llamacpp/default	@noonghunna (1× 3090)	Unsloth Q5_K_XL	262K	21 / 21	—	~20 GB	2026-04-21	bulletproof — different engine, different memory allocator, no Cliff 1 / Cliff 2. Slow decode but cliff-immune.
llamacpp/mtp ⭐ (alias llamacpp/default)	@noonghunna (1× 3090)	Unsloth Q4_K_M MTP (unsloth/Qwen3.6-27B-MTP-GGUF)	200K	50.27 / 58.92 (decode, n=3, CV 0.6% / 1.4%; wall 49.69 / 57.50)	1025	~22.4 GB (~1.6 GB headroom)	2026-05-23	Mainline llama.cpp server-cuda-b9246 (PR #22673 MTP merged) + MTP n=2 + q4_0 KV + native template + --reasoning off, at the 200K default (-ub 512) — fills ~183K with ~1.1 GB margin; CTX_SIZE=131072 UBATCH_SIZE=1024 is the faster-prefill override (~51 / 60 decode). verify-stress 8/8 PASS incl. the ceiling-ladder fill to 183K @ 200K — the Cliff 2 single-prompt narrative is config-driven, not architectural. Quality 8-pack 100/150 (67%) — on par with the best Qwen vLLM-dual configs in #119. Aider-polyglot-30 22/30 (73%) — per-lang cpp 4/5, go 4/5, java 3/5, js 5/5, python 3/5, rust 3/5. Per-GPU code TPS (58.92) ≈ vLLM-dual per-card (60–63) — vLLM dual's aggregate edge is purely the second card. Zero patches, no Genesis, no AutoRound. 200K thinking-off rebench; PR #166 · #201.
llamacpp/qwen27b-pi-reasoning-single 🧪	@noonghunna (1× 3090, 370 W)	bytkim Q4_K_M MTP pi-reasoning (bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF)	~188K (200K-alloc)	47.9 / 55.3 (decode, n=5, CV <2%; wall 47.4 / 54.2; thinking-off)	1030	~22.7 GB	2026-06-18	Mainline llama.cpp server-cuda-b9246 + MTP n=2 + q4_0/q4_0 KV, config per the model card (temp 1.0 / top-k 0, reasoning-ON). A community "Pi-style" reasoning/coding-agent fine-tune. MTP head ≡ base llamacpp/mtp — a matched-power A/B put the two dead-even (73% vs 72% draft accept, identical decode); 47.9/55.3 here is ~on par with base 50.27/58.92 (within ~5% on the canonical prompts). ⚠️ Power-sensitive: ~28.5 / 32.9 under a 230 W cap (mainline −42% 370→230 W). 8-pack 104/150 think-off · 106/150 think-on; verify-stress 8/8 (NIAH→183K @ 91% of the 200K pool); soak PASS (0 MiB growth, 0/100 silent-empty, p50 54.5, 102% retention). Context: 200K-alloc fills ~188K usable — 262K-alloc OOMs ~176K (FA scratch grows with alloc), full 262K is beellama-only; author tested only 128K (CTX_SIZE=131072 for the strict card window). Verbose even thinking-off → give it generous max_tokens (a few deterministic-pack misses were finish_reason=length). 🧪 experimental / opt-in (--force). PR #425.
llamacpp/mtp-vision (2026-05-20; superseded by re-tune ↓)	@noonghunna (1× 3090)	Unsloth Q4_K_M MTP + mmproj-F16	49K	56.52 / 66.17 (decode, n=5, CV 1.6% / 1.7%)	1158	~20.5 GB (~3.5 GB headroom)	2026-05-20	First stack profile combining MTP + vision (sweep-verified on build 9235; the older "strip mmproj when MTP" rule was obsolete). Same Q4_K_M MTP GGUF + q4_0 KV + MTP n=2 + -ub 1024 + --mmproj mmproj-F16.gguf (vision projector loaded). Multimodal probe ✅ (model answered "Red" on a synthetic 64×64 red PNG in 0.5s; mmproj pipeline functional end-to-end). verify-stress 8/8 PASS (sapphire iguana 19 @ 9.8K, crimson falcon 18 @ 29.3K, all 7 rungs clean at the 49K ctx). TPS being ~10% higher than the no-vision Config A above isn't strictly A/B-controlled across GPUs — Config A was on GPU 0 after ~2 hrs continuous load; this was a fresh GPU 1; plausibly KV-cache-locality (49K pool 2.6× smaller than 131K) + thermal-state. PR #166.
llamacpp/mtp-vision (re-tuned ⭐)	@noonghunna (1× 3090, 370 W)	Unsloth Q4_K_M MTP + mmproj-F16 @ 1 M-px (IMAGE_MAX_TOKENS=1024)	150K (fills ~138K)	— (decode ≈ 56 / 66, carried from the 49K row — ctx-independent, not re-benched)	—	~22.5 GB boot · 561 MB free @ 138K ceiling	2026-05-25	Vision re-tune (PR #227, #437). The 49K row ↑ and the later 160K compose default were never validated under the current verify-stress [8/8] ceiling ladder. Re-laddered: full-res 4M-px (IMAGE_MAX_TOKENS=4096) OOM-crashes at 160K under the agentic battery; the old "196K @ -ub 512" claim walls at ~125K fill. New measured-safe default = 150K @ 1M-px: completes [8/8], recalls to 138K (91%), 561 MB free at ceiling, no crash. Full-res 4M-px = documented override (lower ctx). >150K context → text-only llamacpp/mtp. n=1.
ik-llama/iq4ks-mtp ⭐	@noonghunna (1× 3090, set+readback 370 W)	ubergarm MTP-IQ4_KS (ubergarm/Qwen3.6-27B-GGUF)	200K	59.67 / 68.78 (decode 60.39 / 72.40, n=3, CV 1.5% / 0.6%)	1109	~22.4 GB	2026-05-23	ik_llama.cpp cu13-server + IQK fused kernels + q4_0 KV + MTP n=2 + -khad/-vhad + native template, thinking-off. ~18–20% FASTER decode TPS than shipped llamacpp/mtp Q4_K_M (49.69 / 57.50 wall, same rig + 370 W cap) — verified by a set-and-readback power-cap A/B: ik leads at both 230 W and 370 W (both llama.cpp engines are power-sensitive on this rig — ik −29% / mainline −42% from 370→230 W; comparison only valid at matched power). 8-pack 101/150 (≈ tie, mainline 100), soak PASS (p50 71.5, 0 err, 0/100 silent-empty, 0 MiB growth), verify-stress 8/8 incl. ceiling-ladder fill to 183K @ 200K, aider-polyglot-30 19/30 (per-lang: cpp 4/5, go 4/5, java 3/5, js 3/5, python 3/5, rust 2/5). Corrects the #184 "tie" — that was a wrong-engine measurement artifact: its "ik ~50/58" is exactly mainline@370, a number ik produces at no power setting (ik = 42/51 @ 230 W, 60/72 @ 370 W). Post-#38-chain rebench (2026-05-28): 8-pack 107/150 (71%) (+6 vs 101) — gains from PR #42 deterministic hermes sampler (0→11/20) + PR #43/#47 cli-40 timeout defaults + honest tags (11→19/40); aider 18/30 (≈ tie, within ±2 single-run noise) — all 30 exercises reached the LLM (PR #44 git-checkout fix). 40-min batch wall completed inside PR #43's TPS-aware scaled budget (50 min) — first real run that would have been prematurely killed by the static 30-min default. Numbers above are pre-#38 baseline (preserved for the audit trail); these are the same compose under post-#38 verifiers + sampler.
ik-llama/iq4ks-two-stage ⭐ (code)	@noonghunna (1× 3090, 370 W)	ubergarm MTP-IQ4_KS (ubergarm/Qwen3.6-27B-GGUF)	200K	59.4 / 97.8 (decode, n=3, CV 1.9% / 5.3%)	—	~21.6 GB	2026-05-24	ik_llama.cpp cu13-server TWO-STAGE spec-dec (PR #1789): ngram-mod n_max=4 + MTP n_max=3, q4_0 KV + -khad/-vhad + native template, thinking-off. Code decode +35% vs iq4ks-mtp MTP-only (97.8 vs 72.40, matched same-rig 370 W); narrative ~tie (59.4 vs 60.39). ngram n_max swept {2,3,4,5,8,16} → inverted-U peaking at 4 (16→81.5, 8→90.4, 4→97.8, 2→78.8): too-long drafts waste verify compute, too-short stop matching repeated code spans. Lossless — 8-pack 98/150 (≈ MTP-only 100), aider-polyglot-30 16/30 (≈ MTP-only 19), both within single-run noise; verify-full green. verify-stress @ 200K fills to 183K with correct needle recall (ceiling VRAM margin 833 MB is just under the 1024 MB sustained-agent guard → CTX_SIZE=180224 for heavy agentic-near-ceiling). ~0.8 GB leaner than iq4ks-mtp at idle. Tune #210 · ctx 200K + ⭐ #212.
ik-llama/iq4ks-mtp-vision (NEW ⭐)	@noonghunna (1× 3090, 370 W)	ubergarm MTP-IQ4_KS (ubergarm/Qwen3.6-27B-GGUF) + mmproj-F16 @ 1 M-px	160K (fills ~147K)	— (decode ≈ 60 / 72, carried from ik-llama/iq4ks-mtp ↑)	—	~22.4 GB boot · 503 MB free @ 147K ceiling	2026-05-25	Vision re-tune (PR #227, #437); graduated EVAL #402 → Production. Full verify-stress [8/8] @ 160K + 1M-px: recalls to 147K (91%), 503 MB free at ceiling, no crash, vision functional (read needle text off a 1024² image). 180K OOMs at fill; 200K leaves only 891 MB after one image — shrinking IMAGE_MAX_TOKENS does NOT buy ctx (vision ceiling gated by KV + fill-time runtime overhead, not the image-encode buffer). ~0.5 GB leaner than the llama vision row → why ik holds 160K where llama lands 150K. Full-res 4M-px = override. n=1 (noisy).
beellama/dflash ⭐ (single-card default 2026-05-30)	@noonghunna (1× 3090, 370 W)	Unsloth Q5_K_S + Anbeeld DFlash-IQ4_XS draft	160K (ships 102K; 200K OOMs on prefill)	50.2 / 99.7 (decode 50.4 / 101.3, n=5, CV 5.6% / 7.4%)	~1066	~22.1 GB @ 102K	2026-05-30	Promoted to the single-card DEFAULT (resolver ENGINE_PREFERENCE[single] #1 + caveats DEFAULTS row). beellama.cpp (Anbeeld llama.cpp fork) DFlash spec-dec + windowed KV, q5_0/q4_1 KV, native template, thinking-off. Code decode ~100 TPS — fastest single-card 3090 code path (vs ik 72, mainline 59); narrative ~50 (ik leads at 60). Output-lossless DFlash → quality = the Q5_K_S target: 8-pack 107/150 (71%) think-off, 113/150 (75%) think-on (same-session, #239). DFlash accept 29–37% (prose/code). Ceiling ladder (2026-05-30): 130K comfortable (1.6 GB headroom), 160K usable (115K-tok prefill OK, 0.8 GB), 200K boots but OOMs on prefill. Served via our unofficial multi-arch image beellama-cpp:multiarch-b9459-07ac3ce (sm_86/89/120 = 3090/4090/5090; sm_89/120 compiled-not-validated — only 3090 verified on-rig). Community fork chain, no official Docker yet (Anbeeld v0.3.0 WIP).

v0.3.0 DFlash-vs-no-spec A/B (2026-06-03, bench.sh narrative n=5, measured no-spec controls): DFlash prose is net-positive on v0.3.0 across the board — Qwen single Q5 no-spec 35.9 → DFlash 45.7 (+27%); Qwen dual Q8 no-spec 23.4 → DFlash 35.6 (+52%); Gemma single Q4 no-spec 34.8 → DFlash 44.6 (+28%). AR ~0.29–0.41 on all three v0.3.0 images (efe856397/e0663be/63abcd3). ⤷ The earlier v0.3.0 "prose-acceptance regression" (~0.07 AR / net-negative) is RETRACTED — it was an AR over-read + a wrong no-spec baseline (the ~37 we'd used was never the real dual-Q8 no-spec). New adaptive-DM HEAD 63abcd3 is neutral. See learnings/qwen3.6-27b.md 2026-06-03 + docs/UPSTREAM.md.

| ik-llama/iq4ks-two-stage (1× 4090) | @laurimyllari (1× 4090 24 GB, AMD Ryzen 7 7800X3D, 450 W) | ubergarm MTP-IQ4_KS (ubergarm/Qwen3.6-27B-GGUF) | 160K (derate from 200K) | 82.5 / 120.9 (decode, n=5, CV 4.4% / 7.9%; wall 81.0 / 114.2) | 2311 | ~22.8 GB boot (21.5 GB at bench) | 2026-05-24 | First 4090 ik-two-stage cross-rig. Required ctx derate 200K→160K — OOM at 200K (a 24 GB 4090's idle desktop+driver VRAM is tighter than a headless 3090; matches his own long-text 180K→90K + snoby dual-dflash-noviz 200K→180K). +39% narr / +24% code vs the 3090 two-stage (59.4 / 97.8) — newer silicon + higher PL. verify-stress 8/8 PASS, soak PASS (silent_empty 0/100, p50 112.65, 0 MiB growth, retention 100.9%). ⚠ 8-pack 101/150 + aider-polyglot 17/30 were measured pre-PR #31 grader fix (toolcall/reasonmath false-negatives) → under-count true capability; re-measure pending. disc #184. | | llama.cpp PR #22673 MTP, custom build (Qwen3.6-27B-MTP-Q4_K_M-GGUF + --spec-type mtp --spec-draft-n-max 3) | @efschu (2× Tesla V100-SXM2-16GB, Xeon Gold 6154, Debian 13, custom-built llama-server docker) | Q4_K_M MTP | 100K | 49.96 / 62.46 | — | 15.6 GB/card (15,596 MiB at 100K ctx) | 2026-05-06 | First V100 (sm_70 Volta) cross-rig data on the matrix — only non-3090/4090/5090 GPU class tested. vLLM blocked (V100=CC 7.0, vLLM needs ≥7.5); fell back to llama.cpp via am17an's PR #22673 with a custom-built docker. All 7 stress checks PASS including 90K NIAH (Cliff 2 territory). 2× cards via tensor split (-sm tensor). MTP n=3, accept rates not in log. ~80 W/card (V100 max 300 W). Issue #80. | | llama.cpp PR #22673 MTP, host build (havenoammo/Qwen3.6-27B-MTP-UD-GGUF + --spec-type mtp --spec-draft-n-max 3 + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350W) | UD-Q4_K_XL + Q8_0 MTP head | 131K | 47.12 / 60.42 | — | ~23.1 GiB | 2026-05-07 | First 1× 3090 llama.cpp MTP data point on Qwen3.6-27B. Decode 47.60 / 61.71 TPS, TTFT 212 / 194 ms. verify-full-mtp.sh PASS 8/8 (locally-adapted), verify-stress-mtp.sh PASS 7/7 including 91K needle at 131K ctx — pushes the documented llama.cpp MTP ctx ceiling from ~64-80K (q8_0 KV) to 131K (q4_0 KV). MTP acceptance 78.7%; recurrent 65-layer bug from froggeric's earlier MTP GGUF did NOT reproduce on havenoammo's UD GGUF. Native host build (no Docker), surfaced engine-coupling shortcomings in our verify/soak harness — see Issue #85. | | llama.cpp PR #22673 MTP, host build (froggeric/Qwen3.6-27B-MTP-GGUF + --spec-type mtp --spec-draft-n-max 3 + q4_0 KV) | @lamentofhighborne (1× RTX 3090, PCIe x8, 350 W) | Q4_K_M MTP | 164K | 47.49 / 55.09 | — | ~22.2 GiB | 2026-05-07 | Second 1× 3090 llama.cpp MTP data point on same rig — froggeric's Q4_K_M MTP GGUF vs havenoammo's UD-Q4_K_XL above. Decode 47.91 / 55.81 TPS, TTFT 96 / 98 ms. verify-full-mtp.sh PASS 8/8, verify-stress-mtp.sh PASS 7/7 incl. 91K needle at 164K ctx. Functional MTP acceptance 86.7%; canonical acceptance 55.3% narr / 71.2% code. Ctx-fit ladder: 262K OOMed MTP, 229K served without MTP, 196K initialized MTP but daemon died at 90K stress; 164K was the stable stress-passing ceiling on this rig. Beats havenoammo on narr (47.49 vs 47.12, +0.8%) and ctx ceiling (164K vs 131K) but trails on code (55.09 vs 60.42, −9%). Manual long-context needles also passed at 120K (39.39 decode TPS, 81% MTP accept) and 150K (35.44 decode TPS, 80% MTP accept). MTP+vision incompat (per froggeric's model card); separate no-MTP+vision path passed 65K and 150K. Issue #94. |

Thinking on vs off — code-gen (HumanEval+ / LiveCodeBench v6)

Stock Qwen3.6-27B, single-card llama.cpp (--reasoning on), FREE thinking-on vs enable_thinking=false, max_tokens=10240, temp 0.0, n=1/problem, 2026-05-24.

Arm	HumanEval+ 164	LiveCodeBench v6 70	All 234	mean out tok (HE+ / LCB)	FREE truncated (HE+ / LCB)
no-think (shipped default)	155 (94.5%)	63 (90.0%)	218 (93.2%)	319 / 2610	—
FREE (thinking-on)	151 (92.1%)	45 (64.3%)	196 (83.8%)	2902 / 6662	0% / 49%

Thinking is net-negative for code on stock Qwen3.6 — and bounding it doesn't flip that. Two failure modes: on easy code (HE+) thinking finishes (0% truncation) yet overthinks correct→wrong (−4, at ~13× the tokens); on hard code (LCB v6) it runs away — 49% never converge within the 10K budget and truncate→fail (−18). Giving FREE a bigger budget or grammar-bounding the reasoning (the structured-CoT bounded-thinking.yml row above: 93.9% HE+ / 66.0% LCB v6, separate eval, different conditions) caps the runaway but lands LCB at ~66% — still ~24 pp under no-think's 90%. Bounding makes thinking terminate, not out-reason just writing the code. (The FREE arm's completion-only LCB slice looks like ~parity — 35/36 — but that's selection bias: the problems thinking finished naturally are the easy ones.) Verdict: thinking-off is the correct default for code-gen; thinking-on — free or grammar-bounded — ties at best, at multiples of the token cost. Thinking's payoff, if any, is on non-code reasoning (math/science) — untested here; the quality-test.sh --reasoning packs (GSM-Symbolic, GPQA, default_thinking: on) are the probe. A clean same-run 3-arm (no-think / FREE / FSM-bounded) on LCB would pin the grammar-rescue exactly.

Dual-card (2× RTX 3090, TP=2)

NVLink auto-detection: dual-card composes now detect NVLink presence automatically. The dual-nvlink*.yml files are deprecated stubs that extend the unified compose with NVLINK_MODE=force_on. All NVLink bench rows below were measured with NVLink enabled (either via auto-detection or the deprecated stub). PCIe rows used NCCL_P2P_DISABLE=1.

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
dual.yml ⭐	@noonghunna (2× 3090 PCIe, no NVLink)	fp8	262K (237K single-prompt verified)	69 / 89	—	~23.6 GB	2026-04-29	tested 2-card baseline. fp8 KV, 2 streams, full feature set. PASSES v2 continuous soak (Cliff 2b clean).
dual.yml ⭐ (vLLM v0.24.0 pin)	@noonghunna (2× 3090 PCIe, no NVLink; caps GPU0 370 W / GPU1 420 W)	fp8	262K	70.0 / 91.5 (decode 70.7 / 93.5)	—	~21.5 GB/card	2026-06-30	vllm-stable v0.22.0 → v0.24.0 pin-bump validation — ≈ par with the v0.22.0 baseline (69/89). TP=2 symmetric, MTP n=3 accept 3.51, KV pool 622K / 2.37×. verify-full 8/8 (streaming tool-calls clean → qwen3coder sidecar droppable), verify-stress NIAH→240K, soak-continuous PASS (0 growth / 0 err / 100% retention), quality toolcall 11/15 (73%) · instructfollow 15/15 (100%) (--quick, thinking=mixed, ≥ baseline). vllm-marlin-pad confirmed native (#45295 + #45176) — served overlay-free. v0.24.0 default Mamba-cache align + experimental prefix-caching → NIAH-clean, no mitigation needed.
dual-max (FP8 wts + int8-PTH KV, v0.24.0) 🧪	@noonghunna (2× 3090 PCIe, caps 370/420 W)	int8-PTH	262K	82.0 / 104.6 (decode 83.1 / 108.2)	—	~21.4 GB/card	2026-06-30	The 8-bit decode corner. FP8 weights via Marlin W8A16 (no native FP8 on Ampere → weight-only dequant; vLLM warns "may degrade compute-heavy workloads"). MTP n=3, KV pool 295K/1.13×, TTFT 158/167 ms. 8-pack --full 107/150 (thinking-off: toolcall 14 · IF 15 · structout 14 · dataextract 9 · reasonmath 12 · bugfind 14 · hermes 11/20 · cli 18/40). Corrects the stale "~56 TPS" probe — real v0.24.0 decode is 83/108 (fastest decoder on the rig). Head-to-head vs W8A8 ↓.
dual INT8 W8A8 (Avesed) + int8-PTH KV, v0.24.0 🧪	@noonghunna (2× 3090 PCIe, caps 370/420 W)	int8-PTH	262K	76.2 / 96.4 (decode 76.9 / 99.0)	2062 → 1021 (prefill t/s @10K→90K)	~21.4 GB/card	2026-06-30	The 8-bit prefill corner. INT8 weights + dynamic INT8 activations → native CUTLASS INT8 prefill (no dequant). TTFT 122 ms (vs FP8 158). Prefill +17–51% vs FP8 at matched ctx (2062 vs 1364 @10K); decode trades ~8% (96 vs 105). 8-pack --full 107/150 — TIED with FP8 (thinking-off: toolcall 14 · IF 15 · structout 14 · dataextract 10 · reasonmath 12 · bugfind 14 · hermes 10/20 · cli 18/40) → activation-quant cost = zero. NIAH clean to 240K, pool 295K/1.13×. Same compose as dual-max, only --model+--quantization compressed-tensors differ. Experimental data point — the 8-bit prefill corner vs FP8's decode corner; NOT shipped as a slug (pending cross-rig + thinking-on arm before any catalog decision).
dual-turbo.yml	@noonghunna (2× 3090 PCIe)	TQ3	262K	58 / 76 per-stream (269 TPS aggregate at 4 streams)	—	~19.8 GB	2026-04-29	TQ3 KV — 4.67× concurrency for multi-tenant agent workloads.
dual-turbo.yml ⭐	@noonghunna (2× 3090 PCIe)	TQ3	262K	81.21 / 108.20 single-stream	—	20.0 GB	2026-05-05	v7.72.2 uplift: Genesis pin 7b9fd319 + vLLM 01d4d1ad3 (Sander's PROD pin). 6 redundant local sidecars dropped (PN35/PN30/PN25/P78/PN34 supersede). 5 measured runs each, CV 2.3%/0.9%. AL 3.46. VRAM −2.1 GB/card vs v7.69 baseline (PN35 native + PN59 fold value). All 8/8 verify-full checks pass.
dual-dflash.yml	@noonghunna (2× 3090 PCIe)	fp8	185K	82 / 125	—	~23.6 GB	2026-04-29	DFlash N=5 + 1.75 GB draft / card. AL ~4.4. Fastest 2-card short-prompt code path.
dual-dflash.yml	@apriori (2× 3090 + EPYC 7302P, Arch Linux, 230 W cap, NODE topology, no NVLink)	fp8	185K	78.44 / 122.71	—	~24.0 GB	2026-05-05	First EPYC + Arch cross-rig data on dual-dflash — matches @noonghunna baseline within run-to-run CV (78/127 reference, narr drift +0.4 / code −3.4%). PASSES continuous soak (0 MiB VRAM growth, 0 errors, 0/25 silent-empty, 100% TPS retention) — first independent confirmation dual-dflash is Cliff 2b clean cross-rig. 3 turns >30s TTFT warning (informational). Discussion #18.
dual-dflash-noviz.yml	@noonghunna (2× 3090 PCIe)	fp8	200K	78 / 127	—	~23.8 GB	2026-04-29	DFlash + no vision tower. +15K ctx vs dual-dflash.
dual-dflash-noviz.yml	@snoby (2× 4090 PCIe — 5-GPU rig, GPUs 2,3, no NVLink, #46)	fp8	180K	92.55 / 148.99	—	~21.8 GB	2026-05-04	First non-3090 cross-rig data. Required max-model-len drop from 200K→180K vs 3090 baseline (boot OOM at 200K) — 4090 ctx-ceiling gotcha pending investigation. +17% TPS lift vs same compose on 3090 (78→92.55 narr / 127→148.99 code).
dual.yml	@guybrush01 (2× 5090 PCIe 5.0 x8, Ryzen 9 9950X3D2, CachyOS, driver 610.43.02, 575 W cap, no NVLink)	fp8	262K	153.41 / 196.91 (decode 154.62 / 200.13)	—	~31 GB/card (934 MB free at 262K ceiling)	2026-06-25	First Blackwell / dual-5090 cross-rig data. Qwen3.6-27B dual (AutoRound-INT4 + fp8 KV + MTP n=3) on vLLM v0.22.0, TP=2, sm_120 — boots + serves the full 262K with no changes. ~2× the 3090 dual (vs @noonghunna v7.72.2 ~89/115) at tiny TTFT (~51/53 ms), CV 1.5%/1.2%. MTP accept 2.71–3.03. verify-stress 8/8 (NIAH→240K, Cliff 2 clear); soak-continuous PASS (0 err, 0/25 silent-empty, 0 MiB growth, p50 decode 252). Blackwell symm-mem all-reduce N/A → PYNCCL fallback (custom all-reduce off anyway, as on our PCIe rigs). PCIe 5.0 x8 ≈ Gen4 x16. 8-pack quality 110/150 (quality-test.sh --full, thinking=mixed, 2026-06-27) — lands on our dual-3090 baseline (109/150) within n=1 noise (det 63 vs 64 · sandbox 47 vs 45), confirming 8-pack quality is weights+KV+sampler-bound, not arch-bound; the two soft packs (dataextract 9/15 numeric-typing, cli-40 20/40 agentic) are the known Qwen-family floors. Issue #474 (quality run).
dual-nvlink.yml	@JusefPol (2× 3090 PCIe x8 + NVLink 4× bonded, i7-11700K, 365 W/card)	fp8	262K	108.81 / 138.55	—	~23.7 GB	2026-05-04	First NVLink cross-rig data. +58% narr / +56% code TPS vs dual.yml PCIe-only baseline (69 / 89) — NVLink reduces the per-token NCCL allreduce latency floor; compounds at multi-stream. verify-stress 8/8 PASS incl. 91K needle. PASSES v2 continuous soak (5 sessions × 5 turns, 0 MiB growth, 100% TPS retention). MTP n=3, 65–98% per-position accept. PR #31.
dual-nvlink-turbo.yml ⭐	@danbedford (2× 3090 NVLink, 230W cap)	TQ3	262K	102.34 / 133.98	—	~22.3 GB	2026-05-05	v7.72.2-rebench (image nightly-01d4d1ad3). 4-stream TurboQuant KV + NVLink. +11% narr / +12% code vs same-rig PCIe dual-turbo (#73 below) — controlled A/B on identical hardware, only NCCL_P2P_LEVEL differs. Custom all-reduce ENABLED (disabled on PCIe). CV 3.1% narr / 1.8% code. PR #56 + Issue #69.
dual.yml	@danbedford (2× 3090 NVLink-cable-attached, run as PCIe via NCCL_P2P_DISABLE=1, 230W cap)	fp8	262K	89.24 / 114.57	—	~23.7 GB	2026-05-06	First controlled PCIe-vs-NVLink A/B on same rig — pair with dual-nvlink.yml row immediately above. +15% narr / +15% code lift from NVLink (#74 102/132 vs this 89/115). CV 3.8%/2.5%. Note: this corrects the "+58% narr / +56% code" claim from JusefPol's row — that comparison conflated NVLink lift with v7.72.2 lift (his baseline was 2026-04-29 dual.yml at 69/89 on the older image). On a strictly v7.72.2-controlled comparison NVLink adds ~15%, not ~58%. Issue #77.
dual-turbo.yml	@danbedford (2× 3090 NVLink-cable-attached, run as PCIe via NCCL_P2P_DISABLE=1, 230W cap)	TQ3	262K	91.58 / 120.00	—	~22.0 GB	2026-05-06	Companion to dual-nvlink-turbo row above for the controlled A/B. NVLink lift on TQ3 path: +11% / +12%. CV 3.2%/1.9%. Issue #73.
dual-nvlink.yml	@danbedford (2× 3090 NVLink, 230W cap)	fp8	262K	102.09 / 131.59	—	~24.0 GB	2026-05-06	Second cross-rig data on dual-nvlink.yml (vs JusefPol's earlier 108.81/138.55). Lower than JusefPol partly explained by his lower power cap (365 W/card vs 230) — on memory-bandwidth-bound decode, 2 GB/card more thermal headroom doesn't compound much, so close-but-lower at half the wattage is consistent. CV 2.6%/1.4%. Issue #74.
dual-dflash.yml	@danbedford (2× 3090 PCIe NVLink-cable-attached but NCCL_P2P_DISABLE=1, 230W cap)	FP16	185K	86.62 / 141.02	—	~24.0 GB	2026-05-06	Third cross-rig DFlash data point (after @noonghunna 82/125 + @lolren 87/142). Code TPS 141 ties lolren's 142 as the highest measured on club-3090. CV 2.4%/5.0%. Issue #75.
dual-dflash-noviz.yml	@danbedford (2× 3090 PCIe NVLink-cable-attached but NCCL_P2P_DISABLE=1, 230W cap)	FP16	200K	88.31 / 142.79	—	~23.9 GB	2026-05-06	DFlash + no vision tower. Beats @noonghunna baseline 78/127 (+13%/+12%). CV 2.3%/2.9%. Issue #76.
dual-nvlink-dflash.yml ⭐ NEW	@danbedford (2× 3090 NVLink, 230W cap, i9-11900KF)	FP16	185K	101.55 / 163.33	—	24.06 GB/card	2026-05-07	First NVLink-enabled DFlash row. Mirrors dual-dflash.yml shape but enables NCCL P2P over NVLink + custom_all_reduce. +17% narr / +16% code over his own PCIe dual-dflash row above (86.62 / 141.02 — same rig with NCCL_P2P_DISABLE=1). Decode 102.43 / 166.54 TPS, CV 1.8%/1.9%. PASSES continuous soak (0 errors, 0 silent-empty, 0 MiB growth, 100% TPS retention, p50 66.71). verify-full 8/8 + verify-stress 8/8 incl. 91K Cliff 2 needle. PR #92.
dual-nvlink-dflash-noviz.yml ⭐ NEW	@danbedford (2× 3090 NVLink, 230W cap)	FP16	188K	103.24 / 167.45	—	~23.97 GB/card	2026-05-07	NVLink + DFlash + no vision — pushes the with-vision 185K ctx ceiling to 188K by dropping MoonViT (~0.78 GB freed). Empirically determined: 189K had only 1/3 success rate (flaky on freshly rebooted system), 188K is the stable ceiling. +17% narr / +17% code over his own PCIe dual-dflash-noviz row above (88.31 / 142.79). Decode 104.07 / 171.01 TPS, CV 2.2%/3.6%. PASSES continuous soak (p50 66.75, 100% retention). verify-full 8/8 + verify-stress 8/8. PR #96.
dual.yml-shape + patched P2P drivers (no NVLink hardware)	@aaronlockhartdev (2× 3090 PCIe x16, EPYC 7F52, Arch Linux, custom Dockerfile via Sam McLeod's guide — patched aikitoria/open-gpu-kernel-modules + vLLM cuda.py return True patch)	fp8	262K	93 / 125	—	n/a	2026-05-07	First patched-driver P2P cross-rig data point — answers the question raised in disc #70. Same-rig controlled A/B: unpatched baseline 91 narr / 114 code → patched P2P 93 / 125 = +2% narr / +9% code. Compared to NVLink hardware lift (+15% / +15% per @danbedford's controlled A/B): patched P2P captures ~60% of NVLink's code gain but ~13% of NVLink's narr gain — code workloads (spec-decode K+1 verify is heavily cross-card matmul) benefit more from cross-card bandwidth than narr decode (more sequential per-token). For ~95% of dual-3090 owners without NVLink, the trade is small TPS lift vs custom kernel module + DKMS maintenance burden. Issue #91.
dual-dflash-noviz.yml-shape + patched P2P drivers (no NVLink hardware, custom_all_reduce ENABLED)	@aaronlockhartdev (2× 3090 PCIe x16, EPYC 7F52, Arch Linux, patched kernel module + NCCL_P2P_LEVEL=PHB)	fp8	200K	100.47 / 160.15 (decode 101.53 / 164.44)	—	~22.2 GB/card	2026-05-07	Second patched-P2P cross-rig data point — extends #91 dual.yml result to the DFlash + no-vision path. Same-rig controlled A/B: unpatched baseline 82.55 narr / 134.45 code → patched P2P 100.47 / 160.15 = +22% narr / +19% code. Significantly larger lift than dual.yml-shape (+22%/+19% here vs +2%/+9% on dual.yml) — DFlash's K+1 cross-card verify pattern stresses peer-bandwidth more than fp8-only dual.yml. Important methodology update: NCCL_P2P_LEVEL=PHB alone with the default vLLM image produced the same lift as the full vLLM cuda.py patch — the in-container vLLM source patch is unnecessary, only the kernel module patch matters. CV 4.6%/2.4%. custom_all_reduce ENABLED (vs disabled on the dual.yml row). Issue #95 + disc #70.
dual-dflash-noviz.yml	@hlo-world (2× 3090 PCIe x4 each, Ryzen 5 5600, Ubuntu 24.04, driver 595.71.05; aikitoria patched driver installed host-side but run used stack-default NCCL_P2P_DISABLE=1 — P2P NOT engaged)	fp8	200K	76.15 / 117.31 (decode 76.94 / 121.75)	—	~23.8 GB/card	2026-05-18	First dual-3090 PCIe x4 data point. Lanes negotiated x4/x4 (GPU max x16); report.sh auto-flagged the narrow slot. Boot log [nvlink] NVLink DISABLED — NCCL_P2P_DISABLE=1, custom all-reduce OFF → this is an x4-lanes / P2P-off number, not a patched-P2P anchor. vs @noonghunna x16-ish baseline 78/127: only −2% narr / −8% code — dual-dflash-noviz is far less interconnect-bound than the x4 warning implies, even without P2P. CV 5.4%/3.2%. For a true patched-P2P anchor see @aaronlockhartdev's validated recipe (#95): patched kernel module + NCCL_P2P_LEVEL=PHB + custom_all_reduce ENABLED (the vLLM cuda.py patch is unnecessary). First in-the-wild exercise of the #156 lspci report subsection (sudo-less degradation note rendered correctly). Issue #158
dual.yml + NVLINK_MODE=pcie_p2p (PR #291)	@hlo-world (2× 3090 PCIe x4/x4, Ryzen 5 5600, Ubuntu 24.04, aikitoria patched driver 595.71.05 — same rig as #158 above)	fp8	262K	77 / 103	—	~22.4 GB/card	2026-06-01	First validation of the shipped NVLINK_MODE=pcie_p2p mode (PR #291) — the one rig that can engage PCIe-P2P (patched driver, no NVLink). Boot: NVLINK_MODE=pcie_p2p — NCCL_P2P_LEVEL=PHB, custom all-reduce ON. Controlled same-rig A/B: P2P off (NCCL_P2P_DISABLE=1) 70 / 90 → P2P on 77 / 103 = +10% narr / +14% code. Confirms the #290 regression fix end-to-end. Larger lift than @aaronlockhartdev's x16 dual.yml (+2%/+9%, #91) — consistent with x4 lanes making the link more of a bottleneck, so P2P helps more. (Config runs MTP n=3 per the Qwen3_5MTP boot arch; the issue titles say "dflash".) Issues #294 (off) + #295 (on).
dual.yml	@oven1231231234 (2× 3090 NVLink, PCIe 3.0 x16)	fp8	262K	67.24 / 86.14 (decode 67.89 / 87.80)	—	~23.9 GB/card	2026-06-26	NVLink arm of a same-rig NVLink-vs-PHB toggle (#488). soak-continuous PASS (p50 decode 122.6, 0 err, 0 MiB growth, 0/25 silent-empty, 100% retention). TTFT 143 ms, CV 2.2%/1.6%. MTP AL 3.41–3.54, ~84% draft accept. Within-rig +56–60% over the PHB arm below (consistent across soak/narr/code) — but NOT power-matched (325/298 W here vs 228/253 W on PHB; benches ~8 h apart) and larger than the controlled #77 A/B (+15% at pinned 230 W). PCIe 3.0 x16 ≈ 4.0 x8 (#77) in bandwidth, so the bus doesn't explain the gap → treat the magnitude as an upper bound pending a power-pinned re-run.
dual.yml	@oven1231231234 (2× 3090 PHB / no-P2P, PCIe 3.0 x16)	fp8	262K	42.74 / 53.53 (decode 43.15 / 54.84)	—	~24.1 GB/card	2026-06-26	PHB arm (true no-bridge topology; pair with the NVLink row above, #488). soak-continuous PASS (p50 decode 78.5). TTFT 220–230 ms; lopsided util (89/39%) consistent with allreduce serialization. Narrative 43 TPS is below a single 3090 (~45–50) → suspect power-throttle and/or a pathological no-P2P path on this board, not pure interconnect cost; see #488 for the power-matched re-run ask.
carnice-bf16mtp.yml	@noonghunna (2× 3090 PCIe, no NVLink)	fp8	262K	72 / 80	—	~22.25 GB	2026-05-04	Carnice-V2-27B (Hermes agentic fine-tune) + BF16 MTP overlay. Full 262K context, 2 streams. 71.75 narr / 80.35 code wall TPS (n=5 each, CV ~11%), MTP AL 3.02-3.14, TTFT 141ms. Patched chat template for Hermes JSON tool calls. verify-full 7/8 PASS. soak PASS.
beellama/carnice-v2-dual-q8-mtp 🧪	@noonghunna (2× 3090 PCIe, no NVLink, 370 W cap)	q8_0/q8_0	262K	40.7 / 44.0 (decode, n=5, CV <1.3%)	1197	21.8 / 21.2 GB/card	2026-06-16	Carnice-V2-27B Q8_0 + embedded MTP head on beellama v0.3.2-preview (layer-split, -ts 0.55,0.45) — the quality-max dual Carnice (#403). TTFT 79/78 ms. verify-full all-pass · verify-stress 8/8 (NIAH ladder → 240K, 5.2 GB ceiling margin) · soak fresh 20×5 PASS (0 MiB growth, 0/100 silent-empty, p50 42.2 TPS, 100% retention). 8-pack think-OFF 103/150 / think-ON 105/150 (wash) — in-band (≈ qwopus-coder 103; Q5 single = 110). MTP accept ~81% @ n=1. q8_0 KV chosen over the originally-requested kvarn6: KV A/B = +17% prefill (1003 vs 860 t/s — escapes KVarN software-compression compute, q4=q8=1004 so it's the path not the bit-width), higher fidelity (q8 > kvarn6 q6-class), reference-aligned, and fits 262K on dual. -b/-ub/--no-mmap A/B'd FLAT — KV type was the only prefill lever. n=2 = +13% validated opt-in (DRAFT_N_MAX=2; lossless, lower 62% accept costs speed only). DFlash ruled out (only BASE-27B drafters exist → ~10% accept on the fine-tune; no Carnice-matched drafter). beellama v0.3.2 rolling pre-release → 🧪.
dual.yml ⭐	@lolren (2× 3090 PCIe + Ryzen 9 5950X, 250W/card cap)	fp8	262K	89.78 / 117.60	—	~22.3 GB	2026-05-05	First cross-rig data on the v7.72.2 uplift (image nightly-01d4d1ad3, post-PR #59). +30% narr / +32% code over @noonghunna 2026-04-29 baseline (69/89 on older image) — confirms the v7.72.2 dividend cross-rig. CV 3.3%/2.0%. MTP AL ~3.5, per-pos accept 94/84/72%. Disc #18.
dual-dflash.yml	@lolren (2× 3090 PCIe + Ryzen 9 5950X, 250W cap)	FP16	185K	87.10 / 142.0	—	~22.1 GB	2026-05-05	Older image nightly-7a1eb8ac2. +6% narr / +14% code over @noonghunna baseline (82/125) — likely Ryzen 5950X advantage on prefill. DFlash AL ~4.5, per-pos accept 93/81/68/56/48%, avg accept 69%. Disc #18.
bounded-thinking.yml	@lolren (2× 3090 PCIe + Ryzen 9 5950X, 250W cap, MTP-disabled-suspected)	TQ3	180K	64.86 / 64.96 (CV 0.1%)	—	~22.3 GB	2026-05-05	Anomaly: lolren reports "no spec-decode" on this run despite bounded-thinking.yml shipping --speculative-config mtp n=3 by default. Near-identical narr=code TPS + extreme CV stability (0.1%) suggests MTP was inactive — likely because his image was older nightly-7a1eb8ac2 (pre-v7.72.2 + pre-PN35). Re-test on nightly-01d4d1ad3 should restore MTP path → expect ~50/66 narr/code with normal CV. Tracked. Disc #18.
dual.yml	@JDWarner (Mixed RTX A5000 + RTX 3090, both Razer Core X eGPU enclosures over Thunderbolt 3, Intel NUC11TNH i5-1135G7, 16 GB RAM, headless, A5000=230W cap / 3090=290W cap, PCIe x4 Gen 3 per card)	fp8	262K	56.83 / 72.47 (soak p50 93.09)	—	~23.6 GB/card	2026-05-09	Soak: ✓ PASS (5×5, 0 errors, 0 silent-empty, 100% TPS retention, 0 MiB growth). First TB3 dual-eGPU + mixed-arch cross-rig data. The setup that "shouldn't work": each card on a separate TB3 controller → ~3.94 GB/s effective per card vs ~32 GB/s on PCIe x16 Gen 4 (~8× cut), mixed Ampere SKUs (workstation A5000 + consumer 3090 with different mem bandwidth + clocks), 16 GB system RAM total. Result: matches dual.yml PCIe x16 baseline within run-to-run noise — confirms decode on Qwen3.6-27B is per-card-bandwidth bound, cross-card NCCL allreduce is small enough that even an 8× link cut doesn't dominate. Extends @aaronlockhartdev's #91/#95 finding (patched-P2P only +2%/+9% on dual.yml) in the opposite direction: even with 8× less cross-card bandwidth, decode holds. MTP AL 3.39-3.52, per-pos accept 0.93/0.83/0.70 (89% avg). verify-full + verify-stress all PASS. Genesis pin 7b9fd319 (v7.72.2). Issue #107.
dual/docker-compose.yml (default)	@ygafarov (3090 via USB4 eGPU dock + 5070 Ti via OCuLink — heterogeneous Ampere + Blackwell consumer dual-eGPU, AMD Ryzen AI MAX+ 395 / Strix Halo miniPC, CachyOS, 123 GB RAM, 290 W cap both cards, PCIe x4 per card — USB4 ≈ 3.94 GB/s, OCuLink ≈ 7.88 GB/s)	fp8	200K	65.10 / 85.81	—	17.1 / 15.7 GB	2026-05-12	First heterogeneous Ampere + Blackwell consumer dual-eGPU on the matrix. TP=2 bound by the slower USB4 link in allreduce + sm_86 kernels (5070 Ti spends back-half of step waiting — 91% util but only 125 W out of 290 W cap). KV pool 200K @ 1.00× concurrency — VRAM cap from the 5070 Ti's 16 GiB (model takes 13.8 GiB/card → only ~2.2 GiB left for KV on the smaller card). verify-stress 8/8 incl. 91K needle recall (Cliff 2 clean). Soak ⚠ borderline (360 MiB > 200 MiB threshold — same eGPU-bus accretion as ygafarov's own #113 single-card row above at 240 MiB; 100% TPS retention + 0 silent-empty + 0 errors so not a leak). MTP AL 3.50, per-pos accept 0.94/0.86/0.70. CV 4.5%/1.8%. Slower than ygafarov's own single-3090 #113 row (68.86/91.70 at 48K) — on this rig the single-card path is recommended; the 5070 Ti adds VRAM cap pain without TPS gain. Driver 595.71.05, vLLM nightly-1acd67a79, no Genesis (Blackwell consumer not on allowlist). Issue #120.
dual.yml	@OVDEN13 (2× 3090 + Ryzen 7 5700X, PCIe Gen 4 x4 + x8 asymmetric, 300 W cap, no NVLink, Ubuntu 26.04 / driver 595.58.03 / CUDA 13.2, 92 GB RAM, bare metal)	fp8	262K	75.80 / 99.04	—	22.3 GB/card	2026-05-15	First asymmetric PCIe-lane data point on the matrix (Gen 4 x4 to one card, x8 to the other — half the cross-card bandwidth of @danbedford's Gen 4 x16 baseline). −15% narr / −14% code vs @danbedford's same-compose PCIe-only A/B (89.24 / 114.57, #77) — gap consistent with the Gen 4 x4 card being the bottleneck for cross-card NCCL allreduce. MTP AL 2.71–3.51, per-pos accept ranges 0.81/0.56/0.35 → 0.95/0.85/0.72; the wide variance and lower floor vs @lolren's stable 0.94/0.84/0.72 (disc #18) suggest draft-forward sync stalls on the slow lane. Both GPUs pegged at 297 W power cap (80% / 87% util — mildly asymmetric, consistent with x4+x8 imbalance). CV 1.2% narr / 4.2% code. Soak: ✓ PASS (20×5 fresh mode, p50 101.41 TPS, p95 TTFT 1645 ms, 0 errors, 0 silent-empty, 100.7% retention, 0 MiB growth). Stack: master @ b1c68b4 (v0.7.2-7-g, pre-v0.7.3 tag), vLLM nightly-1acd67a795eb, Genesis pin 7b9fd319. Issue #142.
dual.yml	@mgabor3141 (2× 3090, ASUS PRIME X399-A + AMD TR 1950X (Zen 1, 16c/32t, 2017) + 16 GB DDR4 (single-DIMM), GPU 0 + GPU 1 both PCIe 3.0 ×16, 280 W cap, no NVLink, CachyOS bare metal, headless, iommu=pt)	fp8	262K	58.98 / 79.19	—	~22.6 GB/card	2026-05-21	Validates the v0.7.3 HARDWARE.md prediction for PCIe Gen 3 + pre-Zen3 rigs (predicted band 53.9-62.9 / 70.6-82.3, lands mid-band). Same-rig hardware-only upgrade from the Z77 + 3770K + Gen 2 ×4 row below: +8% narr / +15% code on fixing the cross-card PCIe bottleneck. Soak: ✓ PASS clean (p50 114.02, 0 errors, 100% retention, 0 MiB growth). verify-stress 8/8 incl. 91K needle. Required iommu=pt on kernel cmdline (AMD-Vi Translated mode crashes both GPUs with Xid 154 under sustained TP=2 DMA; diagnosis in #178 comment, now documented in HARDWARE.md). vLLM nightly-bf610c2f (v0.7.3 routing puts non-TQ3 composes here). [Issue #178].
dual.yml	@mgabor3141 (2× 3090, ASUS P8Z77-V LX + i7-3770K Ivy Bridge, 2012 + 16 GB DDR3, GPU0 PCIe 3.0 ×16 / GPU1 PCIe 2.0 ×4 ≈ 2 GB/s, 280 W cap, no NVLink, CachyOS bare metal, headless)	fp8	262K	54.46 / 68.64 (decode 54.93 / 70.27)	—	~22.6 GB/card	2026-05-21	Slowest cross-card link on the matrix (~2 GB/s — half of @JDWarner's TB3 dual-eGPU). The "slap 3090s into e-waste" baseline — and it's serviceable. Sits just below @JDWarner's TB3 row (56.83 / 72.47, ~3.94 GB/s), confirming the per-card-bandwidth-bound thesis holds even at ~2 GB/s cross-card — extends @aaronlockhartdev #91/#95 + @JDWarner #107. TTFT 158 / 171 ms, CV 1.6–2.2%. MTP AL 3.46, per-pos accept 95/86/75% (avg 81.9%). Soak: ✓ PASS clean (p50 99.84, 0/25 silent-empty, 100% retention, <200 MiB growth); verify-stress 8/8 + quality 101/150 all PASS. Cliff 2 doesn't fire (TP=2 splits the GDN forward). vLLM nightly-bf610c2f. Issue #178.
dual.yml-shape (hand-rolled, stock vllm/vllm-openai:nightly, NO Genesis) — Proxmox VFIO GPU-passthrough VM	@duart (2× 3090 NVLink, Dell T630 + 2× Xeon E5-2695 v4 Broadwell, 256 GB DDR4, PCIe 3.0 ×16, models on ramdisk)	fp8	262K	76.46 / 75.00 (decode 77.86 / 77.79)	—	~21.5 GB/card	2026-05-21	First Proxmox-passthrough + enterprise-server datapoint. Self-solved 41/55 → 76/78: root cause was NOT the report's idle LnkSta 2.5 GT/s (that's PCIe ASPM at idle — trains to 8 GT/s under load) but enabling MTP FULL_AND_PIECEWISE cudagraph (+50% vs PIECEWISE-only). Stock upstream nightly v0.21.1rc1.dev132 without Genesis → ~15% below @lolren's Genesis dual.yml (89.78 / 117.60) and ~30% below the NVLink Genesis rows — the gap is the v7.72.2 Genesis dividend, not the rig/virtualization. NUMA-aligned (vCPUs+mem+GPU same node), 8 vCPU sufficient (vLLM uses ~3). CV ~1%. verify-stress 8/8 incl. 91K Cliff 2 needle; soak PASS clean (p50 92.90, 0/25 silent-empty, 0 MiB growth, 100% retention). Disc #162.

Quad-card (4× RTX 3090, TP=4)

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
multi4.yml	@whamp (4× 3090 PCIe x4/x16/x8/x16, 300 W cap, no NVLink)	fp8	262K	63 / 76	—	~23.5 GB	2026-05-03	TP=4 capacity king. 6.77× concurrency at 262K. PASSES v2 continuous soak (20 sessions, 0 MiB growth, 90.8% TPS retention). PR #44.
multi4.yml	@alanspires #127 (6× 3090 VFIO-passthrough, AMD EPYC 7313 host, all cards 250W cap, no NVLink)	fp8	262K	74.93 / 92.85	—	~21.7 GB	2026-05-14	TP=4 on a 6-card rig — GPUs 4-5 free (Qwen num_kv_heads=4 doesn't divide 6, so TP=6 invalid). First VFIO-passthrough + virtualized data point — scripts (verify-full / verify-stress / soak) all PASS on the virt envelope without modification. KV pool 1,774,963 tokens at 6.77× concurrency. Soak p50 122.84 / p95 161.14 across 5 multi-turn sessions, 0 errors.
multi4-dflash.yml	@whamp (4× 3090 PCIe x4/x16/x8/x16, 300 W cap)	fp8	262K	64 / 104	—	~22.0 GB	2026-05-03	TP=4 + DFlash. 2.27× concurrency at 262K. PASSES v2 continuous soak (5 sessions, 0 MiB growth, 100% TPS retention). Bench-vs-soak inversion: bench shows DFlash wins by 37% on short-prompt code, soak shows DFlash loses by 47% on multi-turn agent — DFlash AL likely collapses on mixed prompts. PR #44.
multi-max	@Whamp (4× 3090 PCIe x4/x16/x8/x16, 230 W, aikitoria 595.71.05-p2p kernel)	int8-PTH	262K	85.09 / 102.43	—	~23 GB/card	2026-06-19	First real 4-card TP=4 validation of vllm/qwen-27b-multi-max (shipped 🧪 with duals as proxies). fp8 weights + INT8-PTH KV. verify-full + verify-stress 7/7 (Cliff 2) + soak-continuous PASS (0 err, 0/25 silent-empty, 0 MiB growth, p50 80.92, 100% retention). TP=4 narr (85) ≈ our 2-card TP=2 → TP scaling flat past TP=2 on PCIe; multi4's value is the 6.77× KV pool, not single-stream TPS. P2P kernel installed (engagement in this run TBD — see thread). Issue #446.
multi-fast	@Whamp (same rig as above)	fp8	262K	59.00 / 73.88	—	—	2026-06-22	Same-session baseline for the int8-PTH A/B below — INT4 weights + fp8 KV. Issue #446.
multi-fast + KV_CACHE_DTYPE=int8_per_token_head	@Whamp (same rig)	int8-PTH	262K	92.50 / 121.85	—	~22.2 GB/card	2026-06-22	Standout TP=4 config — INT4 weights + INT8-PTH KV. vs same-session multi-fast fp8 KV (59/74) = +57% narr / +65% code from the KV format alone (far bigger than anything int8-PTH buys at TP=2, where the KV format is ~a throughput wash — worth understanding). vs multi-max fp8-weights (85/102) = +9%/+19% from INT4-vs-fp8 weights. MTP accept 3.38–3.54. Candidate for a shipped INT4+int8-PTH multi4 config if the KV win holds without P2P. Issue #446.

Verify-stress + soak-continuous matrix

Not TPS, but load-bearing. Every shipped variant is validated against:

• bash scripts/verify-full.sh — fast functional smoke (8 checks)
• bash scripts/verify-stress.sh — boundary tests including Cliff 2 needle recall (probe 7: 60K + 90K needles)
• SOAK_MODE=continuous bash scripts/soak-test.sh — multi-turn accumulating-context cliff (Cliff 2b at ~25K)

Variant	Rig	verify-full	verify-stress 8/8	soak-continuous	Date
minimal.yml (single-card vLLM)	@noonghunna	PASS	PASS at 64K	FAIL — Cliff 2b fires	2026-05-03
long-text.yml	@noonghunna	PASS	PASS at 180K	FAIL — Cliff 2b fires	2026-05-03
long-vision.yml	@noonghunna	PASS	PASS at 145K	FAIL — Cliff 2b fires	2026-05-03
bounded-thinking.yml	@noonghunna	PASS	PASS at 180K	FAIL — Cliff 2b fires	2026-05-03
tools-text.yml	@noonghunna	PASS	PASS at 75K	FAIL — Cliff 2b fires	2026-05-03
llamacpp/default	@noonghunna	PASS	PASS at 262K	PASS — different engine, no cliff	2026-04-21
dual.yml (TP=2)	@noonghunna	PASS	PASS at 262K (237K single-prompt)	PASS	2026-05-03
dual-turbo.yml (TP=2)	@noonghunna	PASS	PASS at 262K	PASS (assumed by activation-split argument; not yet measured cross-rig)	2026-04-29
dual-dflash.yml (TP=2)	@noonghunna	PASS	PASS at 185K	TBD	—
dual-dflash-noviz.yml (TP=2)	@noonghunna	PASS	PASS at 200K	TBD	—
multi4.yml (TP=4)	@whamp	PASS	PASS at 262K (incl. 58K + 91K needles)	PASS (20 sessions, 0 MiB growth, 90.8% retention)	2026-05-03
multi4-dflash.yml (TP=4)	@whamp	PASS	PASS at 262K (incl. 58K + 91K needles)	PASS (5 sessions, 0 MiB growth, 100% retention; ⚠ 4 turns >30s; n=5 small)	2026-05-03

The single-card vLLM Cliff 2b status is canonicalized in #41 — fix is gated on upstream Sandermage genesis-vllm-patches#19. See docs/CLIFFS.md for the byte-level explanation.

Cross-engine — Luce DFlash (lucebox-hub) on Qwen3.5-27B

Not directly comparable to vLLM rows above (different engine, different bench script, different model — Qwen3.5-27B not 3.6 because the 3.6 DFlash draft is still under training as of 2026-05-04). Bench harness: lucebox-hub/dflash/scripts/bench_he.py, HumanEval 10 prompts, n_gen=128.

Config	Rig	Mean tok/s	AL	Accept %	Notes
Same-card, default KV	@noonghunna (1× 3090)	73.97	6.39	41.3%	Range 52.7–108.7 across 10 HE prompts. Bench 2026-05-04.
Same-card, K8V4 (-ctk q8_0 -ctv q4_0)	@noonghunna (1× 3090)	74.68	6.38	41.4%	Range 54.6–109.1. +1% over default KV — basically identical. KV-format optimization doesn't help at HE-scale (<150-tok prompts × 128-tok gen) where KV pool isn't the bottleneck. Asymmetric quant via PR #56/#54 merged 2026-04-28.
Dual-GPU split (PR #80 --target-gpu 0 --draft-gpu 1 --draft-feature-mirror)	@noonghunna (2× 3090, no NVLink, P2P "Chipset Not Supported")	75.24	6.39	41.3%	Range 54.2–110.0. +1.7% over same-card — but NOT a fair test of the split's value. CUDA P2P access is disabled at the chipset level on this rig (PHB topology, consumer-board limitation). The lucebox dual-GPU code path requires P2P for direct draft-feature transfers; without it, falls back to host-staging copies (CPU↔GPU bouncing). The published 51.86 tok/s on dual 2080 Ti 22GB (PR #80) presumably ran with P2P available. Verdict for our hardware class: dual-GPU split needs a P2P-capable interconnect (NVLink or peer-supported chipset) to deliver its value. PHB+CNS rigs see no benefit.

PFlash long-context compression on 1× 3090 — measured ceiling 131K source

Bench harness: lucebox-hub/dflash/scripts/phase_split_dual_gpu.py bench-niah (PFlash drafter only, no target loaded — measures the prefill compression phase). Drafter: Qwen3-0.6B-BF16.gguf, BSA enabled, keep_ratio=0.05.

Source ctx	Compressed	Ratio	PFlash time	tok/s	Key + answer retained
16,372	788	0.048	1.08 s	15,117	✓ ✓
32,764	1,628	0.050	1.80 s	18,205	✓ ✓
65,524	3,252	0.050	4.37 s	15,009	✓ ✓
131,068	6,524	0.050	10.80 s	12,135	✓ ✓
199,996	OOM at layer 25 (390 MiB ephemeral alloc)	—	—	—	✗ ✗
259,996	OOM at layer 18 (507 MiB ephemeral alloc)	—	—	—	✗ ✗

Compression-phase result: PFlash drafter scoring works up to 131K source on 1× 24 GB / 3090 — compresses to 6.5K (5%) in 10.8s with NIAH key + answer retained. Vanilla llama.cpp pp131072 takes ~257s per Luce's published numbers, so the compression phase alone is ~24× faster at this context. Adding target prefill on the compressed 6.5K would estimated ~1-2s (untested), suggesting ~12-13s end-to-end TTFT vs ~257s vanilla.

Above 131K source the drafter's ephemeral forward-pass tensors (K_curr/V_curr/Q_last per layer at full sequence length) exceed 24 GB. K-cache quantization (--pflash-k-type q8_0) didn't help — the failing allocs are forward-pass not cache. Bench lucebox-pflash-niah-q8k-20260504-150600/ confirmed identical OOM at 200K and 260K with both BF16 and q8_0 K cache.

On the @weicj 24K → 262K phase-split claim (PR #78): not refuted but not reproduced on our hardware class either — their setup was 2× 22 GB Ti with target also loaded on one card; "24K single-card" was target+drafter co-resident. Our 131K is drafter-alone on 24 GB, which already passes their dual-GPU 262K-style scaling sanity-check. Reproducing 262K specifically would need investigation of their drafter config (chunk_size, lookahead, BSA window) — drafter activation footprint at 200K+ is the binding constraint regardless of how many GPUs are present.

What we have NOT validated — gates before "shippable"

This is directional evidence (TTFT compression + NIAH retention at 131K), not a complete validation. The gates we hold every other shipped compose to are still open for PFlash:

Gate	Status	Notes
TTFT speedup at long context	✅ measured (~24× compression alone)	Single test — needs reproduction across prompt shapes
NIAH single-needle retrieval	✅ measured at every ctx ≤131K	Synthetic test only — single key+answer pair per prompt
Target prefill on compressed tokens	❌ unmeasured	Bench harness measures PFlash phase only
Decode TPS after compressed prefill	❌ unmeasured	End-to-end TTFT + decode pipeline not tested
HumanEval+ / LCB v6 pass@1	❌ not applicable — those benches have <2K-token prompts; PFlash's compression path wouldn't even engage	Need long-context coding benches (repo-understanding, RULER+code)
Long-context QA accuracy (RULER, LongBench, multi-needle)	❌ unmeasured	The actual quality gate — does compression preserve task performance, not just synthetic needle retrieval?
verify-stress.sh 7/7	❌ 0/7 PASS (2026-05-04, see below)	OpenAI server gate — multiple distinct failures
SOAK_MODE=continuous	❌ blocked	Daemon dies during stress; soak can't run on a dead daemon
Multi-turn compression stability	❌ blocked by 3rd-cycle CUDA bug	Daemon hits illegal-mem-access on 3rd compress regardless of GPU layout

verify-stress on PFlash-enabled lucebox server (2026-05-04, single-card and dual-GPU)

Tested via URL=http://localhost:8004 MODEL=luce-dflash bash scripts/verify-stress.sh against a lucebox-hub/dflash/scripts/server.py boot with --prefill-compression auto --prefill-threshold 8000 --prefill-keep-ratio 0.05 + Qwen3-0.6B-BF16 drafter. Two configurations:

• Single-GPU: target+dflash draft+pflash drafter all on GPU 0 → OOM at 75 MiB on 2nd request, daemon exits, all subsequent probes 503. Logs: results/lucebox-pflash-verify-stress-20260504-152713/.
• Dual-GPU: local server.py patch reading LUCEBOX_TARGET_GPU=0 LUCEBOX_DRAFT_GPU=1 LUCEBOX_DRAFT_FEATURE_MIRROR=1 to pin dflash draft to GPU 1. Probes 1 (10K + 30K) survive but return wrong needle answers because PFlash @ keep=0.05 drops the needle phrase. Probe 2 (25K tool prefill) crashes the daemon with CUDA error: an illegal memory access was encountered on the 3rd compress cycle. Probes 3-7 all 503. Logs: results/lucebox-pflash-verify-stress-dualgpu-20260504-153220/.

Probe	Single-GPU	Dual-GPU	Failure mode
1. 10K + 30K needle	✗ blank reply	✗ wrong content	PFlash drops needle phrase from top-5% kept chunks
2. 25K tool prefill	✗ daemon dead	✗ HTTP 500 → daemon dies	CUDA illegal-mem-access on 3rd pflash compress
3. IDE-agent	✗ HTTP 503	✗ HTTP 500	Cliff 1 mech B class on lucebox path; daemon already dead in single-GPU run
4-6. Multi-turn / LCB / reasoning	✗ HTTP 503	✗ HTTP 503	Daemon died at probe 2
7. 60K + 90K needle	✗ HTTP 500	✗ HTTP 500	Daemon dead

Two distinct failure classes in the integrated PFlash + DFlash + OpenAI server path:

1. Compression-vs-retrieval at keep=0.05: short factual needles (color animal num) don't survive top-15-chunks selection in 10-30K contexts. The standalone NIAH bench (#230) used a needle/filler pattern PFlash's importance scorer favors; verify-stress's pattern doesn't replicate that. Means PFlash's "key+answer retained" claim is filler-pattern-dependent at moderate contexts.
2. 3rd-cycle multi-cycle daemon stability: lucebox-hub upstream bug — CUDA illegal-mem-access in ggml_backend_buffer_free after the 3rd pflash compress cycle, regardless of single- vs dual-GPU. Daemon doesn't recover; subsequent requests 503.

Honest read: PFlash gives us a compelling single-stat win (24× TTFT compression + NIAH-retention) at 131K source on 1× 3090 in the standalone bench harness, but the integrated OpenAI server path fails the verify-stress gate. PFlash is not a shippable club-3090 path today. Re-evaluate when (a) lucebox-hub fixes the 3rd-cycle CUDA stability bug, (b) --pflash-gpu lands in server.py (currently only standalone bench has it), and (c) the importance scorer / keep ratio reliably preserves arbitrary short needles. Long-context QA harness (RULER or similar) is the meaningful next investment if any of those three land.

Setup gotcha for anyone re-running on consumer rigs: check nvidia-smi topo -p2p r before configuring --target-gpu / --draft-gpu. If the matrix shows CNS (Chipset Not Supported), the dual-GPU split won't deliver its claimed uplift on that hardware regardless of whether you have multiple GPUs. NVLink-bonded setups would also typically expose P2P (a different cross-rig contributor would need to confirm on lucebox specifically; @JusefPol's #31 NVLink win was measured on vLLM TP=2, not lucebox). PHB-only consumer boards typically lack P2P.

---

Qwen3.6-40B-Deckard

Dense 40B uncensored community merge of Qwen3.6 (DavidAU Opus-Deckard). Q6_K GGUF with a BF16 MTP head injected by PiehSoft (PiehSoft/Qwen3.6-40B-Deckard-MTP-Q6_K). Arch is qwen35-dense (standard GQA, 97 layers) per the GGUF header — not a Qwen3-Next hybrid. Dual 3090 only (31 GB > 24 GB). llama.cpp mainline, immutable pin server-cuda-b9570.

Dual-card (2× RTX 3090) — llama.cpp

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
mtp.yml (MTP n=2)	@noonghunna (2× 3090, PCIe)	q8_0/q8_0	131072	36 / 46 (MTP on)	—	~16.4+17.8 GB @16K	2026-06-09	✅ Production. MTP n=2 sweet spot (41.60 tok/s aggregate, 0.81 accept). MTP off: 22.7 narr (+59%/+104% with MTP). 128K ceiling @q8_0 KV (192K OOMs). 8-pack (think-off): 105/150 — MTP-off == MTP-on (spec-dec lossless); det 62/75 (toolcall 15·instructfollow 14·structoutput 13·dataextract 10·reasonmath 10) + sandbox 43/75 (bugfind 13·hermesagent 15/20·cli-40 15/40). verify-full 8/8, verify-stress 8/8, soak-continuous PASS (0 MiB growth, 0/25 silent-empty, 25 turns). ≈ Qwen3.6-27B band. Arch confirmed qwen35-dense from GGUF header.

---

See also

• docs/SINGLE_CARD.md — single-card variant picker
• docs/DUAL_CARD.md — 2-card variant picker
• docs/MULTI_CARD.md — 4+ card variant picker
• docs/STRUCTURED_COT.md — bounded-thinking benchmark on HumanEval+ + LiveCodeBench v6
• docs/CLIFFS.md — known failure modes and which variants escape them
• CONTRIBUTING.md — how to add a row

---

Gemma 4 31B (community-experimental)

2026-05-29 cleanup: active vLLM Gemma 31B registry is now vllm/gemma-mtp, vllm/gemma-int8, and vllm/gemma-mtp-tp1, pinned to immutable vllm/vllm-openai:v0.21.0; DFlash, AWQ, TQ3, bf16-long, and the separate 262K alias are retired from active composes.

2026-05-31 update: the two dual slugs were renamed to name the real difference (both carry MTP n=4; only the KV format differs) — vllm/gemma-mtp → vllm/gemma-bf16-mtp (BF16 KV) and vllm/gemma-int8 → vllm/gemma-int8-mtp (INT8 PTH KV). gemma-bf16-mtp moved to stock v0.22.0 (no overlay) and its default laddered 32K → 131K under the dual-card max-context-priority rule (docs/DUAL_CARD.md). Then (#287) gemma-int8-mtp ALSO moved to v0.22.0 — #40391 rebased onto v0.22.0 + re-delivered as a lean boot-time diff-apply (was 7 full-module mounts); default ctx raised to native 262K; #41800/#41991 dropped (in stock); #42006 streaming-multi-tool-fix re-instated on both duals (lean diff). Then (#289) --reasoning-parser gemma4 wired on both — without it, reasoning-on left the <|channel> trace in content → structured packs broke (invalid_json); it's a vLLM config gap, not a Gemma limit.

8-pack quality (current grader, --full, 2026-05-31): int8 thinking-OFF 105/150, int8 reasoning-ON 107/150, bf16 reasoning-ON 111/150 — all within the ±5–7 8-pack noise band (reasoning-on ≈ a wash; InstructFollow 8→14 the one clear gain; bf16 ≈ int8 KV on quality). Reasoning-on costs latency (ReasonMath/BugFind p95 ~doubled) for no aggregate gain → thinking-OFF stays the default, reasoning-on is a clean opt-in once the parser is wired. Full per-pack breakdown + serving (TPS / TTFT / VRAM / KV pool) for all three configs: discussions/239.

Cross-rig data on Google's official Gemma 4 MTP "assistant" drafter (released 2026-05-05). PR #41745 merged 2026-05-06 → today's nightly contains it natively (overlay dropped 2026-05-08). The companion compose dual/int8.yml (added 2026-05-08) vendors PR #40391 (rebased) + PR #42006 + PR #41991 to unlock per-token-head INT8 KV → 8.2× context lift on Ampere (32K → 262K). See announcement discussion #67 for the original Gemma 4 setup story; Phase 2 INT8 PTH validation in progress 2026-05-08.

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	AL	Per-pos accept (code)	Peak VRAM	Date	Notes
dual/qat-awq-int4/base.yml (TP=2, max-num-seqs=2, MAX_MODEL_LEN=229376) — v0.24.0 ⭐⭐ prod default	@noonghunna (2× 3090 PCIe)	bf16	224K	~59 decode (code, CV 0.1%; soak p50 58.7)	~420–716 (depth-dep)	n/a (MTP-off)	—	~23 GB/card (47,088 MiB total)	2026-07-02	vllm/gemma-31b-dual — cyankiwi QAT-AWQ-int4, STOCK vLLM v0.24.0, OVERLAY-FREE. The v0.24.0 consolidation 31b — folds onto vllm-stable, retires the 31b's vllm-gemma-stable dependence. bf16 KV (not int8-PTH): on v0.24.0 int8-PTH allocates 262K but SILENTLY craters recall past ~32K (needs #40391, open/unmerged upstream — verified 2026-07-01: both cyankiwi + w4a16 crater identically; the SAME cyankiwi weights recall clean to 112K+ on v0.22.0+#40391). bf16 has NO cliff: verify-stress ALL 5 ceiling rungs to 210K (91%), VRAM margin 1162 MB > 1024 (the 0.97/245K thin-margin flag resolved at 0.95/224K). verify-full 9/9 (tools + streaming-tool-calls + reasoning, MTP-off). Soak PASS (0 err / 0 MiB growth / 0-of-100 silent / 99.6% retention). MTP DISABLED — Gemma-4 MTP×tools broken on v0.24.0 (#39043; #42006 unmerged). Trades ~224K vs int8-mtp's 262K for zero overlays; int8-PTH 262K returns when #40391 merges. Supersedes gemma-int8-mtp / bf16-mtp / qat-w4a16 (all now 🗑️ Deprecated).
dual.yml (TP=2)	@noonghunna (2× 3090 PCIe, no NVLink, 230W cap)	bf16	32K	108.87 / 142.25	—	3.94-4.04	92 / 79 / 68 / 59 %	22.5 GB/card	2026-05-05	First Ampere consumer cross-rig data on Google MTP drafters. +1.79× narr / +2.31× code over baseline (61 TPS no-spec-decode same TP). PASSES continuous soak (100 turns, 0 errors / 0 silent-empty / 0 MiB growth, 98.3% TPS retention). bf16 KV (fp8 blocked on Ampere — see TP=1 row). PR #41745 overlay + transformers 5.8.0 entrypoint.
dual.yml (TP=2) re-bench post-#41745 merge	@noonghunna (2× 3090 PCIe, 230W cap)	bf16	32K	105.91 / 141.11	—	3.94	(warming)	21.5 GB/card	2026-05-08	Re-validated on post-merge nightly 1acd67a795... (PR #41745 overlay dropped, transformers entrypoint upgrade dropped). Within CV of 109/142 baseline → cleanup is parity-clean. KV pool 99K tokens, 3.03× concurrency at 32K.
dual/bf16-mtp.yml (TP=2, max-num-seqs=4, MAX_MODEL_LEN=131072) ⭐	@noonghunna (2× 3090 PCIe, 230W cap)	bf16	131K	118.83 / 154.26 (CV 2.1% / 2.8%)	—	3.87-3.98	90 / 80 / 69 / 56 %	22.2 GB/card	2026-05-31	Max-ctx-priority default (was 32K) — vllm/gemma-bf16-mtp on stock v0.22.0, NO overlay. BF16 KV pool 196,527 tok @ mem-util 0.95 (measured, vLLM boot log); max-num-seqs=4 is a CAP not a reservation, so raising the ceiling is ~free for short-request concurrency (see docs/DUAL_CARD.md). 4.0× context lift over the 32K bf16 ceiling at zero TPS cost (118/154 ≥ the 109/142 baseline, within CV). PASSES verify-stress 8/8 incl. NIAH at 57K / 90K / 94K / 120K (sapphire narwhal 21 recalled at 120,329 tok = 91% of n_ctx). ⚠️ vision + near-max ctx: only ~1.4 GB/card free at 120K single-stream → drop max-num-seqs/mem-util for vision-heavy long-ctx.
dual-int8.yml (TP=2, max-num-seqs=4) ⭐	@noonghunna (2× 3090 PCIe, 230W cap)	int8_per_token_head	98K	96.16 / 127.11	—	3.79	(warming)	22.2 GB/card	2026-05-08	3.07× context lift over bf16 ceiling on Ampere — INT8 PTH KV unblocks Gemma 4 long-context. Vendors PR #40391 rebased + PR #42006 + PR #41991 stacked (see models/gemma-4-31b/vllm/patches/). KV pool 354K tokens, 3.6× concurrency. ~10% TPS cost vs bf16 / 32K. PASSES verify-stress 8/8 incl. 91K Cliff-2 needle. PR #40391's per-token-head page-size fix routes via get_padded_attention_kv_cache_shape(); INT8 (not fp8) is the right Ampere dtype because Triton fp8e4nv kernel is not supported on sm_86 (Ada/Blackwell only).
dual-int8.yml (TP=2, max-num-seqs=1, MAX_MODEL_LEN=262144) ⭐⭐	@noonghunna (2× 3090 PCIe, 230W cap)	int8_per_token_head	262K (model native max)	95.27 / 125.93	—	3.93	(warming)	22.1 GB/card	2026-05-08	8.2× context lift vs dual.yml — full Gemma 4 native context (262144) unblocked on dual 3090 Ampere. KV pool 455K tokens, 1.74× concurrency at full 262K. PASSES verify-stress 8/8 + 137K NIAH PASS (correctly recalled needle from 137,557-token prompt, 5min wall, ~458 prefill TPS). Per-token TPS preserved at full max-model-len (95/126 at 262K vs 96/127 at 98K — bench prompt size dominates, not max-model-len). Override MAX_MODEL_LEN=262144 MAX_NUM_SEQS=1.
int8.yml (TP=2, max-num-seqs=4, MAX_MODEL_LEN=262144) — v0.22.0 ⭐ prod default	@noonghunna (2× 3090 PCIe, 230W cap)	int8_per_token_head	262K	106.31 / 139.51 (CV 1.8% / 2.0%)	—	3.68-3.94	88 / 73 / 59 / 48 %	22.1 GB/card	2026-05-31	vllm/gemma-int8-mtp on stock v0.22.0 — #40391 rebased onto v0.22.0 + lean boot-time diff-apply (patches/vllm-pr40391-v0.22.0/, was 7 full-module mounts) + #42006 streaming-tool-fix + --reasoning-parser gemma4. KV pool 447,199 tok, 1.72× concurrency at 262K. Soak (continuous) PASS (0 err / 0 silent-empty / 0 VRAM growth / 100% TPS retention / p50 87.96). Quality (--full, thinking-off) 105/150 (det 59/75 · sandbox 46/75); reasoning-on A/B with the parser = 107/150 (wash, +2 within noise — IF+6/BF+2 vs RM-2/HA-2 at higher latency → thinking-off stays default, reasoning-on is opt-in). TPS ≈ the v0.21.0 rows (no regression from the v0.22.0 move + overlays).
qat-w4a16/int8.yml (TP=2, n-swept default n=3, MAX_MODEL_LEN=262144) — 🧪 QAT-int4 A/B	@noonghunna (2× 3090 PCIe, 370W)	int8_per_token_head	262K	73.96 / 87.72 (CV 1.7% / 1.4%)	—	~2.3	64 / 39 / 25 %	~22.8 GB/card (KV 419K tok)	2026-06-07	vllm/gemma-31b-qat-w4a16-dual — unsloth QAT W4A16 (compressed-tensors int4) A/B vs the autoround-int4 int8.yml. Boots clean on stock vllm-gemma-stable (NO #44494 workaround — tower-based Gemma4ForConditionalGeneration, unlike the 12B unified arch). 8-pack 109/150 vs autoround's 105 (+4, within ±5-7 noise ≈ quality TIE; real instructfollow edge IF 15-vs-8, offset by hermes/cli/TC/RM). BUT weaker spec-decode: AL ~2.3 vs autoround's ~3.9 — the QAT-int4 logits align worse with the assistant drafter, so it's slower (74/88 @370W < autoround's 106/139 @230W despite MORE power; power NOT matched, but the AL gap is the clean signal). n-swept @370W: n2 72.9/86.1 · n3 74.0/87.7 · n4 71.6/87.8 — all within ~3% (the fast acceptance decay caps the spec benefit); n=3 set as the default (top narr + tied code + ~20% less drafting than autoround's n=4). Comparable quality, slower → autoround-int4 (gemma-int8-mtp) stays the default. Boots/serves only; NIAH + soak not yet run.
dual-dflash.yml (TP=2, n=7)	@noonghunna (2× 3090 PCIe, no NVLink, 230W cap)	bf16	32K	95.16 / 167.55	—	~3.0 narr / 5.23 code	89 / 78 / 66 / 57 / 50 / 43 / 39 %	22.7 GB/card	2026-05-06	First Ampere consumer cross-rig data on z-lab Gemma 4 DFlash block-diffusion drafter (vLLM PR #41703 — Codex-rebased onto upstream/main). +2.74× code / +1.56× narr over baseline. PASSES continuous soak (100 turns, 0 errors / 0 silent-empty / 0 MiB growth, 98.6% TPS retention, p50 55.8 TPS — 5.8% higher than n=5). DFlash dominates MTP on code (+18%); MTP wins on narrative (+15%). n-sweep: n=5 109/141 (best narr) → n=6 99/161 (knee) → n=7 95/168 (code-optimal default) → n=8 91/167 (dominated) → n=15 82/172 (past knee). PR #41703 overlay (12 RO-mounted files) + transformers 5.8.0 + nightly e47c98ef.
dual-dflash.yml (TP=2, n=7) re-bench	@noonghunna (2× 3090 PCIe, 230W cap)	bf16	32K	104.48 / 176.66 (CV 2.1% / 3.6%)	—	2.85 narr / 4.11-4.94 code	(warm)	22.3 GB/card	2026-05-08	Re-validated after Phase 2 INT8 PTH session. Same overlay + same e47c98ef pin — modest uplift over 2026-05-06 (warm-cache + ambient variance — CV ranges overlap at +1σ). KV pool 42,848 tokens, 1.31× concurrency at 32K. Ampere upper-bound for Gemma 4 + DFlash: code-optimal at 177 TPS. Combining DFlash drafter with PR #40391 INT8 PTH KV (Phase 3, dual-dflash-int8.yml) is the next structural step — would unlock long-context code-optimal.
dual-dflash-int8.yml (TP=2, n=7, MAX_MODEL_LEN=262144, MAX_NUM_SEQS=1) ⭐⭐⭐	@noonghunna (2× 3090 PCIe, 230W cap)	int8_per_token_head + drafter bf16	262K (model native max)	86.86 / 145.96 (CV 0.9% / 2.2%)	—	5.0-5.3 long-ctx code	(warm)	22.0 GB/card	2026-05-08	8.2× context lift over dual-dflash.yml 32K bf16 baseline — DFlash + INT8 PTH KV unblocked on Ampere via vLLM PR #42102 (our patch). Matches the 32K bf16 baseline's code TPS within CV at 8× more context (146 vs 168 = -13% perf cost for 8× ctx). KV pool 168,178 tokens, 0.64× concurrency at full 262K — effective single-stream serving ceiling ~168K. NIAH PASS at 98,444 tokens (bronze octopus 17 recalled cleanly, 157s wall = ~625 effective prefill TPS). DFlash drafter uses BF16 KV in independent pool (target uses INT8 PTH); the patch partitions them at unify-time, drafter cache_dtype overridden to "auto" in qwen3_dflash.py, FA metadata scheduler reads per-spec dtype. n-sweep at 262K config 2026-05-08: n=5 81.91/138.87 (-6/-5%), n=7 86.86/145.96 (default), n=8 86.63/152.06 (+0/+4% but CV 5.7% — within noise). n=7 retained as default — sweet spot didn't shift meaningfully from the 32K bf16 baseline. DFlash code-optimal advantage preserved at long ctx: 146 code TPS vs dual-int8.yml's 126 at 262K = +16% code (offset: -10% narr). Pin nightly e47c98ef; needs dual-dflash-int8.yml compose (still ⚠️ flagged DOES NOT BOOT until PR #42102 lands — currently requires the vllm-src patch mounts).
single.yml (TP=1)	@noonghunna (1× 3090)	bf16 / fp8	—	boot OOM	—	—	—	—	2026-05-05	Upstream-blocked on Ampere consumer. bf16 KV: weights+drafter+profiling at 8K ctx + mem-util 0.95 leaves zero KV pool ("No available memory for the cache blocks"). fp8 KV: Triton fp8e4nv not supported in this architecture on sm_86 (Ampere supports fp8e4b15/fp8e5 only); but fp8_e5m2 is rejected by gemma4_mm.py:1336 allowlist. Compose preserved for re-test when (a) vLLM adds Ampere-aware fp8 dispatch OR (b) PR #41745 relaxes the assert. Gemma 4 26B-A4B MoE single-card is the obvious follow-up.
dual-awq.yml (TP=2, MAX_NUM_SEQS=4, MTP n=4) ⭐	@noonghunna (2× 3090 PCIe, 230W cap)	bf16	65K	104.59 / 130.56 (CV 1.7% / 0.4%)	—	3.07 narr / 3.55-3.88 code	0.79 / 0.59 / 0.43 / 0.31	19.8 GB/card	2026-05-08	Cross-rig reproducer of @3dluvr's #103 bench — AWQ-4bit weights instead of AutoRound INT4. Bypasses PR #40391 (per-token-head bug) entirely because AWQ doesn't use FP8 KV — trades weight quant precision for ctx instead of trading KV precision. Vendor: cyankiwi/gemma-4-31B-it-AWQ-4bit (~17 GB on disk, AWQ-pack-quantized group_size=32, asymmetric, MSE observer; routed via vLLM compressed-tensors loader → Marlin kernel). KV pool 89,228 tokens, 1.36× concurrency at 65K. Numbers comparable to dual.yml (BF16 INT4 AutoRound, 105.91/141.11) on narrative; slightly slower code at this n. Default config — multi-stream agent / RAG.
dual-awq.yml (TP=2, MAX_NUM_SEQS=1, MTP n=8, MAX_MODEL_LEN=118304) ⭐⭐	@noonghunna (2× 3090 PCIe, 230W cap)	bf16	118K	101.16 / 141.90 (CV 2.5% / 0.4%)	—	3.7 narr / 5.13 code	0.89 / 0.77 / 0.64 / 0.54 / 0.45 / 0.37 / 0.26 / 0.21	19.8 GB/card	2026-05-08	3.7× context lift over dual.yml's BF16 32K ceiling. Closer match to @3dluvr's #103 anchor (113/163 at 195K with --dtype half --async-scheduling --cudagraph_capture_sizes [9]). Our config: --dtype bfloat16, default cudagraph capture sizes [1,2,4,8,16] (auto-clamped to single-stream). vLLM auto-estimated 195K won't fit at 0.85 mem-util (10.18 GiB KV needed vs 7.25 GiB available); 118K is the achievable ceiling at this mem-util / cudagraph budget. NIAH PASS at 88K (recalled "bronze octopus 17" from 88,527-token prompt, 11-tok completion in 127.1s wall, ~696 effective prefill TPS). KV pool 118,304 tokens, 1.00× concurrency. Code AL 5.13 (n=8 saturates well on code, less so on narr where AL is 3.7). n=8 vs n=4 trade: code +9% (130 → 142), narrative -3% (104 → 101) — n=8 dominates n=4 for code. Override MAX_MODEL_LEN=118304 MAX_NUM_SEQS=1 MTP_N=8. A/B finding 2026-05-08 — three tuning flags tested individually on this rig (with bench n=3 each, sync-baseline 101.16/141.90):
Flag	Narr Δ	Code Δ	Verdict
---	---:	---:	---
--dtype half (vs bfloat16)	-3%	-2%	Ampere has NO fp16 hardware accel on sm_86 — bfloat16 is canonical
--async-scheduling	~0%	~0%	within CV noise (CV 6.3% on narr)
cudagraph_capture_sizes [9]	-3%	+2.5%	single-shape capture optimizes n=8 hot batch (1 base + 8 spec = 9), hurts narrative-dynamic patterns
None close the -13% narr / -11% code gap to 3dluvr's anchor. Remaining gap likely rig-specific (3dluvr's EPYC 7J13 / different PCIe topology / 275W cap / etc) rather than tunable via flags. Cross-rig productionizable settings: stick with default --dtype bfloat16, default cudagraph, default scheduling. Custom override cudagraph_capture_sizes [9] worth it ONLY if you serve >95% n=8-MTP-single-stream code traffic (e.g. dedicated coding-agent endpoint) where the +2.5% code lift exceeds the -3% narrative loss.
dual.yml-shape forced TP=1	@apnar (1× RTX 5090 32 GB, air-cooled, 600 W)	bf16	32K	159.67 / 215.10 (decode 160.71 / 217.30)	27.5 GB	2026-05-07	First single-5090 Gemma 4 MTP data point. First non-OOM single-card Gemma 4 result on the matrix — the 32 GB Blackwell envelope clears the 24 GB Ampere boot OOM. CV 1.9%/1.8%, peak 426 W. +46% narr / +51% code over @noonghunna's 2× 3090 TP=2 baseline (109/142) — single-card 5090 beats dual-3090 on Gemma 4. Disc #67.
dual-dflash.yml-shape forced TP=1 (mem-util 0.96, max-model-len 12000)	@apnar (1× RTX 5090 32 GB, air-cooled, 600 W)	bf16	12K	150.40 / 261.06 (decode 151.16 / 264.62)	28.8 GB	2026-05-07	First single-5090 Gemma 4 DFlash data point. Trade vs MTP row above: ~6% narr loss, +21% code lift (215→261). 1st-warmup TTFT outlier (73 s) suggests cudagraph warmup taking longer on first request; subsequent warmups stable at <40 ms. CV 3.6%/2.8%, peak 440 W. Required mem-util 0.96 + max-model-len 12K to fit BF16 weights + DFlash N=5 drafter on 32 GB — DFlash drafter footprint pushes out ctx ceiling vs MTP's 32K. Disc #67.

Single-card (1× 3090) — llama.cpp (no vLLM)

Two single-3090 llama.cpp paths, measured 2026-05-27 (bench.sh, n=5; full writeup discussion #239). Both: gemma-4-31B-it Q4_K_S + q5_0/q4_1 KV; SWA-windowed KV scales context to ~131–200K. beellama is now the shipped single-card Gemma-4 default (⚠️ caveats — served via our unofficial multi-arch image, sm_89/120 unvalidated; see docs/UPSTREAM.md). The mainline-FA_ALL_QUANTS row is the no-fork reference (from-source build); re-point the default to it when llama.cpp#23398 Gemma-4 MTP merges.

Config	Rig	KV	Bench ctx	Narr / Code TPS	PP tok/s	DFlash accept	8-pack (think off / on)	Peak VRAM	Date	Notes
mainline llama.cpp + FA_ALL_QUANTS (no spec-dec)	@noonghunna (1× 3090)	q5_0/q4_1	65K	36.8 / 36.8	~277	—	109/150 / —	19.2 GB	2026-05-27	The no-fork supported path. Built from source with -DGGML_CUDA_FA_ALL_QUANTS=ON — the head_dim=512 quant-KV FA kernel the stock ggml-org/llama.cpp image omits (→ ~12 TPS otherwise). CV 0.1%, TTFT 0.3 s. ⚠️ DOES NOT REPRODUCE (2026-05-28): a re-bench of this exact config (same from-source binary, same q5_0/q4_1 KV, same 65K) measured 13.6 / 15.4 decode TPS — ~2.5× below this row. Ruled out as causes via on-rig diagnosis: launch flags (--reasoning auto --jinja A/B = no effect), throttle/thermal/power-cap (all inactive, 320W of a 370W cap, 69°C), contention (sole process), and memory-clock starvation (ran 9501 MHz ≈ stock max). The LACT +1000-mem/+200-core OC was off this session (would explain only ~15–20%, not 2.5×). 43–59% GPU util during decode points to an unexplained per-token stall. Number needs re-validation under a known/controlled rig state (OC on, fresh GPU) before it's trusted — do not propagate. Surfaced incidentally while validating benchlocal-cli #59.
beellama.cpp + DFlash ⭐ (single-card default)	@noonghunna (1× 3090)	q5_0/q4_1	100K	47 / 88	~855	~20–80% (workload)	109/150 / 114/150	21.8 GB	2026-05-27	Single-card Gemma-4 default (⚠️ caveats) — the only viable fast single-card path (vLLM FA-walled at head_dim=512, ik walls ~24K, stock llama.cpp ~12 TPS). DFlash external drafter — 1.28× narr / 2.40× code over mainline; TTFT ~0.1 s. TPS thinking-neutral (47/88 both modes). Thinking-ON nets +5 on the 8-pack (agentic gains; reasonmath −1 noise). Served via unofficial multi-arch image beellama-cpp:multiarch-b9459-07ac3ce (sm_86/89/120; sm_89/120 compiled-not-validated). CV 3–6%.

Gemma 4 12B (gemma4_unified — experimental)

New gemma4_unified arch (vLLM PR #44429 / llama.cpp #24077, merged 2026-06-03). Card advertises 256K; HF config trains to 131072 but the global layers' Proportional RoPE extrapolates cleanly past it — reachable on all three engines once the engine stops sizing the RoPE cache to max_position_embeddings. All rows 2026-06-04, @noonghunna, NIAH = needle-in-haystack exact recall. Quality 8-pack pending.

Single-card (1× RTX 3090) — 256K via --override-kv

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
llamacpp/gemma-12b-single-q8kxl ⭐	1× 3090	q8_0	256K	51.3 / 51.4	~1000	~16 GB	2026-06-04	Mainline ggml-org/llama.cpp:server-cuda + Unsloth Q8_K_XL. 256K via --override-kv gemma4.context_length=int:262144 (lifts llama.cpp's n_ctx_train cap; native p-RoPE extrapolates — NOT YaRN, gemma4 ignores rope-scaling). NIAH clean at 150K + 200K + ~246K. No spec-dec (gemma4-assistant MTP arch unmerged, llama.cpp #23398). Fastest single-card 256K path.
beellama/gemma-12b-single-q8kxl	1× 3090	q5_0/q4_1	256K	47.0 / 46.9	~990	~15 GB	2026-06-04	beellama v0.3.0 stable + Q8_K_XL turbo KV, same override-kv 256K (NIAH 200K + 246K ✓). KV-quant is TPS-neutral (q8_0 = 47.7 ≈ q5_0/q4_1); the ~9% gap vs mainline is engine, not KV. No spec-dec; no DFlash drafter for the 12B.
vllm/gemma-12b-single-int8-mtp	1× 3090	int8 AutoRound	256K	117 / 122.5	KV 310,328 tok · 1.18×	~20.7 GB	2026-06-04	Intel AutoRound INT8 (W8A16) + assistant external drafter on stock vllm/vllm-openai:gemma4-unified (no overlay). Full 256K fits ONE card (gemma4 KV ~32 KB/tok; drafter resident). NIAH exact-recall 140K/170K/200K/230K/241K. n-sweep (code-gen): n2 96.7 / n3 115.5 / n4 117.0 / n5 122.5 TPS, accept_len 2.62/3.33/3.67/4.06, vs ~50 undrafted. Default n=4 (SPEC_N=5 code-max). TTFT 65 ms. Quality 8-pack 105/150 (70%) think-OFF (toolcall 13 · instructfollow 14 · structoutput 14 · reasonmath 13 · dataextract 10 · bugfind 14 · hermesagent 11/20 · cli 16/40) — on par with the bf16 dual's 94/150 (INT8 ≈ bf16, no degradation; spread is agentic-pack variance). bf16 KV only. ⚠️ Production w/ caveats — ephemeral arch-preview image (pin a digest; → ✅ on a stable vLLM gemma4_unified release).

Dual-card (2× RTX 3090, TP=2) — vLLM gemma4-unified (stock; p-RoPE overlay dropped 2026-06-04, config fix)

Compose	Rig	KV	Max ctx	Narr / Code TPS	Concurrency @256K	Peak VRAM	Date	Notes
vllm/gemma-12b-dual-bf16-mtp ⭐	2× 3090 PCIe	bf16	256K	76.4 / 120.9	KV 426,382 tok · 1.63×	~22 GB/card	2026-06-04	Stock vllm/vllm-openai:gemma4-unified (no overlay — 256K from the upstream config fix vllm#39914, overlay dropped PR #309) + assistant MTP drafter (gemma-4-12B-it-assistant, n=4): +61% narr / +154% code vs undrafted (47.5/47.5). AL 2.4 narr / 3.7–4.0 code; accept ~36% / ~70%. The only 256K + MTP dual path. NIAH 150K+200K exact recall; TRITON_ATTN (head dims 256/512). Quality (rebench-full 2026-06-04): 8-pack 94/150 (63%) think-OFF · 85/150 (57%) think-ON; soak PASS (0 err · 0/100 silent-empty · 0 MiB growth · p50 115.7 TPS · 94.8% retention). ⚠️ Production w/ caveats — ephemeral arch-preview image (pin a digest; → ✅ on a stable vLLM gemma4_unified release).

---

DiffusionGemma 26B-A4B (dLLM — experimental)

Block-diffusion LLM (DiffusionGemmaForBlockDiffusion), not autoregressive — it emits blocks of tokens per step, so decode TPS reads much higher than AR models and is high-variance (entropy-dependent). Runs on Ampere via stock vllm/vllm-openai:v0.24.0 (DiffusionGemma arch native — PR #45163 merged 2026-06-12; bumped 2026-07-02 off the former :gemma branch digest) + our 3 bind-mounted Ampere/TP fixes (Marlin-K-pad ×2 + diffusion_gemma TP-vocab/dtype) — the Marlin-pad is what clears the sm_86 FP8 shared-mem tile wall (Padding FP8 MoE Marlin intermediate 352→384), still needed because #45295's dense marlin-pad (native in v0.24.0) doesn't cover the marlin_moe_wna16 MoE fp8 path. Slug vllm/diffusiongemma-dual (🧪, --force).

Config	Rig	KV	Max ctx	Narr / Code TPS	KV pool	Peak VRAM	Date	Notes
vllm/diffusiongemma-dual (TP=2, fp8) — maintainer	@noonghunna (2× 3090 PCIe, no NVLink)	bf16	262K	~177 / 180 (peak ~1100 low-entropy)	413K tok · 1.58×	~23 GB/card	2026-06-11	Reference numbers from the shipped compose status_note. dLLM block-diffusion; 8-pack 100/150 (5-pack 84%); NIAH→250K; eager-only; max_new_tokens 16384 (self-terminates ~1.2–1.8K words).
vllm/diffusiongemma-dual (TP=2, fp8) — cross-rig ⭐	@steamEngineer (2× 3090 NVLink NV4, X299 i9-10920X quad-channel, PCIe 3.0 ×16, Proxmox LXC, 350 W)	bf16	262K	294 / 370 (decode, n=5, CV 8.5% / 21%; wall 209 / 242)	413,470 tok · 1.58×	~23.1 / 22.9 GB/card	2026-06-14	First cross-rig confirmation of the shipped compose (#405) — same image digest 9c719fc… (vLLM 0.22.1rc1.dev357), our Marlin-pad mounts visible working in the boot log. Decode notably above the PCIe reference (294 vs ~177) — plausibly NVLink lifting the TP=2 all-reduce, though block-diffusion variance is high (CV up to 21%), so not a clean attribution. TTFT ~1.4 s narr / ~1.1 s. verify-stress 8/8 (NIAH usable to 184K, quality ceiling 214K — recall miss at 81%); verify-full 8/9 (the lone fail is the SSE-chunk check — a false-fail for block-diffusion, which emits whole blocks, not token-by-token; the text was correct). Soak not run (was blocked by the auto-detect bug, fixed in #414). Agentic: O(n)-like TTFT growth + 3/12 tool-call misses at depth. 8-pack not run.

MoE models (v0.7.3 — preview track)

First-pass numbers for the two MoE models onboarded in v0.7.3. Preview track = vllm-nightly-clean engine (no Genesis patches, no TQ3 KV, no MTP yet) — exercises the upstream MoE loader PR #42521 (qwen3_5_moe weight loading, merged 2026-05-14) without other overlays. Production track (Genesis-anchored, MTP, longer context) lands in v0.7.4 after Genesis v7.73.x re-anchors on a post-#42521 nightly. Update 2026-05-30: the dual path is now promoted to Production (vllm/qwen-35b-a3b-dual, 262K + vision, vLLM v0.22.0 stable) — Genesis proved unnecessary here (DeltaNet-hybrid KV is cheap, so 262K fits TP=2 with 7.8× concurrency) and built-in MTP is net-negative on vLLM at every depth tested. See the ⭐ PRODUCTION row below.

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	AL	Per-pos accept	Peak VRAM	Date	Notes
qwen3.6-35b-a3b/dual/preview.yml (TP=2) ⭐	@noonghunna (2× 3090 PCIe, 230 W cap)	fp8_e5m2	16K	182.68 / 177.45 (decode 186.98 / 186.90)	—	n/a (no drafter)	n/a	21.94 GB/card	2026-05-15	First v0.7.3 MoE preview row on the matrix. Engine vllm-nightly-clean (nightly bf610c2f, post-#42521). No Genesis, no MTP — exercises the upstream qwen3_5_moe loader cleanly. CV 0.3% / 0.9% (very stable). TTFT 126 ms. GPU 0 at 99% util / 292 W, GPU 1 at 85% / 244 W. Decode TPS basically identical narr vs code (187 / 187) — characteristic of MoE memory-bandwidth-bound decode (3 B active params per forward, weights fit cache). ~2× the Qwen 3.6 27B dense dual.yml baseline (89/118 wall) on the same hardware — MoE's active-params advantage on Ampere. Next steps: TQ3 KV + MTP (built-in head) after Genesis v7.73.x lands; longer ctx after upstream MoE expert dispatch overhead is measured. Compose: models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/preview.yml (since promoted → dual/autoround-int4/fp8.yml, see row below).
qwen3.6-35b-a3b/dual/fp8.yml (TP=2, vision, no-MTP) ⭐ PRODUCTION	@noonghunna (2× 3090 PCIe)	fp8_e5m2	262K	178.5 / 173.7 (decode 182.3 / 182.3)	—	n/a (no drafter)	n/a	22.1 GB/card	2026-05-30	Promoted preview → Production (slug vllm/qwen-35b-a3b-dual, engine vLLM v0.22.0 stable, no overlays). Full gate: max-ctx probe (262K fits, 7.81× concurrency, 2.05M-token KV pool — DeltaNet-hybrid KV is cheap, so the 16K preview cap was ~16× too conservative; kv-calc's MoE weight-anchor was wrong, real INT4 weights ≈10 GB/card not 20); verify-stress NIAH-clean to 240K (91%, no Cliff-2); soak-continuous PASS (0 growth / 0 err / 0 silent / 100% retention); quality --full det 79/90 (88%) · hermes 11/20 · cli 14/40 · aider-polyglot 13/30 (43%) (≥ the ik-llama apex-fit-q8q5 ref); live vision smoke (read a test image correctly @ 262K). CV <0.5%. MTP REJECTED at n=2 and n=3 (n2 ≈ −45% / n3 ≈ −51% wall TPS despite AL 2.4-3.4 — the MoE built-in-MTP draft forward pays inter-GPU TP=2 sync the acceptance can't amortize; ik-llama avoids this by running single-card). Served drafter-less; the built-in-MTP A/B compose (fp8-mtp.yml) has since been removed — net-negative on this MoE at vLLM TP=2, finding recorded in this row. Compose: models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/fp8.yml.
qwen3.6-35b-a3b/dual/preview-mtp.yml (TP=2, MTP n=3, built-in head) ⚠️	@noonghunna (2× 3090 PCIe, 230 W cap)	fp8_e5m2	16K	90.36 / 115.33 (decode 91.48 / 119.65)	—	3.44 (narr)	0.927 / 0.810 / 0.698	22.72 GB/card	2026-05-15	MTP MAKES THINGS SLOWER on Qwen MoE preview path. A/B vs preview.yml above (same config, same nightly, same hardware): −51% narr / −35% code wall TPS despite 81.2% avg draft acceptance and AL 3.44. Per-position accept (0.927 / 0.810 / 0.698) is healthy — the draft head works correctly. Surfacing cause: asymmetric GPU util (GPU 0 at 39% / 233 W vs GPU 1 at 79% / 191 W; non-MTP run had both at 85-99%) indicates the draft forward pass on MoE imposes inter-GPU sync overhead the acceptance gain can't amortize. Vendor warning at boot: max_num_scheduled_tokens=4096 ... suboptimal performance ... increase max_num_batched_tokens to accommodate the additional draft token slots. CV 5.4% / 2.0% (less stable than no-MTP). VRAM +0.78 GB/card vs no-MTP (MTP head workspace). Practical implication: for v0.7.3, route Qwen 35B-A3B users to preview.yml (no MTP) as the default — MTP gives worse latency despite high acceptance. Re-test after (a) Genesis v7.73.x re-anchors on a post-#42521 nightly (Cliff 2 mitigations may unblock the underlying scheduler overhead), (b) max_num_batched_tokens raised to ~12K to satisfy the vLLM warning, or (c) MTP n=2 to see if smaller spec depth changes the calculus. Compose: models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/preview-mtp.yml.
qwen3.6-35b-a3b/ik-llama/single/mudler-apex-compact/mtp.yml (TP=1, q4_0 KV, MTP n=5, --fit)	@noonghunna (1× 3090, 370 W cap)	q4_0 / q4_0	200K	96.47 / 144.01 (decode 98.54 / 150.98, n=5, CV 5.4% / 4.5%)	~3000 @10K → ~1283 @180K	(ik-llama MTP active, AL not in bench log)	(not reported)	20.46 GB	2026-05-28	First-ever real run of the existing EVAL-only compose (APEX weights weren't on disk before today). Pre-fit-mtp.yml baseline, kept as the q4-KV / max-headroom variant. verify-full 8/8, verify-stress 8/8 incl. 180K NIAH at 91% of n_ctx. Confirms ik built-in MTP on the 35B-A3B MoE does NOT pay the vLLM −51% penalty (compare row above) — MoE×MTP is a vLLM-scheduler-specific problem, not architectural. TTFT 205 ms. Compose: models/qwen3.6-35b-a3b/ik-llama/compose/single/mudler-apex-compact/mtp.yml.
qwen3.6-35b-a3b/ik-llama/single/mudler-apex-compact/fit-mtp.yml (TP=1, q8/q5 KV, MTP n=4) ⭐⭐	@noonghunna (1× 3090, 370 W cap)	q8_0 (K) / q5_0 (V) — asymmetric K-high/V-low	196K	103.25 / 149.12 (decode 105.63 / 156.80, n=5, CV 3.0% / 1.5%)	~3000 @10K → ~1283 @180K	(ik built-in MTP active, AL not in bench log)	(not reported)	21.09 GB	2026-05-28	Captures @laurimyllari's --fit + asymmetric KV config from discussion #241. A/B vs mtp.yml row above (same engine, same weights, same rig + 370 W): +7% narr / +4% code wall TPS at tighter CV (3.0/1.5% vs 5.4/4.5%) for +0.6 GB VRAM — the Anbeeld K-high/V-low pattern materialised. verify-full 8/8, verify-stress 8/8 incl. 180K NIAH (91% of 196K, 26.4 GB margin, 1283 tok/s prefill at ceiling). Adds --no-mmap + --cache-ram 4096 per his fit-mtp.yml. Deterministic quality 76/90 = 84% on PR #38-fixed verifiers (toolcall 14/15, instructfollow 15/15, structoutput 14/15, dataextract 11/15, reasonmath 12/15, bugfind 10/15). Sandbox-pack quality measured post-fix (benchlocal-cli #42 hermes thinking-on deterministic sampler + #43 per-pack timeout defaults + #44 aider git-checkout guard + club-3090 #245 wrapper-omit): hermesagent-20 11/20 (55%), cli-40 12/40 (30%), aider-polyglot-30 12/30 (40%). cli-40 retains 7 "timeout" failures but those are sandbox-internal agent-give-up (max latency 19 s, well under the 300 s budget) — distinct from wall-clock, tracked in a follow-up benchlocal-cli issue. soak-continuous PASS (0 errors, 0/25 silent_empty, 0 VRAM growth, 100% retention, p50 decode 223 TPS). vs @laurimyllari on 4090 (205/254 TPS, 105-107/150 8-pack pre-PR #38): 3090 trails on TPS as expected (newer silicon + power), deterministic quality is on par. MoE × MTP no-penalty confirmed in ik-llama (cf. preview-mtp.yml). Compose: models/qwen3.6-35b-a3b/ik-llama/compose/single/mudler-apex-compact/fit-mtp.yml. Registry tag: ik-llama/apex-fit-q8q5.
qwen3.6-35b-a3b/ik-llama/single/byteshape-iq4xs/mtp.yml (TP=1, q4_0 KV, MTP n=2, --fit) ⚠️	@noonghunna (1× 3090, 370 W)	q4_0 / q4_0	262K (--fit auto)	112.90 / 128.61 (decode 115.60 / 137.07, n=5, CV 1.9% / 1.8%)	~1859 @95K → ~1050 @240K	(ik built-in MTP n=2 active)	110/150 (73%, off)	22.5 GB	2026-06-02	Community intake from PR #293 (@Rhonstin) — byteshape IQ4_XS 4.19bpw MoE GGUF (embedded MTP head), single-card 35B-A3B. First-party validated on 1× 3090: verify-full all-pass, verify-stress 8/8 incl. NIAH→240K (91% of 262K, no Cliff; KV self 1584 MiB @ 262K — MoE KV is cheap), soak-continuous PASS (0 err, 0 VRAM growth, 0/25 silent-empty, 100% retention, p50 223 TPS). 8-pack 110/150 (think-off): toolcall 15/15 · instructfollow 14/15 · structoutput 14/15 · dataextract 13/15 · reasonmath 13/15 · bugfind 13/15 · hermesagent-20 11/20 · cli-40 17/40 — reproduces @Rhonstin's 111/150 within noise. q4/q4 KV buys full 262K (vs the apex sibling's q8/q5 @196K). Intake fixes vs #293: image cu13-server, port 8058. Compose: models/qwen3.6-35b-a3b/ik-llama/compose/single/byteshape-iq4xs/mtp.yml. Registry tag: ik-llama/byteshape-iq4xs-mtp.
Qwen3.6-35B-A3B-Cerebellum-v3 (1× 3090, mainline llama.cpp) — author data point, NOT a catalog compose	@deucebucket (1× 3090, Bazzite bare metal, host llama.cpp b9603, 420 W limit / ~300–316 W draw, desktop compositor on same GPU)	q8_0 / q8_0	131K (196K boots @ ~15.9 GB, fill-probed ~119K — NIAH ladder not run)	147.9 / 146.2 (decode 150.5 / 149.9; report.sh --full re-run 141.8 / 141.2)	—	n/a (no drafter)	n/a	~15.1 GB (15,115 MiB @ 131K)	2026-06-12	Author-reported; not club-validated or cataloged — declined as a catalog compose (PR #393): headline value is the lean ~15 GB footprint (fits a 16 GB card, where the ik-llama siblings at 21–22.5 GB don't), but we have no 16 GB card to validate/support that tier, and on 24 GB it's context/quality-dominated by apex-fit-q8q5 / byteshape-iq4xs. Quant: ablation-informed per-tensor mixed precision (each tensor's Q2_K sensitivity measured, precision allocated under a size budget) → 11.96 GB GGUF on a stock ggml-org image. Det-5-pack 63/75 (toolcall 13 · instructfollow 15 · structoutput 14 · dataextract 9 · reasonmath 12; benchlocal --medium, sandboxed packs not run — no docker). TPS at -np 4 provenance (single-card convention is -np 1). Drove the README llama.cpp ❌→✅ correction (559a3fe). Weights + per-question evidence: deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF.
qwen3.6-35b-a3b/llama-cpp/dual/morikomorizz-q6kp/mtp.yml (TP=2, q8_0 KV, MTP n=3) 🧪	@noonghunna (2× 3090 PCIe)	q8_0 / q8_0	262K	113.4 / ~150 (decode, n=3 shipped; wall 111.6 / ~144, CV<1% narr)	~1820 @240K	(MTP draft-mtp n=3; acc narr 47% / code 83%)	n/a	19.9 / 18.4 GB/card	2026-06-14	Uncensored "HauhauCS-Aggressive" 35B-A3B fine-tune — morikomorizz Q6_K_P GGUF with embedded nextn MTP head, on mainline llama.cpp b9570, dual-card layer-split -ts 0.55,0.45 (rebalances the MTP draft card; even 1,1 skews ~3 GB at 262K). MTP loads clean (speculative decoding context initialized) — the prior HauhauCS-MTP ret=-3 was an ik-llama/older-build issue, not mainline. n-sweep @262K (canonical bench.sh, same fresh container): n=3 113.4/~150 vs n=1 125.2/136.4 → n=3 = −9% prose / +10% code (code-leaning default by maintainer request; n=1 is prose-best). ⚠️ An earlier rebench read 147/162 @ n=1 that did NOT reproduce on fresh boot (125 reproducible — GPU-state TPS variance; the 147 is not trusted). verify-stress 8/8 incl. NIAH→240K (91% of 262K), soak fresh 20×5 PASS (0 MiB growth, 0/100 silent-empty, p50 162.4 TPS, 99.6% retention). 8-pack think-OFF 103/150 (69%) · think-ON 105/150 (70%) (wash; reasoning-on costs heavy latency — hermes hit 300s — for +2): toolcall 14/14 · instructfollow 12/14 · structoutput 14/14 · dataextract 11/8 · reasonmath 11/12 · bugfind 13/13 · hermesagent 11/11 (of 20) · cli 17/19 (of 40). Mid-band for the class (byteshape 110); uncensoring buys compliance, not capability. REASONING=on default (vs stack thinking-off). Community GGUF, digest-UNPINNED + uncensored → 🧪. Compose: models/qwen3.6-35b-a3b/llama-cpp/compose/dual/morikomorizz-q6kp/mtp.yml. Registry tag: llamacpp/hauhaucs-35ba3b-dual.
gemma-4-26b-a4b/dual/awq.yml (TP=2) ⭐	@noonghunna (2× 3090 PCIe, 230 W cap)	bf16	32K	138.88 / 138.67 (decode 139.92 / 139.98)	—	n/a (no drafter)	n/a	23.45 GB/card	2026-05-15	First v0.7.3 Gemma MoE production-track row. Engine vllm-nightly-clean (nightly bf610c2f) + vendored vLLM PR #40886 overlay (compressed-tensors AWQ MoE key remapping — applied at boot via anchor-based Python patcher in models/gemma-4-26b-a4b/vllm/patches/vllm-pr40886-awq-moe-keys/install.sh). Weights: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit (17 GB). CV 0.2% / 0.0% — extraordinarily stable, characteristic of MoE memory-bandwidth-bound decode. TTFT 53 ms. GPU 0 at 98% util / 360 W, GPU 1 at 59% / 299 W. Identical narr/code wall TPS (139 / 139) — same MoE signature as Qwen 3.6 35B-A3B preview above (per-token weight reads dominate, prompt content distribution doesn't matter). ~76% of Qwen 35B-A3B preview's narr TPS — consistent with 4 B vs 3 B active params per forward. Compose: models/gemma-4-26b-a4b/vllm/compose/dual/awq/bf16.yml.
gemma-4-26b-a4b/dual/awq-mtp.yml (TP=2, MTP n=4) ⭐⭐	@noonghunna (2× 3090 PCIe, 230 W cap)	bf16	32K	155.05 / 207.02 (decode 156.68 / 210.61)	—	3.04 narr / 3.79 code	0.77 / 0.55 / 0.40 / 0.29 (narr); 50.9% avg (narr), 73.5% avg (code)	23.50 GB/card	2026-05-15	MTP boosts Gemma MoE: +12% narr / +49% code over awq.yml. Engine vllm-nightly-clean + PR #40886 overlay + external google/gemma-4-26B-A4B-it-assistant drafter (832 MB BF16). Both GPUs symmetric at 98% util — the external-drafter path doesn't pay the inter-GPU sync penalty that the Qwen 35B-A3B built-in MTP head pays (see preview-mtp.yml row above where MTP made things 50% slower). Mechanism: external drafter is a small dense model (Gemma4AssistantForCausalLM, ~0.5 B params) — the draft forward bypasses MoE expert routing entirely. The 32-49% code lift confirms MTP is structurally compatible with Gemma's hybrid-SWA attention on the AWQ compressed-tensors path. Recommended default for Gemma 26B-A4B going forward; awq.yml stays as the no-drafter A/B reference. Compose: models/gemma-4-26b-a4b/vllm/compose/dual/awq/mtp.yml.
gemma-4-26b-a4b/single/awq/int8.yml (TP=1, INT8-PTH KV, MTP n=4) ⭐⭐ ⚠️ Production-w/-caveats	@noonghunna (1× 3090, 370 W)	int8_per_token_head	176K	168.0 / 217.4 (decode 169.2 / 219.9, CV 1.9% / 2.0%)	~4416 @10K → 976 @161K	3.06–3.79	0.77–0.85 / 0.58–0.74 / 0.42–0.65 / 0.29–0.54	23.6 GB	2026-06-06	First single-3090 Gemma-4-26B-A4B — INT8-PTH KV (vendored PR #40391 on stock v0.22.0, engine vllm-gemma-stable) lifts the single card from the prior bf16/16K boot-OOM to 176K. Slug vllm/gemma-26ba4b-single; cyankiwi AWQ-4bit (Marlin WNA16 MoE) + external MTP drafter (gemma-26b-it-assistant, 832 MB BF16, n=4). Gate PASS (rebench gemma-26ba4b-int8r): verify-full ✓, verify-stress NIAH-clean to 161K (91% of 176K, all boundary checks), soak 20×5 PASS 0-growth. 8-pack 109/150 think-ON (98/150 think-off) — toolcall 13/15 · instructfollow 14/15 · structoutput 14/15 · dataextract 9→11/15 · reasonmath 13→14/15 · bugfind 13/15 · hermesagent 10/20 · cli-40 12→21/40 — on par with gemma-4-31B's think-ON 107/150. Caveats: needs the #40391 overlay; 262K reachable only WITHOUT the MTP drafter (drafter weights + cudagraph capture cost ~86K tok — mem_util 0.94, 0.96 OOMs the capture tail); think-OFF agentic/extraction softer (cli-40 30% / DataExtract 60%, recover to 52%/73% with thinking). Compose: models/gemma-4-26b-a4b/vllm/compose/single/awq/int8.yml.
gemma-4-26b-a4b/dual/docker-compose.yml (TP=2, Intel AutoRound INT4)	@noonghunna (2× 3090 PCIe)	—	—	boot fail (SM86)	—	—	—	—	2026-05-15	Ampere-blocked on Intel AutoRound INT4. moe_intermediate_size=704 is not a multiple of group_size=128 (5.5×); Marlin K-dim alignment fails. SM86 has no WNA16 kernel for unaligned K-dim — only SM90+ Cutlass W4A8 / Machete handle arbitrary shapes. Same failure on TP=1 (no split) and TP=2 (split to 352 per rank). Both Intel quant variants (int4-mixed-AutoRound and int4-AutoRound) hit the same error. AWQ is the Ampere path (row above). AutoRound compose preserved here as a documented-blocker for SM90+ rigs (RTX 5090 / Pro 6000 should boot it).

---

Ornith-1.0 (DeepReinforce agentic-coding fine-tunes — 🧪 experimental)

DeepReinforce agentic-coding RL fine-tunes of the Qwen3-Next family (qwen35 9B dense-hybrid · qwen35moe 35B-A3B). Both land in-band with the base Qwen on general capability — the fine-tune's value, where any, is coding (the 35B edges the base on aider; the 9B is a footprint play). ik_llama GGUF, drafter-free ngram, full 262K.

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
ik-llama/ornith9b-single	@noonghunna (1× 3090)	q8_0	262K	100.6 / 100.6 (decode 102.0 / 102.7)	—	13.4 GB	2026-06-25	Qwen3-Next dense-FFN HYBRID (arch qwen35: 8 full-attn + 24 GDN, NON-MoE) — 262K KV only 4.25 GB. No MTP → ngram (~0.59 accept, works despite DeltaNet). verify-stress 8/8, soak PASS. 8-pack 91/150 off / 95 on; cli-40 7/40 is partly a <solution>-sign-off artifact (→13 hardened). NICHE ONLY — gemma-4-12b beats it on quality (105) + speed (117/122). The lean 13.4 GB footprint is the only reason to pick it. Slug shipped #477.
ik-llama/ornith35b-dual	@noonghunna (2× 3090)	q8_0	262K	106.5 / 103.6 (decode 108.8 / 108.6)	—	~22 GB/card	2026-06-26	qwen35moe MoE hybrid (40L, 256 experts / 8 active ~3B, NO MTP). 262K KV ~2.7 GB. ngram opt-in (Q8 dual tight; needs n_max=32 + balanced -ts). verify-stress 8/8, soak PASS. 8-pack 105/150 off == on — TIES base qwen3.6-35b-a3b (byteshape 110, within noise) but EDGES it on aider-polyglot-30: 15/30 (off == on) vs the base's 12–13 + perfect bugfind 15/15. Coding-leaning 35B-A3B; the base stays the general pick. Slug shipped #479.

---

Agents-A1 (InternScience 35B agentic MoE — ⚠️ production w/ caveats)

InternScience's own 35B agentic MoE (Qwen3-Next MoE architecture; card declares its OWN base — not a Qwen fine-tune slug), served from the official FP8-dynamic compressed-tensors checkpoint. The agentic thinking-ON specialist: ties the base qwen3.6-35b-a3b on general capability, beats it on cli-40. First model onboarded end-to-end through the Bring & Validate lane (T2 producer-zero).

Compose	Rig	KV	Max ctx	Narr / Code TPS	PP tok/s	Peak VRAM	Date	Notes
vllm/agents-a1-dual	@noonghunna (2× 3090 PCIe, 370/420 W)	fp8_e5m2	262K	150.9 / 149.6 (decode 153.9 / 154.0, n=5, CV 0.1%, TTFT 130/131 ms)	7.9K→4.6K t/s across 94K→240K (NIAH-rung diagnostic)	~22.0 GB/card	2026-07-03	Qwen3-Next MoE (geometry == 35B-A3B; ~3B active), NO MTP head (safetensors-verified) → drafter-free. On sm_86 vLLM serves Marlin FP8-MoE weight-only (activation quant inert — sm_89+ runs the real checkpoint). verify-stress 8/8 — staggered NIAH exact-recall to 240K (91%), VRAM Δ0 across the ladder; soak PASS (0 growth, 0/100 silent-empty, 99.8% retention). 8-pack --full: OFF 105/150 · ON 110/150 (post benchlocal #79+#81 harness; fresh sandbox images): toolcall 15/15 OFF · IF 15/15 ON · cli-40 thinking-ON 23/40 — the stack's highest (base: 17/40) · hermes 12/20 OFF but regresses to 9/20 thinking-ON (verified model behavior). vs base 35B-A3B: general TIES (ON 110=110), decode −13% (154 vs 178) — reach for A1 on tool/CLI-agent work with thinking ENABLED. Vision retained. Dual-only (36 GB weights).

---

Quality benches — Aider Polyglot 30

Pass rate on a curated 30-exercise subset of aider-polyglot-benchmark (5 per language across cpp/go/java/javascript/python/rust, mix of easy/medium/hard). Tests edit-format reliability AND algorithmic correctness — does the model emit diffs aider can apply, AND do the resulting tests pass.

Run via benchlocal-cli aider-polyglot-30 pack. Different from the TPS rows above — this is a quality / agentic-coding signal, not a throughput measurement.

Model	Compose	Rig	Pass / Total	%	Wall (real)	Wall (sum-dur)	Tokens (P+C)	Date	Notes
Qwen 3.6 27B (AutoRound INT4)	dual.yml (TP=2)	@noonghunna (2× 3090 PCIe, 230W cap)	20 / 30	66.7%	19.0 min	34.0 min	436K + 111K = 547K	2026-05-10	enable_thinking=false (server-side --default-chat-template-kwargs '{"enable_thinking": false}' + per-request extra_body belt). With thinking ON: 0/30 (1500s timeout exceeded before any exercise completed — Qwen burns the token budget on hidden CoT). Per-language: cpp 3/5 · go 4/5 · java 4/5 · js 4/5 · python 2/5 · rust 3/5. threads=2.
Gemma 4 31B (Intel AutoRound INT4)	dual.yml (TP=2)	@noonghunna (2× 3090 PCIe, 230W cap)	17 / 30	56.7%	19.2 min	19.2 min	380K + 72K = 452K	2026-05-10	Default thinking off (Gemma 4's chat template requires explicit enable_thinking=true to enable). Per-language: cpp 2/5 · go 4/5 · java 1/5 · js 4/5 · python 3/5 · rust 3/5. threads=2.

Notable:

• Qwen 3.6 27B beats Gemma 4 31B by +10 pp despite 4 GB fewer parameters. Java is the biggest swing (4/5 vs 1/5 — affine-cipher specifically tripped Gemma).
• Gemma is meaningfully faster wall-clock (sum-of-exercise-durations 19 vs 34 min), suggesting Qwen produces longer per-turn answers but they convert to passes more reliably.
• Qwen with thinking ON is unusable for this kind of bench on club-3090 hardware: hits the 1500s subprocess timeout cap (now bumped to 2700s) before completing any exercises — the hidden CoT eats the per-exercise token budget. Set enable_thinking=false for any agentic / multi-turn workload.

To re-run cross-rig: bash scripts/quality-test.sh --pack aider-polyglot-30 --enable-sandboxed-packs against your endpoint. See docs/QUALITY_TEST.md for the harness setup.

---

Head-to-head — Qwen 3.6 27B vs Gemma 4 31B on dual 3090 (TP=2, MTP)

Single-rig comparison run 2026-05-10 on @noonghunna's 2× 3090 PCIe rig (230 W cap each). Both models served via dual.yml compose with MTP speculative decode and enable_thinking=false. Aim: agentic-workload pick guidance.

Config & pin delta — what differs between the two runs

	Qwen 3.6 27B	Gemma 4 31B
Weights quant	Lorbus AutoRound INT4 (mtp.fc preserved BF16)	Intel AutoRound INT4
MTP type	Built-in head, method: mtp n=3	External 0.5B BF16 drafter (gemma-4-31b-it-assistant) via vllm#41745, n=4
KV cache	fp8_e5m2	BF16
Max ctx	262144 (262K)	32768 (32K — BF16 KV ceiling on 2× 24 GB; use gemma-int8-mtp for 262K)
--gpu-memory-utilization	0.92	0.92
TP	2	2
vLLM nightly	01d4d1ad (2026-05-04)	1acd67a7 (2026-05-08) — newer, post Gemma 4 MTP merge
Genesis pin	v7.64 (loaded via setup.sh)	n/a (Gemma 4 doesn't use Genesis patches)
thinking	OFF (server-side --default-chat-template-kwargs)	OFF (Gemma 4 default; needs explicit enable_thinking=true to enable)

TPS (canonical bench.sh: 800-word essay narrative + quicksort C++ code)

Bench	Qwen 3.6 27B	Gemma 4 31B	Gemma vs Qwen
Narrative decode TPS	68.83	108.46	+57%
Code decode TPS	87.54	139.88	+60%
TTFT narrative	151 ms	70 ms	~2× faster
TTFT code	125 ms	62 ms	~2× faster
VRAM/card	23.7 GiB	22.5 GiB	−0.8 GiB
MTP mean accept length	3.30–3.56	3.05–3.96 (warming)	similar
MTP avg accept rate	76.5–85.2%	51.2–73.9% (warming)	Qwen slightly higher when warm
CV (narr / code)	3.9% / 4.1%	1.4% / 1.0%	Gemma noticeably more stable
GPU power (per card)	308 W / 263 W	351 W / 296 W	Gemma uses ~13% more power

Quality — 8-pack quality-test.sh --full (150 scenarios)

Same endpoint, same thinking-off default. Note — BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1 env var required for hermesagent-20 to reach host vLLM from the sandbox container (auto-set by scripts/quality-test.sh for localhost URLs since 83bf73d).

Pack	Qwen 3.6 27B	Gemma 4 31B	Δ (Gemma − Qwen)	Workload type
toolcall-15	10/15 (67%)	9/15 (60%)	−7 pp	Multi-tool call sequencing
instructfollow-15	13/15 (87%)	13/15 (87%)	tied	Format-constraint following
structoutput-15	13/15 (87%)	13/15 (87%)	tied	JSON / CSV schema emission
dataextract-15	15/15 (100%)	15/15 (100%)	tied	Information extraction
reasonmath-15	6/15 (40%)	6/15 (40%)	tied	Math (thinking-off cost = identical)
bugfind-15	12/15 (80%)	14/15 (93%)	+13 pp	Code debugging
hermesagent-20	10/20 (50%)	12/20 (60%)	+10 pp	Multi-turn agentic (Hermes harness, v0.7.4 grader)
cli-40	21/40 (52%)	20/40 (50%)	−2 pp	CLI task completion
TOTAL	100/150 (67%)	102/150 (68%)	+1 pp

Quality — Aider Polyglot 30 (per-language breakdown)

Cross-reference Quality benches — Aider Polyglot 30 above. Same dual.yml configs, same compose, single-shot.

Language	Qwen 3.6 27B	Gemma 4 31B	Δ
C++	3/5 (60%)	2/5 (40%)	−1
Go	4/5 (80%)	4/5 (80%)	tied
Java	4/5 (80%)	1/5 (20%)	−3 (Gemma fell apart on affine-cipher)
JavaScript	4/5 (80%)	4/5 (80%)	tied
Python	2/5 (40%)	3/5 (60%)	+1
Rust	3/5 (60%)	3/5 (60%)	tied
Total	20/30 (66.7%)	17/30 (56.7%)	−10 pp

Matched-config rebench (later same day) — most of the "Gemma +60% TPS" gap was config drift

After the first run, the two legs differed on (a) vLLM nightly, (b) KV dtype, (c) max context, (d) MTP n. Re-ran both with everything matched (vLLM 1acd67a7 · int8_per_token_head KV · 262K ctx · MTP n=4 · TP=2 · mem-util 0.92 · max-num-seqs=2 · AutoRound INT4 W4A16 group_size 128). Compose files added: models/qwen3.6-27b/vllm/compose/dual/autoround-int4/int8.yml (new, this work) + models/gemma-4-31b/vllm/compose/dual/autoround-int4/int8.yml (existing, SPEC_N_MAX parameterized).

Notable side-finding: Qwen3-Next + INT8 PTH KV (vllm#40391) works end-to-end on our stack — first validation. Real chat completions, MTP acceptance ~75%, +27%/+33% TPS vs Qwen's prior fp8 KV row above (most of which is the newer vLLM nightly + INT8 PTH KV combo, not architectural). Qwen int8.yml is now a candidate shipping path.

Matched-config performance (2026-05-10 PM)

Metric	Qwen 3.6 27B int8.yml	Gemma 4 31B int8.yml	Gemma vs Qwen
Narrative decode TPS	88.05	98.12	+11%
Code decode TPS	120.17	129.34	+8%
TTFT narrative	152 ms	79 ms	−48%
TTFT code	137 ms	79 ms	−42%
CV narrative / code	1.6% / 5.3%	2.1% / 0.6%	both clean
MTP avg accept (warm)	75.3%	70.6%	Qwen +5 pp
MTP per-pos rates (n=4)	0.94 / 0.83 / 0.69 / 0.55	0.89 / 0.75 / 0.65 / 0.55	Qwen earlier-pos better

Concurrency + VRAM (matched config)

Metric	Qwen	Gemma	Δ
VRAM used / card	21.4 GiB	22.5 GiB	+1.0 GiB Gemma
Available KV cache / card	11.10 GiB	10.82 GiB	-0.28 GiB Gemma
GPU KV cache size (tokens)	605,495	466,892	+30% Qwen
max_num_seqs (configured)	2	2	tied
Max concurrency @ 262K/req	2.31×	1.78×	+30% Qwen
Practical concurrency @ 100K/req	~6.0×	~4.7×	+28% Qwen
Practical concurrency @ 32K/req	~18.9×	~14.6×	+30% Qwen

Matched-config headline

• Per-stream speed: Gemma wins ~10% decode TPS + ~45% TTFT — that's the model-architecture-attributable gap. Dense attention prefills faster than DeltaNet hybrid.
• Throughput at scale: Qwen wins ~30% more KV pool — supports more concurrent streams at the same per-stream context budget. Lighter weights + smaller per-token KV from the hybrid attention.
• Workload pick: Single-agent / chat → Gemma. Multi-tenant fleet → Qwen.
• Original mismatched table below preserved for the "shipping defaults" comparison. The +60% gap from this morning was nearly all vLLM-nightly + KV-class drift, not model intrinsics.

Headline takeaways (original mismatched-config bench, kept for reference)

• TPS: Gemma 4 31B is ~60% faster across both narrative and code, plus ~2× faster TTFT and 0.8 GiB less VRAM per card. For throughput-bound workloads at ≤32K ctx, Gemma 4 wins decisively at this config. For long-context (>32K), Qwen 3.6 27B's 262K ceiling + fp8 KV is the only option in this matchup (use gemma-int8-mtp for Gemma at 262K — separate compose, not benched here).
• Quality aggregate: Effectively tied — 180-scenario combined total: Qwen 120/180 (66.7%), Gemma 119/180 (66.1%). Within noise. Neither model is generically "better"; the choice is workload-shaped.
• Quality by workload: Gemma wins agentic + bug-fix (bugfind +13 pp, hermesagent +10 pp). Qwen wins polyglot code editing (aider-polyglot +10 pp, driven mostly by Java). Tool-call ordering favors Qwen (toolcall +7 pp). Five packs are dead-even.
• reasonmath 6/15 (40%) on BOTH — the thinking-off cost is identical and model-independent. Math problems with strict-format expectations need thinking-on (and the aider-polyglot-30 row above shows what Qwen-with-thinking does to wall time — 0/30 hit the 1500s subprocess cap, now 2700s).
• Pin/config delta: Gemma's vLLM nightly is 4 days newer (post Gemma 4 MTP merge), KV is BF16 vs Qwen's fp8_e5m2, and Gemma uses an external 0.5B drafter vs Qwen's built-in MTP head. Same dual.yml topology, same --gpu-memory-utilization 0.92, same TP=2.

How to reproduce on your rig

# Qwen leg
gpu-mode 27b                                                    # bring up at :8010
RUNS=3 WARMUPS=1 bash scripts/bench.sh                          # TPS
bash scripts/quality-test.sh --full                             # 8-pack quality
benchlocal-cli run --pack aider-polyglot-30 \
  --endpoint http://localhost:8010 --model qwen3.6-27b

# Gemma leg (cycle GPU)
gpu-mode gemma                                                  # bring up at :8030
URL=http://localhost:8030 MODEL=gemma-4-31b \
  RUNS=3 WARMUPS=1 bash scripts/bench.sh
URL=http://localhost:8030 MODEL=gemma-4-31b \
  bash scripts/quality-test.sh --full
benchlocal-cli run --pack aider-polyglot-30 \
  --endpoint http://localhost:8030 --model gemma-4-31b

End-to-end takes ~3 hours on dual 3090. quality-test.sh auto-detects the endpoint and served-model-id, so URL/MODEL overrides are only needed when running against non-default ports.