llama.cpp — quick recipe + gotchas

If you want a lighter-weight setup, run on non-NVIDIA hardware, or just prefer llama.cpp's ergonomics, here's how to run Qwen3.6-27B on it. Note: mainline llama.cpp's Qwen3-Next support is still landing (4 PRs open). The fastest path today is via Luce z-lab's DFlash fork, which adds a custom DFlash spec-decode draft model.

TL;DR

✅ Runs on this stack (with caveats)
✅ Vision via mmproj model
✅ Fastest cold start of any engine (~30 sec)
✅ Smallest footprint (single binary, ~50 MB)
✅ Best path for 262K context on a single 3090 — see "Going to 262K" recipe below
⚠️ Server feature parity behind vLLM (no auto-tool-choice in upstream server binary; need a wrapper)
⚠️ Concurrent serving is single-threaded (forks per request) → sluggish UX under load
✅ MTP spec-decode merged on mainline (PR #22673, 2026-05-16). Luce DFlash fork also available for N=5 code workloads.

Why pick llama.cpp over vLLM on a single 3090?

The honest answer: disk + VRAM size.

Quant + format	Disk size	VRAM at idle	Headroom for KV at 262K
Lorbus int4-AutoRound (vLLM)	~18-19 GB	~19-20 GB	~3-4 GB (can't fit f16 KV at 262K — needs TQ3 KV compression on vLLM)
GGUF Q4_K_M (llama.cpp)	~16.8 GB	~17-18 GB	~6 GB
GGUF UD-Q3_K_XL (llama.cpp)	~14.5 GB	~15-16 GB	~8-9 GB
GGUF Q3_K_M (llama.cpp)	~13.6 GB	~14-15 GB	~9-10 GB

The ~4 GB savings going from AutoRound (vLLM) to UD-Q3_K_XL (llama.cpp) translates directly into more KV cache room. That's the difference between "fits 262K with quantized KV" (llama.cpp) and "fits 48K with quantized KV" (vLLM single-card default).

Quality cost of going from int4-AutoRound to UD-Q3_K_XL is small on Qwen3.6-27B (the model is quantization-friendly), but real on harder reasoning benchmarks. Trade-off is yours to weigh.

Why llama.cpp doesn't hit the prefill cliffs vLLM does

Both engines run the same model. Both do chunked prefill. Pre-2026-04-30 PM, vLLM at 192K + TQ3 + vision OOM'd on 25K-token tool messages (Cliff 1) while llama.cpp at 262K + q4_0 KV processed them cleanly. We've since closed Cliff 1 on vLLM via the PN12 anchor sidecar (which actually pools FFN intermediates instead of fresh-allocating per layer). The structural reason llama.cpp doesn't fire either of the two cliffs in the first place is still worth understanding — and it's why Cliff 2 (single-prompt >50–60K) still fires on vLLM single-card and doesn't on llama.cpp. Three pieces:

1. Different attention library entirely. vLLM links Dao-AILab FlashAttention 2 (_vllm_fa2_C.varlen_fwd). llama.cpp uses ggml-cuda kernels — fattn-mma-f16.cu, fattn-tile-f16.cu, fattn-vec-f16.cu. There's no max_seqlen parameter to leak. ggml's attention forward dispatches per batch with output buffers sized by current prompt length; FA2's varlen kernel allocates softmax_lse as [num_seqs, num_heads, max_seqlen] — sized by an upper bound that vLLM passes through cudagraph-capture metadata as max_model_len. So a 25K-token tool prefill at vLLM's max-model-len=192K reserves softmax_lse for 192K; the same prefill on llama.cpp reserves softmax_lse for 25K. See upstream root cause Dao-AILab/flash-attention#1011 (open since 2024).

2. Different KV / workspace model. vLLM uses paged attention with varlen kernels that need per-call workspace pre-allocated from a worst-case bound. llama.cpp pre-allocates the full KV cache at boot as one contiguous slab per layer (sized by --ctx-size), then attention forward calls allocate workspace dynamically by current chunk. The static-slab + dynamic-workspace pattern means activation pressure per call is proportional to actual token count, not max-model-len.

3. Cudagraph capture path differs. llama.cpp's cudagraph support (added recently for decode TPS) is decode-only — prefill goes through the imperative ggml graph. So there's no path for max_model_len to leak through capture metadata into kernel signatures. The cap can't leak because there's nowhere for it to leak through.

Bonus — Cliff 2 is also handled differently. llama.cpp's Qwen3-Next implementation processes DeltaNet/GDN layers with online state updates rather than allocating the O(seq_len × chunk_size) intermediate that fla.ops.chunk_gated_delta_rule_fwd materializes in vLLM. So Cliff 2 doesn't fire either — single prompts at 50K+, 100K+, 200K+ all process cleanly.

Net effect: same model, same hardware, but the engines are different memory machines. vLLM's design optimizes for batched serving with fixed-shape kernels and cudagraph capture (which requires worst-case workspace allocation). llama.cpp's design serves one request at a time with dynamic shapes (cheaper per-call, no batching wins). The 3-4× TPS gap (70+ vLLM vs ~21 llama.cpp on this stack) and the prefill-cliff gap come from the same architectural choice — they're two sides of the same trade.

This is exactly why our launch frame is two routes, not one (README): vLLM dual-card for max throughput in environments where you control prompt shape, llama.cpp single-card for max robustness when prompts can balloon unpredictably.

Pros

Pro	Detail
Lightweight	Single binary, ~50 MB. No Docker required (though Docker images exist).
Fastest cold start	~30 sec from launch to first token. vLLM takes ~2 min.
Lowest VRAM overhead	No inference-framework taxes — pure model + KV cache.
GGUF format	Many quant options (Q4_K_M, Q5_K_S, IQ4_XS, etc.). Easy to swap.
Cross-platform	Works on AMD (ROCm), Intel (oneAPI), Apple Silicon (Metal), CPU-only. vLLM is NVIDIA-only.
Active community	Lots of distros — Ollama, LM Studio, LocalAI, koboldcpp, etc.
MTP spec-decode on mainline	PR #22673 merged 2026-05-16. Qwen3-Next MTP head loads natively with `--multi-token-prediction --draft-max N`. No fork needed.
Luce DFlash fork available	For N=5 code workloads, Luce's fork ships DFlash N=5. Requires source build.

Cons

Con	Detail
Qwen3-Next support still landing	Need the right binary build. Mainline `llama.cpp` works but lags vLLM on edge cases (some attention variants, MTP head loading).
Server feature parity behind vLLM	Upstream `llama-server` doesn't expose `--enable-auto-tool-choice`. Need a wrapper (Open WebUI, LM Studio, Ollama with custom modelfile) for tool-call extraction.
No TurboQuant equivalent	KV cache is fp16 / fp8 / q4_0 / q5_1 / q8_0 / turbo3 (in Tom's fork). None as compact as vLLM's TQ3 → max usable ctx is ~64K with Q4_K_M on a single 3090.
Concurrent serving is sluggish	`llama-server` forks per request. Two simultaneous requests → second waits or both slow. Not designed for multi-tenant.
MTP now on mainline	PR #22673 merged 2026-05-16 — Qwen3-Next MTP head loads natively. For N=5 code workloads, Luce DFlash fork is still available but requires a source build.

Quick recipe — mainline llama.cpp + GGUF

1. Get a GGUF quant

Unsloth ships Qwen3.6-27B GGUFs (Q4_K_M ~16 GB, IQ4_XS ~14 GB, Q5_K_S ~18 GB). Or download the full BF16 and quantize yourself with llama-quantize.

⚠️ Don't use aria2c to download multi-GB GGUFs. It silently corrupts files during stall cycles — they'll have the right size but wrong bytes. Use hf download instead. Always sha256sum verify if available.

# Use hf CLI (pip install 'huggingface-hub[hf_transfer]')
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir $MODEL_DIR/qwen3.6-27b-gguf/

Easier: WEIGHTS=gguf bash scripts/setup.sh qwen3.6-27b does this download for you — Q4_K_M + mmproj, SHA-verified, into the path the composes expect — and skips Genesis (llama.cpp doesn't need it).

Confirm size matches the HuggingFace listing. If a sha256 is published, verify it.

2. Build llama.cpp

git clone https://github.com/ggerganov/llama.cpp /opt/llama.cpp
cd /opt/llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

For ROCm: -DGGML_HIPBLAS=ON. For Apple Silicon: builds with Metal by default.

3. Launch the server

For a sane mid-context default (65K, plenty for chat + light agent work):

/opt/llama.cpp/build/bin/llama-server \
  -m $MODEL_DIR/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
  -c 65536 \
  --host 0.0.0.0 --port 8020 \
  -ngl 999 \
  --jinja

-ngl 999 puts all layers on GPU (use -ngl 35 or similar to split with CPU if you have less VRAM). --jinja enables chat template processing.

Going to 262K (full Qwen3.6 context) on a single 3090

This works on llama.cpp because Q4_K_M is ~16 GB on disk + a quantized KV cache at q4_0 leaves comfortable headroom. The vLLM stack can't get here on one card without splitting prompts — this is genuinely llama.cpp's win.

Memory math at 262K context with Q4_K_M + q4_0 KV (single 3090):

Model on disk: ~16 GB
KV cache at 262K (q4_0 K + q4_0 V): ~5 GB
Total VRAM: ~21 GB, leaving ~3 GB headroom for prompts and activation peaks

Recipe (community-reported, validated by multiple users on r/LocalLLaMA):

/opt/llama.cpp/build/bin/llama-server \
  -m $MODEL_DIR/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
  -ngl 99 \
  -c 262144 \
  -np 1 \
  -fa on \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 0.0.0.0 --port 8020 \
  --jinja

What each flag does:

-ngl 99 — offload all layers to GPU (some users use 999 instead, equivalent)
-c 262144 — full 262K context window
-np 1 — single user slot. Don't enable multi-slot here — the KV pool gets divided across slots, eating your headroom.
-fa on — flash attention on (memory + speed both win on Ampere+)
--cache-type-k q4_0 --cache-type-v q4_0 — the unlock — see KV cache type table below

Sustained throughput at 262K with this config is typically 35-45 tok/s on a stock 3090 (community-reported flat curve at any in-budget context).

4. Vision (optional)

Download the mmproj model:

hf download unsloth/Qwen3.6-27B-GGUF mmproj-F16.gguf --local-dir $MODEL_DIR/qwen3.6-27b-gguf/

Add to launch:

--mmproj $MODEL_DIR/qwen3.6-27b-gguf/mmproj-F16.gguf

5. Tool calls (limited)

llama-server doesn't have built-in --enable-auto-tool-choice. Workarounds:

Ollama wraps llama.cpp and adds tool-call extraction (uses Modelfile's TEMPLATE directive). Easiest path.
Open WebUI can do client-side tool-call extraction from <tool_call>...</tool_call> strings.
Custom wrapper — proxy that parses tool-call XML out of completions before returning to client.

For first-class tool calls in OpenAI format, vLLM is still the easiest option.

Recipe — DFlash N=5 via Luce fork (for code workloads)

If you want spec-decode equivalent to vLLM's MTP path:

git clone --recurse-submodules https://github.com/Luce-Org/lucebox-hub /opt/lucebox-hub
cd /opt/lucebox-hub
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# Download draft model (~500 MB)
hf download z-lab/Qwen3.6-27B-DFlash --local-dir $MODEL_DIR/z-lab/Qwen3.6-27B-DFlash/

# Launch
/opt/lucebox-hub/build/bin/llama-server \
  -m $MODEL_DIR/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
  --draft $MODEL_DIR/qwen3.6-27b-dflash-gguf/dflash-N5.gguf \
  --draft-max 5 \
  --draft-min 1 \
  -c 65536 \
  -ngl 999 \
  --host 0.0.0.0 --port 8004 \
  --jinja

Measured on this stack (single 3090, Q4_K_M main + DFlash N=5 draft, code prompts): ~106 TPS mean code TPS, AL 4.74, accept 30.6%. Matches Luce's published README numbers.

Trade-off: the server forks per request, so chat UX feels sluggish (second request waits on first). For long generation tasks (single-shot code synthesis, document summarization), the per-request fork is fine.

🆕 Dual-GPU split (lucebox-hub PRs #78 + #80, May 2026)

If you have two GPUs (e.g. 2× 3090), lucebox-hub now supports a heterogeneous-spec-decode topology — target on GPU 0, draft on GPU 1, no TP weight-sharding. This removes the single-card 65K max_ctx ceiling we previously documented (target + draft + KV all competing for 24 GB).

# Target on GPU 0, DFlash draft on GPU 1
/opt/lucebox-hub/build/bin/llama-server \
  -m $MODEL_DIR/qwen3.5-27b-gguf/Qwen3.5-27B-Q4_K_M.gguf \
  --draft $MODEL_DIR/qwen3.5-27b-dflash-gguf/dflash-N5.gguf \
  --target-gpu 0 --draft-gpu 1 \
  --draft-max 16 --draft-min 1 \
  -c 262144 \
  --host 0.0.0.0 --port 8004 \
  --jinja

Or pin PFlash drafter to its own GPU via --pflash-gpu (separate pflash_daemon workflow — see Luce-Org/lucebox-hub PR #78).

@weicj's measured result on dual RTX 2080 Ti 22 GB:

DFlash dual-GPU: 51.86 tok/s HumanEval 10-prompt, AL 7.09, 44.3% accept (Qwen3.5-27B Q4 target + z-lab DFlash draft)
PFlash phase-split: passing NIAH source ctx 24K → 262K (10.7×) vs single-card co-resident

Recommended for our 2× 3090 stack today: only with Qwen3.5-27B — the Qwen3.6-27B DFlash draft is still under training (per Luce-Org/lucebox-hub README 2026-04-26 snapshot). Untested on 2× 3090; tracked at club-3090 task #229.

Tuning levers

`--cache-type-k` / `--cache-type-v` — the biggest single lever

This is the most consequential knob on a 24 GB card. Most tutorials don't cover it.

KV type	Per-token bytes	Fits at 262K on 24 GB?	Decode speed (vs q4_0)	Notes
`f16` (default)	~12 KB	❌ doesn't fit	n/a	Default in many guides — wrong choice for max-ctx on consumer cards
`q8_0`	~6 KB	⚠️ fits at ~23 GB but slow	~3× slower	Community-reported on Qwen3.6 27B + 3090; flash-attention path doesn't optimize q8 the way it does q4_0
`q4_0` ⭐	~3 KB	✅ fits at ~21 GB	full speed (40 tok/s flat)	The right pick for max context on consumer hardware
`turbo3` (Tom's fork)	~2 KB	✅ fits with margin	full speed (with the fork)	Even more compact; not yet upstream — watch llama.cpp PR #21089

The q8 → q4_0 jump is counter-intuitive because q8 is "higher precision" — you'd expect it to be slower for higher quality. In practice on this hardware:

q8 KV cache hits a slower kernel path on flash-attention
The "more precision" gain on KV is invisible at quant-noise levels of a 4-bit model
q4_0 is the dominated choice on every axis except theoretical KV precision

Test it on your own rig before trusting the 3× claim — but if you're on f16 or q8 KV at 262K and seeing slow decode, swap to q4_0 first.

Other levers

--ctx-size — 65K is the comfortable default ceiling with f16 KV; 262K only works with q4_0 KV (see table above).
--threads N — CPU threads for non-GPU ops. Set to physical-cores / 2 typically.
-fa on (flash attention) — usually faster on modern GPUs. Required for the q4_0 KV path to perform well.

When llama.cpp is the right pick

✅ You want minimal setup (single binary, no Docker)
✅ You're on AMD / Intel / Apple Silicon (vLLM is NVIDIA-only)
✅ You're embedding inference in another tool (LM Studio, Ollama, Faraday, etc.)
✅ You don't need concurrent multi-tenant serving
✅ You're OK with no first-class tool-call extraction (or use Ollama as a wrapper)

When to use vLLM instead

You need full OpenAI API parity (tools, streaming, structured output)
You want max context (>214K) on a single 3090 — vLLM single-card now ships 214K text-only / 198K with vision since the 2026-05-01 v0.20 + Genesis v7.65 dev tip migration (see docs/CLIFFS.md "v0.20 unblock"); llama.cpp goes to 262K
You need concurrent serving (multi-tenant)
You want vLLM's MTP with continuous batching (multi-tenant)
You're hitting llama.cpp's Qwen3-Next limitations and want the actively-developed path

Watch list (when llama.cpp catches up)

llama.cpp PR #21089 — TurboQuant KV cache landing (CPU first, CUDA follow-on). When CUDA path lands, turbo3 becomes a first-class option on llama.cpp.
MTP spec-decode — ✅ merged on mainline (PR #22673, 2026-05-16). No longer pending.
DFlash mainline integration — currently fork-only (Luce's lucebox-hub).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

llama.cpp — quick recipe + gotchas

TL;DR

Why pick llama.cpp over vLLM on a single 3090?

Why llama.cpp doesn't hit the prefill cliffs vLLM does

Pros

Cons

Quick recipe — mainline llama.cpp + GGUF

1. Get a GGUF quant

2. Build llama.cpp

3. Launch the server

Going to 262K (full Qwen3.6 context) on a single 3090

4. Vision (optional)

5. Tool calls (limited)

Recipe — DFlash N=5 via Luce fork (for code workloads)

🆕 Dual-GPU split (lucebox-hub PRs #78 + #80, May 2026)

Tuning levers

`--cache-type-k` / `--cache-type-v` — the biggest single lever

Other levers

When llama.cpp is the right pick

When to use vLLM instead

Watch list (when llama.cpp catches up)

See also

FilesExpand file tree

LLAMA_CPP.md

Latest commit

History

LLAMA_CPP.md

File metadata and controls

llama.cpp — quick recipe + gotchas

TL;DR

Why pick llama.cpp over vLLM on a single 3090?

Why llama.cpp doesn't hit the prefill cliffs vLLM does

Pros

Cons

Quick recipe — mainline llama.cpp + GGUF

1. Get a GGUF quant

2. Build llama.cpp

3. Launch the server

Going to 262K (full Qwen3.6 context) on a single 3090

4. Vision (optional)

5. Tool calls (limited)

Recipe — DFlash N=5 via Luce fork (for code workloads)

🆕 Dual-GPU split (lucebox-hub PRs #78 + #80, May 2026)

Tuning levers

--cache-type-k / --cache-type-v — the biggest single lever

Other levers

When llama.cpp is the right pick

When to use vLLM instead

Watch list (when llama.cpp catches up)

See also

`--cache-type-k` / `--cache-type-v` — the biggest single lever