|
3 | 3 | Notable changes to sparkinfer. Format loosely follows [Keep a Changelog](https://keepachangelog.com); |
4 | 4 | versions track the GitHub [releases](https://github.com/gittensor-ai-lab/sparkinfer/releases). |
5 | 5 |
|
| 6 | +## [0.4.3] — 2026-07-21 |
| 7 | + |
| 8 | +sparkinfer's **prefill stack is the headline of this release**: Qwythos (Qwen3.5) is now |
| 9 | +**+183.6% faster than llama.cpp at 128k prefill**, and Qwen3.6 is **+82.7% faster at 32k prefill** — |
| 10 | +both on the same RTX 5090 / pinned llama.cpp commit (`6f4f53f`). The live chat at |
| 11 | +**[sparkinfer.com/chat](https://sparkinfer.com/chat)** now serves **Qwen3.6**. |
| 12 | + |
| 13 | +### ⚡ Prefill — the main story |
| 14 | + |
| 15 | +Dense + MoE prefill landed a chain of weight-amortized and int8 tensor-core wins since v0.4.2. |
| 16 | +Long-context prompt processing is where agents feel latency first — this release closes that gap |
| 17 | +past llama.cpp at the lengths that matter. |
| 18 | + |
| 19 | +#### Qwythos (Qwen3.5-9B) · Q4_K_M · RTX 5090 |
| 20 | + |
| 21 | +| context | sparkinfer (pp tok/s) | llama.cpp (pp tok/s) | vs llama | |
| 22 | +|---|---:|---:|---:| |
| 23 | +| **4k prefill** | **~19,580** | 11,105 | **~+76%** | |
| 24 | +| **32k prefill** | **~20,170** | 9,772 | **~+106%** | |
| 25 | +| **64k prefill** | **~20,150** | 8,154 | **~+147%** | |
| 26 | +| **128k prefill** | **~17,015** | 6,000 | **+183.6%** | |
| 27 | + |
| 28 | +Headline: **128k prefill +183.6% vs llama.cpp** (17,015 vs 5,999.59 pp tok/s; #557). |
| 29 | + |
| 30 | +#### Qwen3.6-35B-A3B · UD-Q4_K_M · RTX 5090 |
| 31 | + |
| 32 | +Expert-grouped int8 MoE prefill (#530 → #537 → #548 → #553) lifts long-prompt MoE far past the |
| 33 | +v0.4.2 decode-focused frontier. |
| 34 | + |
| 35 | +| context | sparkinfer (pp tok/s) | llama.cpp (pp tok/s) | vs llama | |
| 36 | +|---|---:|---:|---:| |
| 37 | +| **512 prefill** | **~3,510** | 8,737 | (short-N still climbing) | |
| 38 | +| **4k prefill** | **~11,580** | 8,726 | **~+33%** | |
| 39 | +| **16k prefill** | **~14,160** | 8,390 | **~+69%** | |
| 40 | +| **32k prefill** | **~14,587** | 7,984 | **+82.7%** | |
| 41 | + |
| 42 | +Headline: **32k prefill +82.7% vs llama.cpp** (14,587 vs 7,984 pp tok/s). |
| 43 | + |
| 44 | +### 🌐 Chat — Qwen3.6 on sparkinfer.com |
| 45 | + |
| 46 | +| | | |
| 47 | +|---|---| |
| 48 | +| **Chat** | [sparkinfer.com/chat](https://sparkinfer.com/chat) — now includes **Qwen3.6** | |
| 49 | +| **Website** | [sparkinfer.com](https://sparkinfer.com/) | |
| 50 | +| **Demo API** | [api.sparkinfer.com](https://api.sparkinfer.com/) — OpenAI-compatible | |
| 51 | + |
| 52 | +### Prefill optimizations landed since v0.4.2 |
| 53 | + |
| 54 | +- **#531** (`eval:XL`) — faithful batched Qwythos prefill through 128k |
| 55 | +- **#530** (`eval:XL`) — batched weight-amortized MoE prefill for Qwen3.6 |
| 56 | +- **#537** (`eval:XL`) — expert-grouped int8 MoE prefill (large long-ctx jump) |
| 57 | +- **#552** (`eval:L`) — `mma.sync` bf16 prefill GEMM for dense long-ctx |
| 58 | +- **#548** (`eval:XL`) — chunk-parallel GDN scan + faster prefill dequant/GEMM |
| 59 | +- **#557** (`eval:XL`) — selective int8 FFN+attn at long ctx — **128k +183.6% vs llama** |
| 60 | +- **#553** (`eval:XL`) — single-pass Q→i8 row dequant for Qwen3.6 short-N |
| 61 | +- **#561** (`eval:XS`) — expert-group L2 MoE prefill for short-N (N≤512) |
| 62 | + |
| 63 | +### Eval harness & trust |
| 64 | + |
| 65 | +- **#529** — bidir prefill scoring for Qwen3.5 and Qwen3.6 |
| 66 | +- **#564** — REJECT when any no-regression gate fails |
| 67 | +- **#567** — tighter H2 long-context bars (top1≥0.90, KL≤0.5) |
| 68 | +- **#568** — copycat guard: skip main-shared tiny helpers |
| 69 | + |
| 70 | +### What changed since v0.4.2 |
| 71 | + |
| 72 | +| headline | v0.4.2 | v0.4.3 | shift | |
| 73 | +|---|---:|---:|---| |
| 74 | +| Qwythos prefill at 128k | ~6,888 pp/s (~+15% vs llama) | **~17,015 pp/s (+183.6% vs llama)** | **~2.5×** | |
| 75 | +| Qwen3.6 prefill at 32k | ~1,282 pp/s (behind llama) | **~14,587 pp/s (+82.7% vs llama)** | **~11×** | |
| 76 | +| Live chat | Qwythos-focused demo | **Qwen3.6 on [sparkinfer.com/chat](https://sparkinfer.com/chat)** | new | |
| 77 | + |
| 78 | +**Verified:** RTX 5090 · Qwythos prefill **~17,015 pp/s at 128k (+183.6% vs llama)** · Qwen3.6 prefill |
| 79 | +**~14,587 pp/s at 32k (+82.7% vs llama)** · Polaris-attested eval logs · llama.cpp `6f4f53f`. |
| 80 | + |
| 81 | +### Contributors |
| 82 | + |
| 83 | +- **@Paral1995** — #531 (batched Qwythos to 128k), #552 (bf16 `mma.sync` GEMM), #557 (int8 long-ctx FFN+attn) |
| 84 | +- **@James-CUDA** — #537 (expert-grouped int8 MoE), #561 (short-N L2 MoE groups) |
| 85 | +- **@inference2026** — #530 (weight-amortized MoE prefill), #553 (Q→i8 row dequant) |
| 86 | +- **@fansilas** — #548 (chunk-parallel GDN + prefill dequant/GEMM) |
| 87 | +- **@skyrocket2026** — #529 (bidir prefill scoring), #564/#567/#568 (eval gates + copycat), dashboard + release |
| 88 | + |
6 | 89 | ## [0.4.2] — 2026-07-17 |
7 | 90 |
|
8 | 91 | sparkinfer now **beats llama.cpp on Qwythos prefill at every tracked context** — climbing from **290 → 16,083 pp tok/s** |
|
0 commit comments