feat: TurboShimmy INT4 KV cache — v2.1.0 release

Author: Michael-A-Kuykendall

CHANGELOG.md

@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.1.0] - 2026-06-02
+
+### Added
+- **TurboShimmy INT4 KV cache** — per-head-vector INT4 compression for KV buffers,
+  cutting KV VRAM usage ~7× with no measured retrieval loss. Enable via
+  `SHIMMY_KV_QUANT=int4`. See airframe 0.2.0 release notes.
+
+### Fixed
+- **wgpu 27 staging-buffer panic on GTX 1050 Ti and similar older GPUs** (issue #205,
+  reported by @Kuntey). Upgraded to airframe 0.2.1, which fixes `max_buffer_size`
+  being incorrectly capped to `max_storage_buffer_binding_size`. On some older GPU/driver
+  stacks these limits differ, causing a deferred validation error that wgpu 27 surfaced as
+  a cryptic "Staging Buffer is invalid" panic. A pre-flight size guard now returns a clear
+  error message if the model exceeds the GPU's binding limit.
+- Removed spurious `WARNING: Missing Norm Tensor post_attention_norm/post_ffw_norm` log
+  spam for Llama, Mistral, Phi and Qwen models. These tensors are Gemma-only.
+
+### Changed
+- Airframe dependency updated to 0.2.1.
+
+---
+
 ## [2.0.0] - 2026-05-26
 
 ### 🚀 **v2.0 PUBLIC RELEASE** — Airframe GPU Engine

Cargo.lock

@@ -1,6 +1,9 @@
 [package]
 name = "shimmy"
-version = "2.0.1"
+version = "2.1.0"
+# publish = false while airframe 0.2.1 is local (path dep). Remove path dep and
+# this line once airframe 0.2.1 is published to crates.io, then `cargo publish`.
+publish = false
 edition = "2021"
 license = "MIT"
 description = "Lightweight Ollama-compatible inference server with native SafeTensors support. No Python dependencies, cross-platform WebGPU acceleration via Airframe."
@@ -19,7 +22,6 @@ include = [
     "LICENSE",
     "build.rs"
 ]
-publish = true
 
 [features]
 default = ["airframe", "huggingface"]  # Full GPU build; use --no-default-features --features huggingface for CPU-only
@@ -68,7 +70,7 @@ dirs = "5.0"
 reqwest = { version = "0.12", features = ["json", "rustls-tls", "stream"], default-features = false }
 
 # Airframe native GPU inference — public crate on crates.io
-airframe = { version = "0.1", optional = true }
+airframe = { version = "0.2.1", path = "../airframe", optional = true }
 
 [dev-dependencies]
 tokio-tungstenite = "0.20"

Cargo.toml

@@ -36,6 +36,7 @@
 
 - [What Is Shimmy?](#drop-in-openai-api-replacement-for-local-llms)
 - [🔥 Airframe Engine (v2.0)](#-airframe-engine)
+- [⚡ TurboShimmy INT4 KV (v2.1)](#-turboshimmy-int4-kv)
 - [🎯 Supported Models](#-supported-models)
 - [📦 Migrating from v1.x](#-migrating-from-v1x)
 - [⚡ Quick Start (30 seconds)](#quick-start-30-seconds)
@@ -59,6 +60,8 @@ Shimmy is a **single-binary** that provides **100% OpenAI-compatible endpoints**
 
 **🎉 NEW in v2.0.0**: Shimmy now runs on [Airframe](#-airframe-engine), a pure-Rust WGSL GPU engine. No C++ toolchain, no backend flags, no compilation required.
 
+**⚡ NEW in v2.1.0**: [TurboShimmy INT4 KV](#-turboshimmy-int4-kv) — ~7× less KV cache VRAM with one flag. Run Llama-3.2-3B on 4 GB GPUs.
+
 ## 🔥 Airframe Engine
 
 Starting in v2.0.0, Shimmy's default inference engine is **Airframe** — a pure-Rust WebGPU (WGSL) transformer runtime built from scratch.
@@ -79,6 +82,61 @@ SHIMMY_BASE_GGUF=/path/to/TinyLlama-1.1B-Chat-v1.0.Q4_0.gguf ./shimmy serve
 SHIMMY_BASE_GGUF=/path/to/model.gguf SHIMMY_MAX_CTX=4096 ./shimmy serve
 ```
 
+## ⚡ TurboShimmy INT4 KV
+
+**TurboShimmy** is Shimmy's on-GPU INT4 KV-cache compression system, shipping in v2.1.0. It squeezes the KV cache from 32-bit floats down to per-head-vector 4-bit integers — entirely in WGSL compute shaders with no CPU roundtrips — delivering ~7× less KV VRAM with no measurable quality loss at normal context lengths.
+
+**One flag. ~7× less KV VRAM. Same output quality.**
+
+```bash
+# Enable TurboShimmy on any GGUF model
+./shimmy serve --kv-quant int4
+
+# Or via environment variable (docker-compose, systemd, etc.)
+SHIMMY_KV_QUANT=int4 ./shimmy serve
+
+# Windows GPU + long prompts: reduce per-dispatch work to prevent TDR resets
+./shimmy serve --kv-quant int4 --prefill-chunk 8
+```
+
+**Why it matters** — TurboShimmy changes what fits on your GPU:
+
+| GPU VRAM | Without TurboShimmy | With TurboShimmy (`--kv-quant int4`) |
+|---|---|---|
+| 3 GB | Llama-3.2-1B only | **Llama-3.2-3B fits ✅** |
+| 4 GB | Llama-3.2-3B, ctx=2048 (tight) | **Llama-3.2-3B at ctx=8192 ✅** |
+| 6 GB | 3B models, short context | **7B models with reasonable context ✅** |
+
+**VRAM comparison (Llama-3.2-3B, ctx=2048):**
+
+| Mode | KV cache | Total VRAM | Min GPU needed |
+|---|---|---|---|
+| Default (f32) | ~512 MB | ~2.4 GB | 3 GB (tight) |
+| TurboShimmy (int4) | **~72 MB** | **~2.0 GB** | **2.5 GB ✅** |
+
+**VRAM comparison (TinyLlama 1.1B, ctx=2048):**
+
+| Mode | KV cache | Total VRAM |
+|---|---|---|
+| Default (f32) | 88 MB | ~700 MB |
+| TurboShimmy (int4) | **~13 MB** | **~650 MB** |
+
+**When to use TurboShimmy:**
+
+| Situation | Recommendation |
+|---|---|
+| 3B model on a 4 GB GPU | `--kv-quant int4` — enables models that wouldn't fit otherwise |
+| 7B model at ctx=4096+ | `--kv-quant int4` — cuts KV from ~512 MB → ~72 MB |
+| Short chat sessions (ctx ≤ 2048) | `--kv-quant int4` — safe, no quality tradeoff |
+| Long-form generation (ctx > 8192) | Default `f32` — keep maximum quality |
+| Windows GPU + TDR crashes on long prompts | `--kv-quant int4 --prefill-chunk 8` |
+
+**How it works:** Each K/V head vector is independently quantized to 4-bit integers with a per-vector F32 scale factor, encoded as packed nibbles by WGSL compute shaders. Dequantization happens on-the-fly when computing attention scores — also on GPU. The Airframe engine's helical context-shift operates directly on the packed INT4 representation. No CPU roundtrips at any step. Full architecture details in the [Airframe engine](https://github.com/Michael-A-Kuykendall/airframe).
+
+> **Quality validation:** Needle-in-a-haystack benchmarks on Llama-3.2-3B show zero retrieval degradation vs F32 at ctx≤2048 across all tested depths (15%, 50%, 85%). Full benchmark data and setup guide: [TurboShimmy on the wiki](https://github.com/Michael-A-Kuykendall/shimmy/wiki/TurboShimmy).
+
+> **Windows stability:** Airframe v0.2.1 ships a `device.on_uncaptured_error` handler so GPU validation errors surface as clean HTTP 500 responses instead of crashes. Use `--prefill-chunk 8` to prevent Windows TDR resets during long prefills on older GPUs (GTX 10xx/16xx series).
+
 ## 🎯 Supported Models
 
 Airframe v2.0 ships with GPU-verified support across **7 model architectures** and **5 quantization types**, covering the models most commonly run on consumer hardware. Context window is read directly from each model's GGUF metadata — no hardcoded limits.
@@ -452,6 +510,8 @@ I built Shimmy to retain privacy-first control on my AI development and keep thi
 | `SHIMMY_MODEL_PATHS` | *(see Zero Config)* | Colon-separated extra model search paths |
 | `SHIMMY_ENGINE_BACKEND` | `airframe` | `airframe` (default) or `llama` (legacy path) |
 | `SHIMMY_ROPE_SCALE` | *(auto)* | Override computed YaRN scale factor |
+| `SHIMMY_KV_QUANT` | `f32` | KV cache quantization: `f32` (default) or `int4` ([TurboShimmy](#-turboshimmy-int4-kv)) |
+| `SHIMMY_PREFILL_CHUNK` | `64` | Tokens per prefill dispatch. Set to `8` on Windows if you see GPU TDR resets on long prompts |
 | `RUST_BACKTRACE` | *(off)* | Set to `1` to print crash backtraces |
 
 ### CLI Commands
@@ -460,6 +520,8 @@ shimmy serve                              # Start server (auto port allocation)
 shimmy serve --bind 127.0.0.1:8080        # Manual port binding
 shimmy serve --gpu-backend auto           # WebGPU adapter auto-select (default)
 shimmy serve --gpu-backend cpu            # Force CPU (disable GPU)
+shimmy serve --kv-quant int4              # Enable TurboShimmy INT4 KV cache compression
+shimmy serve --kv-quant int4 --prefill-chunk 8  # INT4 + Windows TDR prevention
 shimmy list                               # Show available models
 shimmy discover                           # Refresh model discovery
 shimmy generate --name X --prompt "Hi"   # Test generation

README.md

@@ -13,6 +13,13 @@ services:
       - SHIMMY_BASE_GGUF=/app/models      # Point to mounted models
       - SHIMMY_PORT=11434                 # Server port
       - SHIMMY_HOST=0.0.0.0              # Listen on all interfaces
+      # TurboShimmy v2.1 — cut KV cache VRAM ~60% (recommended for 4 GB GPUs)
+      # Uncomment to enable INT4 KV quantization:
+      # - SHIMMY_KV_QUANT=int4
+      # Uncomment to reduce prefill chunk size (prevents Windows TDR crashes on long prompts):
+      # - SHIMMY_PREFILL_CHUNK=8
+      # Uncomment to override max context length (default: model's native ctx from GGUF):
+      # - SHIMMY_MAX_CTX=4096
     restart: unless-stopped
     deploy:
       resources:
@@ -24,4 +31,4 @@ services:
 
 volumes:
   shimmy-cache:
-    driver: local
+    driver: local

docker-compose.yml

@@ -97,6 +97,40 @@ curl http://127.0.0.1:11435/v1/chat/completions \
 
 ---
 
+## ⚡ TurboShimmy INT4 KV —— v2.1.0 新功能
+
+**TurboShimmy** 是 Shimmy v2.1.0 带来的纯 GPU INT4 KV 缓存压缩系统。通过 WGSL 计算着色器，将 KV 缓存从 32 位浮点数压缩为逐头向量 4 位整数，全程在 GPU 上完成。**一行指令，约 7 倍 KV 显存节省，输出品质不变。**
+
+```bash
+# 启用 TurboShimmy
+SHIMMY_KV_QUANT=int4 ./shimmy serve
+
+# 或通过命令行参数
+./shimmy serve --kv-quant int4
+
+# Windows + 长提示：预防 GPU TDR 重置
+./shimmy serve --kv-quant int4 --prefill-chunk 8
+```
+
+**TurboShimmy 改变了消费级 GPU 的可用范围：**
+
+| GPU 显存 | 未启用 TurboShimmy | 启用后（`--kv-quant int4`） |
+|---|---|---|
+| 3 GB | 仅能运行 Llama-3.2-1B | **Llama-3.2-3B 可运行 ✅** |
+| 4 GB | Llama-3.2-3B，ctx=2048（勉强） | **Llama-3.2-3B，ctx=8192 ✅** |
+| 6 GB | 3B 模型，短上下文 | **7B 模型，合理上下文 ✅** |
+
+**显存对比（Llama-3.2-3B，ctx=2048）：**
+
+| 模式 | KV 缓存 | 总显存 | 最低显存 |
+|---|---|---|---|
+| 默认（f32） | ~512 MB | ~2.4 GB | 3 GB（勉强） |
+| TurboShimmy（int4） | **~72 MB** | **~2.0 GB** | **2.5 GB ✅** |
+
+> **品质验证：** 在 Llama-3.2-3B 上进行的“大海捕针”基准测试表明，ctx≤2048 时 INT4 对比 F32 检索准确率零退化（各测试深度 15%〈50%〈85% 均为 100%）。详细文档：[TurboShimmy Wiki](https://github.com/Michael-A-Kuykendall/shimmy/wiki/TurboShimmy-zh-CN)。
+
+---
+
 ## 🖥️ 已验证的 GPU 支持
 
 | 平台 | GPU 类型 | 后端 |
@@ -133,6 +167,9 @@ Shimmy 是纯 Rust 实现，无 Python 运行时，无 C++ 依赖，启动时间
 **如何选择模型量化格式？**
 日常使用首选 `Q4_K_M`——在文件大小和推理质量之间取得了最好的平衡。若追求最高质量且显存充足，选 `Q8_0`。详见[量化格式详解](QUANTIZATION.md)。
 
+**如何在 4 GB 显存的显卡上运行 3B 模型？**
+启用 TurboShimmy：`SHIMMY_KV_QUANT=int4 ./shimmy serve`。这将 KV 显存减少约 7 倍，使 Llama-3.2-3B 能在 2.5 GB 总显存下运行。详见[上方 TurboShimmy 节](#turboshimmy-int4-kv--v210-)。
+
 **上下文长度不够怎么办？**
 设置 `SHIMMY_MAX_CTX=8192`（或更高）即可，Airframe 会自动应用 YaRN RoPE 缩放。注意超出模型原生上下文 2 倍以上时质量会有所下降。详见[扩展上下文窗口](EXTENDED_CONTEXT.md)。
 

docs/zh-CN/README.md

@@ -133,6 +133,9 @@ Shimmy 是純 Rust 實作，無 Python 執行環境，無 C++ 相依，啟動時
 **如何選擇模型量化格式？**
 日常使用首選 `Q4_K_M`——在檔案大小和推論品質之間取得了最好的平衡。若追求最高品質且顯存充足，選 `Q8_0`。詳見[量化格式詳解](QUANTIZATION.md)。
 
+**如何在 4 GB 顯示記憶體的顯示卡上執行 3B 模型？**
+啟用 TurboShimmy：`SHIMMY_KV_QUANT=int4 ./shimmy serve`。這將 KV 顯示記憶體減少約 7 倍，使 Llama-3.2-3B 能在 2.5 GB 總顯示記憶體下執行。詳見[上方 TurboShimmy 節](#turboshimmy-int4-kv--v210-)。
+
 **上下文長度不夠怎麼辦？**
 設定 `SHIMMY_MAX_CTX=8192`（或更高）即可，Airframe 會自動套用 YaRN RoPE 縮放。注意超出模型原生上下文 2 倍以上時品質會有所下降。詳見[擴展上下文視窗](EXTENDED_CONTEXT.md)。
 

docs/zh-TW/README.md

@@ -37,6 +37,27 @@ pub struct Cli {
     /// Use legacy CPU adapter instead of Airframe GPU
     #[arg(long, global = true)]
     pub legacy: bool,
+
+    /// Enable TurboShimmy INT4 KV cache to cut VRAM usage ~60% with minimal quality loss.
+    /// Values: f32 (default, full precision) | int4 (TurboShimmy — recommended for 4GB+ GPUs)
+    #[arg(
+        long,
+        global = true,
+        value_name = "MODE",
+        default_value = "f32",
+        value_parser = ["f32", "int4"],
+        help = "KV cache precision: f32 (default) or int4 (TurboShimmy — ~60% less VRAM)"
+    )]
+    pub kv_quant: String,
+
+    /// Prefill chunk size for long-context stability on Windows (default: 64).
+    /// If you see GPU TDR crashes on long prompts, set this to 8 or 16.
+    #[arg(long, global = true, value_name = "N")]
+    pub prefill_chunk: Option<u32>,
+
+    /// Override maximum context length (tokens). Default: model's native context from GGUF.
+    #[arg(long, global = true, value_name = "N")]
+    pub ctx: Option<u32>,
 }
 
 #[derive(Subcommand, Debug)]

src/cli.rs

@@ -428,15 +428,24 @@ mod tests {
 
         let discovery = ModelDiscovery::from_env();
 
-        // Should include home-based paths
-        assert!(discovery
-            .search_paths
-            .iter()
-            .any(|p| p.to_string_lossy().contains(".cache/huggingface")));
-        assert!(discovery
-            .search_paths
-            .iter()
-            .any(|p| p.to_string_lossy().contains("models")));
+        // Should include home-based paths. Use slash-normalized comparison so
+        // the test passes on Windows (PathBuf uses backslashes on Windows).
+        assert!(
+            discovery
+                .search_paths
+                .iter()
+                .any(|p| p.to_string_lossy().replace('\\', "/").contains(".cache/huggingface")),
+            "Expected .cache/huggingface in search paths, got: {:?}",
+            discovery.search_paths
+        );
+        assert!(
+            discovery
+                .search_paths
+                .iter()
+                .any(|p| p.to_string_lossy().contains("models")),
+            "Expected models dir in search paths, got: {:?}",
+            discovery.search_paths
+        );
 
         // Restore original environment
         env::remove_var("HOME");