Merge pull request #430 from CrispStrobe/perf/81-round2

build: update ggml to v0.23 and record Q4 audit

Author: CrispStrobe

LEARNINGS.md

@@ -11,7 +11,7 @@ If a lesson is still "live" (affects current work), it's linked from
 ---
 
 <!-- BEGIN INDEX POINTER — generated by tools/gen-learnings-index.py -->
-**Do not read this file top to bottom** — it is ~1 MB across 299 sections.
+**Do not read this file top to bottom** — it is ~1 MB across 301 sections.
 The generated topic + per-model index is [`docs/LEARNINGS-INDEX.md`](docs/LEARNINGS-INDEX.md).
 Find the lesson there, then jump with `grep -n "<heading text>" LEARNINGS.md`.
 Regenerate both with `python tools/gen-learnings-index.py` after adding a section.
@@ -18833,3 +18833,59 @@ and fail at zero.
 Transferable: any decoder that resamples, upsamples, or expands attacker-
 controlled input needs an amplification bound, and `|| true` on a `cp` of
 required inputs is a silent pass wearing a gate's clothes.
+
+## Device-side argmax can cost more than the readback it removes, and an unused graph output still runs (#81, 2026-09-07)
+
+The TDT decoder read a full 8k-token joint-logit row back to the CPU at every
+greedy step, so replacing that transfer with two in-graph argmax results looked
+obviously favorable. A second arm precomputed 4 or 8 encoder frames against the
+same predictor state and scanned duration-skipping blanks on the host.
+
+The P100 Q4 A/B rejected both ideas. All arms were stable and transcript-exact,
+but scalar device selection changed the 134 s clip from 1.2062 s to 1.2042 s
+(0.17%, noise-sized). Batch 4 took 1.2369 s and batch 8 took 1.2700 s; combining
+them with device selection took 1.2284 s and 1.3044 s. Decode-stage medians were
+303.1, 298.8, 326.7, 364.7, 321.7 and 365.8 ms respectively. Duration skipping
+makes speculative batches do work for frames the scalar decoder never visits,
+and contiguous materialization of vocabulary/duration slices adds kernels before
+CUDA argmax can consume them.
+
+There was also a measurement trap: the supposed baseline graph contained the
+new argmax tensors as outputs even when the environment switch was off. ggml
+executes every graph output; ignoring the returned integers does not remove their
+kernels. An opt-in experiment must gate graph construction itself, or use a
+separate graph, if its baseline is meant to represent shipping code. We removed
+the experimental graph and direct-convolution arms after measurement rather than
+leaving dormant nodes in every decoder session.
+
+
+The encoder matrix rejected three similarly plausible shortcuts. ggml's direct
+standard and depthwise CUDA convolutions were 3.3% and 25.8% slower than the
+existing lowering, while the fork's per-head flash-attention path was 57.2%
+slower on sm_60. Selectively preserving FFN tensors at Q8 matched speed but
+changed the transcript; F16 changed it and lost 3.8%. Backend support is not a
+performance result, and a higher-precision intermediate is not automatically a
+safe quantization exception. Keep each lever isolated until both timing and
+end-to-end text parity pass.
+
+## An upstream-sync PR must target the branch consumers pin, and cache export belongs after the build (#81, 2026-09-07)
+
+The ggml v0.23 consolidation initially targeted an older sync branch. Its tree
+contained current upstream, but merging that PR would not have updated the
+`crispstrobe-ops` default branch that CrispASR pins, so GitHub correctly kept
+reporting the fork as 15 commits ahead and 747 behind. Retargeting exposed four
+default-only commits. Each was a backport of a fix already present in the sync
+line; after verifying those original commits were ancestors of the candidate,
+we merged the default history and resolved only the duplicate hunks. The merged
+default branch is 43 commits ahead and 0 behind upstream. Validate ancestry
+against the actual consumer branch, not merely the candidate's upstream merge
+base.
+
+The first model-level A/B then spent 48 minutes successfully compiling both
+trees and failed because a raw Python string passed literal `\\n` characters
+to an embedded child. Worse, the cache export was after the benchmark, so the
+successful build could not refresh the seed. Compile embedded programs locally
+before submission, and export the build cache immediately after a successful
+build as well as after final validation. The corrected P100 run was
+transcript-exact and made v0.23 0.53% faster on the 134 s clip; its 7,927-file
+cache became the next account-matched dataset version.

PERFORMANCE.md

@@ -882,6 +882,56 @@ needs a per-stage look).
 | onnx-asr parakeet-ctc CUDA fp32, 134 s varied speech, in-process | **214×** (tdt 121×) |
 | onnx-asr parakeet-ctc CPU int8, 134 s varied | 5.8× (tdt 5.7×) |
 
+**Q4 round-2 rejected arms (P100, 2026-09-07; Kaggle
+`chr1str/crispasr-issue-81-q4-round-2`, encoder matrix v1 at `b82b7baf`,
+TDT matrix v3 at `c0527382`).** Device-side
+selection preserved exact transcripts but improved the experimental baseline by
+only 0.17% on the 134 s clip. Speculative joint batches lost 2.5–8.1%. The
+selection graph also made its argmax outputs part of the graph when the runtime
+switch was off, so that arm was removed rather than retained as a dormant option.
+
+| arm | 134 s median | x-RT | vs same-run baseline |
+|---|---:|---:|---:|
+| selection-graph baseline | 1.2062 s | 111.15× | — |
+| device argmax | 1.2042 s | 111.34× | +0.17% |
+| batch 4 | 1.2369 s | 108.39× | −2.48% |
+| batch 8 | 1.2700 s | 105.57× | −5.02% |
+| batch 4 + device argmax | 1.2284 s | 109.14× | −1.81% |
+| batch 8 + device argmax | 1.3044 s | 102.79× | −7.52% |
+
+The preceding CTC matrix rejected the other proposed Q4 levers on the same
+134 s varied clip. Direct standard/depthwise convolution reduced throughput,
+and enabling the fork's per-head flash-attention path on sm_60 was more than
+2× slower. Keeping selected FFN tensors at Q8 or F16 either failed transcript
+parity or lost speed.
+
+| CTC arm | x-RT | result |
+|---|---:|---|
+| baseline | **163.09×** | exact/stable |
+| direct initial conv | 157.78× | −3.3% |
+| direct depthwise conv | 120.94× | −25.8% |
+| both direct convs | 117.17× | −28.2% |
+| per-head flash attention | 69.87× | −57.2% |
+| flash + both direct convs | 60.11× | −63.1% |
+| selected FFN Q8 | 163.06× | transcript changed |
+| selected FFN F16 | 156.95× | transcript changed and −3.8% |
+
+**ggml v0.23 fork consolidation (P100 sm_60, 2026-09-07; Kaggle
+`chr1s4/crispasr-ggml-v0-23-q4-a-b` v3).** The old fork pin and merged
+v0.23 runtime used separate source-compatible CrispASR checkouts, the same Q4
+Parakeet TDT model and audio, and an account-matched warm ccache dataset.
+Transcripts were stable and byte-identical. v0.23 was neutral-to-faster:
+
+| clip | old fork | ggml v0.23 | change |
+|---|---:|---:|---:|
+| JFK, 11.00 s | 85.44× | **87.44×** | +2.34% |
+| varied speech, 134.07 s | 115.38× | **115.99×** | +0.53% |
+
+The benchmark exercised ggml runtime commit `069a517d`; the merged default
+commit `2dd13edd` has the same runtime source tree (later commits only add
+patch guards, CI, and merge ancestry). Kaggle validation passed and the
+`chr1s4/crispasr-ccache` seed was refreshed to version 13.
+
 CUDA rows resolved (2026-07-12, kernel `issue81-onnx-bench` v16, real
 134 s varied LibriSpeech, load-excluded, 301-word proof-of-work,
 same-run onnx head-to-head): the honest-methodology re-run with the

PLAN.md

@@ -1,5 +1,21 @@
 # CrispASR — Pending work
 
+## DONE 2026-09-07 — #81 Q4 CUDA optimization round 2
+
+Worktree `.claude/worktrees/perf-81-round2`, branch `perf/81-round2`. Audit the
+CrispStrobe ggml fork against current upstream for convolution, per-head flash
+attention, CUDA graph and reduction improvements. Benchmark direct subsampler
+convolution, adaptive GPU TDT batching, device-side selection, per-head flash
+attention and selective hot-tensor precision as isolated Q4 arms. Preserve
+exact transcript parity, measure P100 through the Kaggle harness, and keep
+Ada/RTX 4070 conclusions explicitly separate from sm_60 evidence.
+
+All isolated runtime arms were rejected; see `PERFORMANCE.md`. The ggml fork
+was consolidated onto v0.23.0 through CrispStrobe/ggml#3, finished 0 commits
+behind upstream, passed hosted CPU/Metal/Vulkan/CUDA validation, and improved
+the same-workload P100 Q4 result by 0.53% on the 134 s clip. CrispASR now pins
+the merged default-branch commit.
+
 ## DONE 2026-09-07 — #81 Q4 FastConformer profiling
 
 The P100 profile found TDT's encoder-to-joint projection consuming about

docs/LEARNINGS-INDEX.md

@@ -17,7 +17,7 @@ per-backend porting detail (Per-model port notes + the family cross-reference).
 Reach for the topic groups when you are stuck on a *technique*, and the
 cross-reference when you already know which model you are touching.
 
-## Index by topic (299 lessons)
+## Index by topic (301 lessons)
 
 **Security & untrusted input** (2)
 
@@ -34,7 +34,7 @@ cross-reference when you already know which model you are touching.
 - L16962 — consent record — BIND to the audio, don't chain the log
 - L18257 — watermark detector — swap the STATISTIC, not the threshold
 
-**Build, CI, release & packaging** (21)
+**Build, CI, release & packaging** (22)
 
 - L637 — A packaging step that destroys the state a later step reads — and scripts that only run in a release job
 - L941 — A guard job that runs ONE compiler family guards one compiler family
@@ -57,6 +57,7 @@ cross-reference when you already know which model you are touching.
 - L11429 — Kaggle as a batch-rebake target: seven fragilities the script has to work around
 - L14873 — CI has no NVIDIA GPU — a CUDA-only graph path (mimo RVQ) is guarded by an on-Kaggle exact-parity smoke, not CI (#309)
 - L15700 — #313 Rust crates on crates.io via git dep
+- L18871 — An upstream-sync PR must target the branch consumers pin, and cache export belongs after the build
 
 **Multi-surface wiring — CLI / C ABI / bindings / server** (29)
 
@@ -90,7 +91,7 @@ cross-reference when you already know which model you are touching.
 - L18525 — A streaming callback over a growing buffer is buffered inference unless the public session owns every cache
 - L18550 — A managed assembly and its native library cannot share a name, and a live test that skips itself is not a test
 
-**Parity, the diff harness & ground truth** (41)
+**Parity, the diff harness & ground truth** (42)
 
 - L39 — The long-form recipe is part of the blueprint — and a comment citing the blueprint is not the blueprint
 - L169 — A model family's per-size behaviour lives in the reference RUNTIME, not the modelling code
@@ -133,6 +134,7 @@ cross-reference when you already know which model you are touching.
 - L15986 — #333 madlad400 quants + T5 parity
 - L16858 — chatterbox hift_pcm(ref_mel) cos≈0.879 was a diff-harness layout bug (source_stft fed transposed), not a…
 - L18488 — Chatterbox Multilingual V3: checkpoint names, quant hashes, and a non-silent clone are not parity
+- L18837 — Device-side argmax can cost more than the readback it removes, and an unused graph output still runs
 
 **ggml graphs, allocation & caching** (44)
 

examples/crispasr-quantize/CMakeLists.txt

@@ -14,6 +14,7 @@
 
 set(TARGET crispasr-quantize)
 add_executable(${TARGET} main.cpp)
+target_compile_features(${TARGET} PRIVATE cxx_std_17)
 
 include(DefaultTargetOptions)
 

examples/talk-llama/llama-model-loader.cpp

@@ -1042,7 +1042,7 @@ static bool weight_buft_supported(const llama_hparams& hparams, ggml_tensor* w,
         ggml_tensor* B = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, d_state, n_group, n_seq_tokens, n_seqs);
         ggml_tensor* C = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, d_state, n_group, n_seq_tokens, n_seqs);
         ggml_tensor* ids = ggml_new_tensor_1d(ctx, GGML_TYPE_I32, n_seqs);
-        op_tensor = ggml_ssm_scan(ctx, s, x, dt, w, B, C, ids);
+        op_tensor = ggml_ssm_scan(ctx, s, x, dt, w, B, C, ids, /*K=*/1);
     } break;
     case GGML_OP_RWKV_WKV6: {
         // FIXME

examples/talk-llama/models/mamba-base.cpp

@@ -113,7 +113,7 @@ ggml_tensor* llm_build_mamba_base::build_mamba_layer(llm_graph_input_rs* inp, gg
             // Custom operator to optimize the parallel associative scan
             // as described in the Annex D of the Mamba paper.
             // => {d_inner, n_seq_tokens, n_seqs} and {d_state, d_inner, n_seqs}
-            return ggml_ssm_scan(ctx, ssm, x, dt, A, B, C, ids);
+            return ggml_ssm_scan(ctx, ssm, x, dt, A, B, C, ids, /*K=*/1);
         };
 
         ggml_tensor* y_ssm = build_rs(inp, ssm_states_all, hparams.n_embd_s(), ubatch.n_seqs, get_ssm_rows);
@@ -242,7 +242,7 @@ ggml_tensor* llm_build_mamba_base::build_mamba2_layer(llm_graph_input_rs* inp, g
 
             // TODO: use semistructured matrices to implement state-space duality
             // => {d_inner, n_seq_tokens, n_seqs} and {d_state, d_inner, n_seqs}
-            return ggml_ssm_scan(ctx, ssm, x, dt, A, B, C, ids);
+            return ggml_ssm_scan(ctx, ssm, x, dt, A, B, C, ids, /*K=*/1);
         };
 
         ggml_tensor* y_ssm = build_rs(inp, ssm_states_all, hparams.n_embd_s(), ubatch.n_seqs, get_ssm_rows);

examples/talk-llama/models/plamo2.cpp

@@ -266,7 +266,7 @@ ggml_tensor* llm_build_plamo2::build_plamo2_mamba_layer(llm_graph_input_rs* inp,
             // Custom operator to optimize the parallel associative scan
             // as described in the Annex D of the Mamba paper.
             // => {d_inner, n_seq_tokens, n_seqs} and {d_state, d_inner, n_seqs}
-            return ggml_ssm_scan(ctx, ssm, x, dt, A, B, C, ids);
+            return ggml_ssm_scan(ctx, ssm, x, dt, A, B, C, ids, /*K=*/1);
         };
 
         ggml_tensor* y_ssm = build_rs(inp, ssm_states_all, hparams.n_embd_s(), ubatch.n_seqs, get_ssm_rows);

ggml

@@ -852,7 +852,7 @@ extern "C" struct silero_lid_context* silero_lid_init(const char* gguf_path, int
     }
 
     if (!lid_load(ctx->model, gguf_path, ctx->backend, ctx->backend_cpu)) {
-        delete ctx;
+        silero_lid_free(ctx);
         return nullptr;
     }