arm64: major decodeBlock performance rework

[lizthegrey]

Summary

Six incremental arm64 decoder optimizations. Commits 1–5 are all scalar (no SVE, no NEON vector regs); commit 6 delegates the long-match bulk-copy case to runtime.memmove to recover the last gap vs. the pure-Go decoder.

Commit 1 — Copy shortcut (41139b4)

Ports the "copy shortcut" optimization from decode_amd64.s. When the literal length is 0..14, there are at least 32 bytes of dst and 16 bytes of src remaining, the match offset is at least 8, the match is within the current block, and the match length is not extended, copy 16 literal + 18 match bytes as straight-line code with no bounds checks or extended-length reads. The match copy is sequenced 8+8+2 so offset == 8 works correctly. Adds one register (R20, dstend32 = dstend - 32).

Commit 2 — Splat fast path for offset 1 and 2 (998937e)

Matches with offset 1 (byte RLE, e.g. zero-padding) or offset 2 (halfword RLE) previously went through copyMatchLoop1's byte-at-a-time loop, dominated by store-to-load forwarding latency. When len >= 8 and offset is 1 or 2, splat the 1- or 2-byte pattern across a 64-bit register and store it 8 bytes at a time, falling through to the byte loop for the tail. Adds bench_rle_test.go with synthetic offset=1..4 RLE benchmarks (RLE1 and RLE2 land in this commit's wheelhouse; RLE3/RLE4 are intentional regression-guards against the pure-Go fallback's exponential-doubling loop for the small-offset path this PR does not modify in asm).

Commit 3 — Widen splat store from 8 to 16 bytes (4ce77f3)

Every arm64 core targeted by this library (Cortex-A72, Neoverse N1/V1/V2, Apple M-series) can retire an STP of two X-registers as a single 16-byte store. Widening the splat loop halves the iteration count and per-iteration loop overhead.

Commit 4 — LDP/STP fast path in shortcut match copy when offset >= 18 (f6dea62)

The 8+8+2 sequencing from commit 1 is only required when offset < 18. A histogram of ~2M matches from real-world compressed columnar data shows ~95% of matches have offset >= 18, so the LDP path is the overwhelmingly common case: four memory ops with no serial dependency chain vs. six in the 8+8+2 fallback. For offset 8..17 the sequencing is preserved.

Commit 5 — Widen copyMatchLoop8 to 16 bytes/iter for offset >= 32 (0894170)

copyMatchLoop8 has carried a comment claiming "a 16-at-a-time loop doesn't provide a further speedup". On columnar data that's no longer accurate: long matches (length >= 19, ~9% of matches by count) emit ~75% of total match-copy bytes, and ~81% of those long matches have offset >= 32. When len >= 16 and offset >= 32, copy 16 bytes/iter with LDP/STP instead of 8 with MOVD.

internal/lz4block/match_copy_test.go adds hand-assembled (offset, matchlen) matrix tests.

Note for future maintainers: CCMP's immediate is 5-bit unsigned (0..31); Go's assembler silently truncates larger values (e.g. $32 -> $0). The entry uses CCMP offset, $31 + BLS to stay in range.

Commit 6 — call runtime·memmove for long non-overlapping match copies (91e8b6f)

For match copies where offset >= len and len >= 256, jump out of the inline 16 B/iter LDP/STP loop and into runtime.memmove. Go's arm64 memmove is hand-tuned NEON (128-bit reads/writes, prefetch, software-pipelined 64 B/iter) and outruns any scalar loop that fits inline once the call-setup cost is amortized. Requires allocating a 48-byte stack frame (previously NOFRAME); NOSPLIT stays since memmove's stack use is well under the nosplit margin.

Threshold of 256 was chosen to avoid regressing Cortex-A72 (the narrow-core floor): at threshold 64 and 128, A72 regressed 5–7% on long-match columnar workloads because its in-order-ish frontend can't hide the call/spill overhead. At 256 A72 is flat. The three modern Graviton cores and Apple M-series all show net wins at 256.

Below 256 bytes and for overlapping copies (small-offset RLE), the inline scalar paths stay unchanged — memmove can't do the RLE cycling pattern, and the splat paths for offset 1/2 already cover their bandwidth ceiling.

Benchmarks

Throughput vs v4.1.26 baseline, 8 runs × 3 s × benchstat-analyzed, on five arm64 cores. G1 = Cortex-A72, G2 = Neoverse N1 (Graviton2), G3 = Neoverse V1 (Graviton3), G4 = Neoverse V2 (Graviton4), M4 = Apple M4:

Benchmark	G1	G2	G3	G4	M4
UncompressPg1661	+124%	+142%	+162%	+156%	+129%
UncompressTwain	+120%	+134%	+149%	+144%	+82%
UncompressColumnarMed	+17%	+107%	+12%	+54%	+52%
UncompressColumnarLong	flat	+79%	+21%	+49%	+50%
UncompressColumnarShort	+8%	+50%	+21%	+19%	+24%
UncompressDigits	+6%	+5%	+8%	+5%	+7%
UncompressRLE1	36×	36×	75×	72×	15×
UncompressRLE2	11×	15×	39×	36×	57×
UncompressRLE3	flat	+13%	flat	flat	flat
UncompressRLE4	flat	+15%	−16%	flat	+1%
UncompressRand	flat	flat	flat	flat	flat
geomean	+104%	+150%	+154%	+166%	+132%

On the G3 UncompressRLE4 −16% regression, and why this PR does not add PCALIGN

An early iteration of this series added a PCALIGN $64 before copyMatchTry4: on the theory that the Arm Neoverse V1 Software Optimization Guide's general advice — "align hot loop entries" / "avoid inner-loop branches crossing 32/64-byte boundaries" — would explain the RLE4 regression. It does not, and the alignment hint has been removed. Details:

• No frontend stall to fix. perf stat -e cycles,instructions,L1-icache-load-misses,stalled-cycles-frontend on BenchmarkUncompressRLE4 with the regressing binary reports <0.2% frontend-idle cycles and <200k L1-icache misses per 3 s bench. V1's frontend is not bottlenecked on this workload — the whole premise PCALIGN addresses doesn't apply.
• The hotspot is an STLF-bound store, not a branch/fetch. Line-level go tool pprof -list=decodeBlock puts 99%+ of RLE4 time in four bytes of asm — copyMatchByteLoop's load / 1-byte store / decrement / branch — with 67% of total cycles in the single-byte store alone. Classic store-to-load-forwarding dependency chain. Alignment doesn't move stores earlier in the pipeline.
• Those four instructions are byte-for-byte identical to v4.1.26. No algorithmic change touches offset < 8 long RLE. What's different is that ~200 lines of new asm now precede copyMatchByteLoop:, so its PC has shifted. V1 appears to have a PC-hashed store-buffer-bank selection; at the new PC the RLE4 store falls on a worse bank than at v4.1.26's PC. N1/V2/A72/M4 do not show the same sensitivity. This is an LSU hashing artifact, not a frontend layout artifact.
• Empirical PCALIGN tuning is darts against a wall. Each of $16, $32, $64 does move RLE4's numbers on V1 (because each shifts copyMatchByteLoop:'s PC by a different amount), but it also shifts every other downstream hot loop. PCALIGN $32 recovered RLE4 by +15% but introduced −2.4% on ColumnarLong and −2% on Digits. Each value wins on one benchmark and loses on another, with no theoretical basis for predicting which.
• The V1 Optimization Guide's advice assumes a frontend-bound workload. Following it when perf counters show no frontend stalls is applying a remedy to a diagnosis we cannot confirm. Shipping a fragile hardware-generation-specific alignment hint that contradicts measurement, for a synthetic benchmark (offset=4 RLE is ~0% of real columnar-data matches), fails a sniff test — the payoff is small and the downside is that any future change to code placement re-rolls the dice for all cores. So: no PCALIGN.

The same layout-sensitivity caveat applies in the opposite direction to the single-digit-% gains in the matrix above: when this PR is linked into a consumer binary with different surrounding code, the small-% deltas may shift by a few percent either way on V1 specifically. The big-% gains (anything above ~20%) come from genuine algorithmic bandwidth improvements and are robust to layout.

A future commit adding an offset-4 splat path (the same trick as offset 1/2) would turn the G3 RLE4 line into a big-× win on every core and close this issue algorithmically rather than through alignment hints, but it adds a branch-table entry that trades icache pressure against what is ~0% of real columnar-data matches — deferred as a follow-up.

Test plan

• go test ./... passes on every ref at every core above.
• GOOS=linux GOARCH=arm64 go build ./... clean.
• Full TestCompressUncompressBlock round-trips at every commit on every core.
• New match_copy_test.go (offset, matchlen) matrix passes at every commit.

Motivation

Observed LZ4 decodeBlock consuming roughly a third of CPU in a service decoding compressed columnar data on arm64 Lambdas. The amd64 path already had the copy shortcut; the arm64 port never got it. The splat path, both widenings, the LDP offset-split, and the memmove delegation are all either new vs. amd64 or algorithmically more aggressive, and target store-to-load-forwarding stalls and under-utilized store pipes that hurt narrow ARM cores more than wider x86 cores, plus the call-worthy runtime.memmove NEON path for bulk copies.

[lizthegrey]

Things we tried and dropped

Flagging one experiment that did not make the cut, in case there is a preference either way.

NEON VDUP + VST1 [V0.B16, V1.B16] 32-byte splat loop (on top of commit 3, guarded by len >= 32):

After `ORR`-ing the pattern into `tmp3`, splat `tmp3` across a pair of 128-bit NEON registers and store 32 bytes per iteration via a Q-pair `VST1`. Falls back to the existing 16-byte STP loop for `len < 32` and for the 0..31 byte tail.

Apple M4 benchstat vs. the 16-byte STP loop (8 runs each):

Benchmark	16B STP	32B NEON	Δ	p
RLE1	53.7 GiB/s	61.5 GiB/s	+14.5%	0.000
RLE2	53.4 GiB/s	62.1 GiB/s	+16.3%	0.000
RLE3	1.29 GiB/s	1.27 GiB/s	-1.8%	0.000
RLE4	1.61 GiB/s	1.58 GiB/s	-2.2%	0.000
Pg1661	1.99 GiB/s	1.99 GiB/s	flat	0.44
Digits	4.62 GiB/s	4.59 GiB/s	-0.8%	0.000
Twain	2.08 GiB/s	2.08 GiB/s	flat	0.88
Rand	8.74 GiB/s	8.65 GiB/s	-1.0%	0.000

The RLE1/RLE2 wins are real but smaller than the 2× the store-pipe argument predicted — M4's LSU scheduler is apparently already extracting most of the benefit from two back-to-back STP X-pair ops. The regressions on RLE3, RLE4, Digits, and Rand are all small (≤2.2%) and on workloads that never execute the splat path, so they're almost certainly code-layout/BTB effects from adding ~8 instructions to the function.

We opted not to include this in the PR: the RLE win is modest on this core, and the regressions on non-RLE paths violate the "don't regress anything" bar we set internally. We did not get a chance to measure on Neoverse V1/V2 silicon before making the call — if you think the dual-store-pipe cores might show a more favorable ratio and you'd like us to chase it, we're happy to spin up a c7g/c8g and get you that data. Otherwise, happy to leave this on the shelf.

[lizthegrey]

@greatroar tagging you here since you probably have opinions.

[lizthegrey]

Per-commit load-bearing check for reviewers: does each of the five commits unlock a distinct axis of improvement that the others don't cover? Yes -- summary below, derived from a full 6-ref sweep (baseline + each commit) across five cores (Cortex-A72, Neoverse N1/V1/V2, Apple M4), 8 runs × 3 s each, benchstat-analyzed.

commit	unique win at this transition
1 shortcut	Pg1661 +121–163% / Twain +73–149% / Digits +6–15% on all 5 cores — no other commit touches the short-lit/short-match text-compression path
2 splat	RLE1 +668–3881%, RLE2 +890–2926% on all 5 cores — zero-gain on these before splat
3 splat-widen	RLE1 +30–90%, RLE2 +31–90% on top of commit 2 — halves splat loop iteration count
4 LDP shortcut for offset≥18	ColumnarMed +6.8% on Neoverse V1, small Pg1661/Twain gains on V2, RLE4 recovery on N1 — structural 4-memop path vs 6-memop; smallest delta of the five but real on V1 columnar
5 copyMatchLoop8 widening (+PCALIGN)	ColumnarMed +20–44%, ColumnarLong +14–38%, ColumnarShort +8–31% on all 5 cores — long-match path

Commit 4 could have been squashed into commit 5 and the end-state would be identical — kept separate because each commit is thematic (one independent mechanism per commit), which is easier to review and easier to revert selectively if needed.

A few intermediate per-step transitions have mild regressions (<~1% or quickly recovered by the next commit); none survive to the final commit, where no (core × benchmark) pair has a statistically significant regression vs v4.1.26. Full numbers are in the PR description table.

[pierrec]

Amazing work. Waiting a bit for @greatroar to chime in for any comment.

[lizthegrey]

Thanks much!