Skip to content

Add key digest arrays to BTree nodes (format 1.1) - #30

Merged
hadashiA merged 4 commits into
mainfrom
optimize-key-digests
Jul 13, 2026
Merged

Add key digest arrays to BTree nodes (format 1.1)#30
hadashiA merged 4 commits into
mainfrom
optimize-key-digests

Conversation

@hadashiA

Copy link
Copy Markdown
Owner

Summary

B+Tree nodes now carry a contiguous array of 8-byte order-preserving key digests, and searches probe that array instead of dereferencing the variable-length keys scattered across the page. One cache line covers eight probes, and the per-probe two-step dependent load (meta → key) collapses to a single load.

Case Before After
Point lookup, 10k rows (fixed key) 24.8 µs 18.1 µs -27%
Point lookup, 10k rows (random keys) 43.4 µs 33.4 µs -23%
Point lookup, 1M rows (fixed key) 59.7 µs 53.9 µs -10%
Point lookup, 1M rows (random keys) 88.2 µs 87.7 µs neutral
Range scan, 1M rows 45.9 µs 47.8 µs neutral

(1 op = 1000 queries. The 1M cases are DRAM-latency bound, where fewer touched lines and a ~20% larger page count cancel out; the win is in the CPU/cache-bound regime.)

Digest definitions

IKeyEncoding gains SupportsKeyDigest / GetKeyDigest (default interface members, so custom encodings are unaffected):

  • Int64LittleEndian — the value with its sign bit flipped: an exact total order, so searches never touch key bytes at all.
  • Ascii — first 8 key bytes packed big-endian (zero-padded): byte-lexicographic order maps onto unsigned comparison; keys sharing an 8-byte prefix collide and fall back to the full comparison.
  • DuplicateKeyEncoding (non-unique secondary indexes) — delegates to the source encoding's digest of the key without the rid suffix.
  • UUIDv7 / custom encodings — no digest (Guid.CompareTo is not byte-lexicographic); their trees keep the plain layout automatically.

Format & compatibility

  • Page layout (digest pages): [PageHeader][NodeHeader][digest u64 × N][meta × N][payload].
  • The flag lives in the upper bits of the node header's kind field (NodeFlags.HasKeyDigests), per page — digest and plain trees coexist in one file, and 1.0 files remain fully readable.
  • DatabaseBuilder.KeyDigests (default true) — disable to produce byte-identical 1.0 output (header stays MinorVersion 0) at the cost of the search speedup. File-size cost when enabled: +8 bytes/entry (~+27% for this benchmark's 13-byte values; proportionally less for larger values).
  • Files built with digests are marked MinorVersion 1. Note: published 1.0 readers do not validate the version and cannot parse digest pages — worth a release-note callout.

Also

  • Fixed a latent bound-search bug (exists on main): Search/SearchAsync with LowerBound/UpperBound returned not-found when the bound was the first entry of the next leaf; they now continue to the right sibling. The new page layout shifted leaf boundaries and exposed this in existing range tests.
  • New benchmark: BigReadBenchmark (1M rows, ~40 MB) covering the DRAM-bound regime.
  • New tests: digest-collision fallback (keys sharing an 8-byte prefix) and the KeyDigests = false layout.

All 75 tests pass.

🤖 Generated with Claude Code

@hadashiA
hadashiA merged commit d9c9a4e into main Jul 13, 2026
1 check passed
@hadashiA
hadashiA deleted the optimize-key-digests branch July 13, 2026 04:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant