Roadmap: coordinode-lsm-tree performance & structured-zstd integration

[polaz]

Status: V5 released (v5.0.0 to v5.2.1)

The full V5 integrity stack shipped on crates.io. v5.0.0 (PR #272) landed the format-changing batch (manifest hardening, per-KV protection, per-block seqno bounds); v5.1.0 (PR #432) and the v5.2.x line followed with post-V5 perf, integrity, and no-std work. The four-axis block model (compression x encryption x ECC) is live and round-trips across all 8 combinations.

Cross-cutting design discipline still binds every format-changing PR:

• Benchmark Symmetry Invariant: discipline for new format features (cross-cutting design rule) #353 Benchmark Symmetry Invariant: every new format feature must be off-by-default OR match a RocksDB baseline OR provide an explicit OFF mode; compare-rocksdb ships a RocksDbParity preset.

Open: Encryption / ECC track

Three independent, orthogonal axes (compression, encryption, ECC), each on/off; BlockTransform models all 8 combinations and every combination must round-trip.

Encryption axis = AAD-bound, LIVE (done #413 via PR #423). The live block path seals through AAD-bound encrypt_block/decrypt_block; block-swap / cross-table / codec-relabel / epoch+suite downgrade are caught (AEAD-verify failure). AAD binds table_id + codec context (compression_type, dict_id, window_log) + block_flags; block byte offset and owning tree id are deliberately NOT bound. Wire format is exactly MetadataFrame || BodyFrame.

ECC axis = Page ECC (#267 / crate::ecc / BlockTransform), configurable scheme (XOR / Reed-Solomon, done #254 via PR #414), opt-in / off by default. Correction is over the on-disk compressed+encrypted bytes, verified by the block's XXH3-128 checksum before decompress. The SEC-DED single-bit fast path (done #255 via PR #437) heals 1-bit flips ahead of RS recovery. Auto-heal (done #411 via PR #446) reschedules a compaction rewrite after a parity-corrected read; patrol scrub (done #412 via PR #456) sweeps for latent errors proactively.

Order	Issue	What	Blocked by	Est
E2	🟡 #253	test: AAD threat-model regression suite. Runs against the live AAD path (done #413); inner-frame dict-id gate landed (PR #441). Remaining: dict-content substitution, decompression-bomb window swap, codec-version drift (need structured-zstd integration).	done #413, done #251	2d
E6	🟡 #256	forensics CLI: dump encrypted block structure without key. dump-block + reconstruct-AAD shipped (PR #444 / #445); remaining: inner-block listing for non-encrypted, Page ECC trailer report, mixed-suite integration corpus.	done #251, done #413	1d
E7	🟡 #257	partial-decode for range queries (query-path; independent of ECC). Adaptive partial-tier + promotion + true resume shipped via PR #415, reaches parity. Opt-in behind LSM_PARTIAL_DECODE; open pending default-enable decision / remaining hardening.	done structured-zstd resume API	2d

Open: Critical Path (format / API contract)

Order	Issue	What	Type	Blocked by	Est
CP-C	🟡 #144	perf: compaction parallelism. Phase 1 (parallel block compression, PR #401) + Phase 2 (sub-compaction split, PR #402) implemented. The headline subcompaction gap (#410) was a bench artifact (PR #427); honest baseline shows ours competitive. Residual ~1.27x range-split is the remaining tuning target.	Performance, no on-disk break	tuning #410	tuning open
CP-D	#274	no-std + alloc migration: full crate. Core engine now compiles + runs on the no-std-only thumbv7em-none-eabihf target (--features alloc, 0 errors) over caller-injected Fs/Clock traits (done #449). Remaining: vendor byteview strategy (+ SIMD opportunity) and the broader surface.	API surface (cfg-gated)	core landed via done #449	epic

Open: no-std track

Issue	What	Note
#358	gate the manifest+version+tree+checkpoint std-bound layer	Superseded by #449: the chosen strategy un-gates the engine onto traits rather than gating leaf modules (gating cascaded into 1200+ unresolved-module errors). Core no-std landed; close-candidate.
#346	io_uring backend on raw Linux syscalls (no_std + alloc)	Consumer of #274, needs the no-std trait surface. Pure-Rust io_uring without tokio-uring / std-bound io-uring.

Open: Performance backlog

Issue	What	Note
#410	subcompaction far slower than RocksDB (3x-13x in compare-rocksdb)	Headline gap was a bench artifact: RocksDB's manual compaction skips the bottommost rewrite (kIfHaveCompactionFilter) while ours forces it. Honest L3 + 4-thread baseline (PR #427) shows ours winning major_compact; only the ~1.27x range-split subcompaction is a real target. Gates the #144 epic.
#426	bench: blob_tree (KV-separation) vs surrealkv + attribute read-gap source	Diagnostic bench, no format/API change.

Cross-cutting design discipline

Issue	What	Status
#353	Benchmark Symmetry Invariant: every new format feature OFF-by-default OR match a RocksDB baseline OR explicit OFF mode; compare-rocksdb ships a RocksDbParity preset	RocksDbParity / LsmTreeDefault / LsmTreeParanoid presets (via LSM_BENCH_PRESET) + docs/BENCHMARKING.md + CONTRIBUTING cross-link implemented (PR pending). Binds all format-changing PRs.

Recently merged (since last actualization)

Release

• v5.2.1 (PR chore: release v5.2.1 #452), v5.1.0 (PR chore: release v5.1.0 #432), v5.0.0 (PR chore: release v5.0.0 #272) on crates.io: full V5 integrity stack + post-V5 perf / integrity / no-std work.

no-std

• done no-std: full alloc engine over caller-injected Fs/Clock traits #449 full alloc engine over caller-injected Fs/Clock traits: Clock trait (injectable wall-clock) + no-std-check promoted to a hard CI gate; core engine compiles + runs on thumbv7em-none-eabihf (alloc, 0 errors). Merged via PR feat(time): injectable Clock trait + no-std-check as a required gate #460.
• done Replace quick_cache with an in-tree no_std-capable block cache #429 replace quick_cache with an in-tree sharded S3-FIFO block cache (no_std-clean, lock-free reads). Merged via PR feat(cache): replace quick_cache with in-tree sharded S3-FIFO cache (#429) #448.

Encryption / ECC

• done feat(encryption): wire AAD-bound block encryption into the live block path #413 AAD-bound encryption activated on the live block path: BlockTransform encode/decode seal via encrypt_block/decrypt_block. Merged via PR feat(encryption): activate AAD-bound block encryption on the live path #423.
• done feat(encryption): patrol scrub — background latent-error sweep #412 ECC patrol scrub: background latent-error sweep (leader-only, rate-limited). Merged via PR feat(ecc): patrol scrub for proactive latent-error correction #456.
• done feat(metrics): unified ECC-recovery counters (RS + SEC-DED) #436 unified ECC-recovery counters (RS + SEC-DED) + wired tree-level SEC-DED reads. Merged via PR feat(metrics): unified ECC-recovery counters, wire SEC-DED reads #457.
• done feat(integrity): configurable ECC-at-rest overhead on the Page ECC path #254 configurable ECC-at-rest scheme (XOR / Reed-Solomon) on the Page ECC path: ecc_scheme in RuntimeConfig/manifest, off by default. Merged via PR feat(ecc): configurable ECC-at-rest scheme (XOR / Reed-Solomon) on the Page ECC path #414.
• done feat(integrity): SECDED single-bit fast path on the Page ECC read path #255 SEC-DED single-bit fast path on the Page ECC read path: cheap 1-bit heal before RS try_recover. Merged via PR feat(ecc): SEC-DED single-bit fast path on the Page ECC read path #437.
• done feat(encryption): ECC auto-heal — correct-on-read then schedule rewrite #411 ECC auto-heal: correct-on-read then schedule a compaction rewrite; self-heal also fires on the partial-decode read path. Merged via PR feat(ecc): self-heal SSTs after a parity-corrected read #446.
• done feat(encryption): wire-format encoder/decoder for AAD-bound block (impl of #250) #251 AAD-bound block wire-format encoder/decoder + KeyChain. PRs feat(encryption): top-level encrypt_block/decrypt_block + KeyChain + wire format (#251) #344 / feat(encryption): AEAD dispatch with AES-256-GCM + ChaCha20-Poly1305 (#251 PR2) #338 / feat(encryption): AAD construction + decode-time error types (#251 foundation) #336.
• done Feature: Per-block Reed-Solomon Page ECC (build-time feature gate) #267 Per-block Reed-Solomon Page ECC. Merged via PR feat(ecc): per-block Reed-Solomon Page ECC (#267) #343.
• done test(encryption): regression suite for AAD threat model (per #250) #253 first wave: AAD threat-model regression suite (23 tests, PR test(encryption): AAD threat-model regression suite (first wave) #361); inner-frame dict-id gate (PR feat(compression): pin inner-frame dict id as configurable-checksum defense-in-depth #441). Issue stays 🟡 for the structured-zstd-dependent scenarios.

V5 format batch (all landed)

• done fix(table): sfa trailer / footer redundancy (single-bit-flip safeguard) #297 V5-2 Manifest hardening: Blocks-based framing (XXH3-128 + optional Page ECC + optional AEAD per Block), 4 KiB head-mirror, canonical CURRENT digest. Merged via PR feat(manifest): Blocks-based manifest hardening (#297) #357.
• done feat(integrity): per-KV protection for memtable / WriteBatch / BlockBuilder (RAM bit-flip detection) #298 V5-3 Per-KV protection: DataKvChecked / FilterKvChecked block variants, conditional memtable layout, configurable per-scope ChecksumAlgorithm, flag-based ECC parity. Merged via PR feat(integrity)!: per-KV checksum protection + flag-based ECC parity (V5 block header) #369.
• done feat: scan_since_seqno() — iterate entries modified after a given sequence number #224 V5-4 scan_since_seqno(): per-block seqno bounds in index entries (delta-varint, PR feat(index): per-block seqno bounds in SST index entries #371) + CDC event stream API + synchronous clear() reclaim (PR feat(scan): scan_since_seqno CDC stream + synchronous clear() reclaim #433).
• done feat(config): runtime-toggleable Tree::update_runtime_config foundation #352 V5-1 RuntimeConfig foundation: Tree::update_runtime_config / Tree::runtime_config, ArcSwap snapshot, ChecksumAlgorithm enum. Merged via PR feat(config): runtime-toggleable RuntimeConfig foundation (#352) #355.

Integrity / verification

• done feat(integrity): online VerifyChecksum / VerifyFileChecksums APIs #300 online VerifyChecksum / VerifyFileChecksums: verify_checksum scrubber with parallelism + throttle (PR feat(verify): verify_checksum scrubber with parallelism + throttle #435), throttle fix (PR fix(verify): don't sleep the scrub throttle after the last SST #443). FS-aware dispatch via done feat(fs): Fs trait capability framework — FS-aware integrity + CoW + reflink optimizations #354.
• done feat(fs): Fs trait capability framework — FS-aware integrity + CoW + reflink optimizations #354 Fs trait capability framework: Fs::capabilities() / try_disable_cow() / reflink_file(); auto-disable Btrfs CoW on SSTs, reflink for checkpoints. Merged via PR feat(fs): filesystem-aware capability framework (CoW-disable + reflink) #428.

Manifest / recovery

• done feat(recovery): cross-process directory lock for exclusive tree access #422 cross-process directory lock for exclusive tree access: open/repair acquire an advisory LOCK; a second opener fails fast with Error::Locked. Merged via PR feat(recovery): cross-process directory lock for exclusive tree access #458.
• done feat(tooling): repair_db support for KV-separated (blob) trees #408 repair_db for KV-separated (blob) trees: Config::repair rebuilds blob-tree manifests by scanning blobs/, follow-up to done feat(tooling): repair_db — rebuild MANIFEST from SST files (disaster recovery) #303. Merged via PR feat(tooling): repair KV-separated (blob) trees #459.
• done perf(manifest): incremental VersionEdit append-log instead of full rewrite per flush #418 incremental manifest: append-only edit log replacing full-rewrite-per-flush (~31%/33% faster on flush-heavy paths, PR feat(manifest): incremental manifest via append-only edit log #421); torn edit-log tail rejected under AbsoluteConsistency (PR fix(manifest): reject torn edit-log tail under AbsoluteConsistency #438).
• done feat(tooling): repair_db — rebuild MANIFEST from SST files (disaster recovery) #303 repair_db: Config::repair() rebuilds a missing/corrupt MANIFEST from on-disk SSTs. Merged via PR feat(tooling): repair_db — rebuild MANIFEST from on-disk SSTs #409.
• done feat(version): ManifestRecoveryMode — PointInTimeRecovery + SkipAnyCorruptedRecords + README docs (follow-up to #299) #323 ManifestRecoveryMode PointInTime + SkipAny + per-record framing. PR feat(version): PointInTimeRecovery + SkipAnyCorruptedRecords + per-record framing (#323) #342.
• done feat(version): MANIFEST recovery mode policies for corrupted recovery state #299 ManifestRecoveryMode core (AbsoluteConsistency + TolerateCorruptedTailRecords). PR feat(config): ManifestRecoveryMode + TolerateCorruptedTailRecords (#299) #317.

Query / compaction / perf

• done perf(table): single-level index for small SSTs (size-based index partitioning) #396 size-adaptive block index: single-level index for small SSTs. Merged via PR perf(table): size-adaptive block index — single-level for small SSTs #397.
• done perf(table): cache-managed single-level index + raise spill threshold #398 raise index spill threshold to 4 MiB (cache-managed single-level index). Merged via PR perf(table): raise index spill threshold to 4 MiB #399.
• done perf(query): partial-decode for range queries using structured-zstd block subset API #257 partial-decode (impl landed, opt-in behind LSM_PARTIAL_DECODE). PR feat(query): cold-block partial decode with resumable zstd #415.
• done perf(compression): reuse one zstd compressor across blocks via compress_independent_frame #405 reuse one zstd compressor across blocks via compress_independent_frame + encoder-side EncoderDictionary reuse (PR feat(tooling): repair_db — rebuild MANIFEST from on-disk SSTs #409). zstd:22 write/overwrite/major_compact ~2x faster; zstd:1 ~30% faster.
• done perf(compaction): evaluate enabling bottommost seqno-zeroing (zero_seqnos at last level) #406 compaction-time range tombstones + bottommost seqno-zeroing. PR feat(compaction): compaction-time range tombstones + bottommost seqno-zeroing #407.
• done perf(compression): pass source-size hint on zstd block compression (L22 ~34x slower without it) #403 source-size hint on zstd block compression (L22 ~34x slower without it). Merged.
• done perf(memtable): geometric arena chunks to avoid 64 MiB zeroing per flush #416 geometric arena chunks (no 64 MiB zeroing per flush, PR perf(memtable): geometric arena chunks (no 64 MiB zeroing per flush) #417) + done perf(memtable): arena decode hot-path regressed fillseq/reads ~33% (#417 follow-up) #419 cheap fixed-block decode (PR perf(memtable): stop zeroing the arena, restore cheap fixed-block decode #420): fixes the overwrite@1k ~2x gap.
• done perf(read): lock-free latest-version fast path for point reads (RocksDB cached-SuperVersion) #393 lock-free latest-version fast path for point reads (cached-SuperVersion). Merged.
• done perf(read): borrow index block on point-read + parse trailer once #375 borrow index block on point-read + parse trailer once. Merged.
• done perf(compaction): account blob-relocation bytes in the I/O rate limiter #390 account blob-relocation bytes in the I/O rate limiter; done perf(io): apply POSIX_FADV_DONTNEED to cold-level (L1+) SST output #391 POSIX_FADV_DONTNEED on cold-level SST output. Merged.

FS / SIMD / build

• done refactor(fs): port Fs/FsFile traits off std::io + std::path to unblock no-std backend gating #311 Fs/FsFile traits off std::io + local crate::io module. PR feat(io): local Read/Write/Seek trait surface to lift Fs off std::io (#311) #347.
• done perf(fs): O_DIRECT foundation — AlignedBuf + FsOpenOptions::direct_io (#133 phase 2) #310 AlignedBuf + FsOpenOptions::direct_io. Merged.
• done perf(fs): configurable sync mode — plain fsync default on macOS (avoid F_FULLFSYNC) #373 configurable sync mode (plain fsync default on macOS); done perf(fs): thread SyncMode through Fs::hard_link copy fallback #377 SyncMode through hard_link copy fallback. Merged.
• done perf(table): AVX-512BW 64-byte lane for longest_shared_prefix_length #379 AVX-512BW 64-byte lane + done perf(table): extend longest_shared_prefix_length SIMD to 32-bit x86 (i686) #381 i686 32-bit SIMD for longest_shared_prefix_length. Merged.
• done build(deps): update dependencies (structured-zstd 0.0.28, bitflags, log) #383 deps (structured-zstd, bitflags, log); done compaction/leveled: relax hardcoded level_count() == 7 to config-driven check #350 config-driven level_count; done Document format-version bump policy for post-release manifest layout changes #351 format-version bump policy doc. Merged.

Tooling

• done feat(tooling): sst-dump CLI — properties / dump / index-dump / filter-stats / hex subcommands (follow-up to #301) #324 / feat(tooling): sst-dump CLI — inspect / dump / verify a single SST file #301 sst-dump CLI: all subcommands (inspect / dump / verify / properties / index-dump / filter-stats / hex / repair).
• done feat(tooling): forensics CLI — dump encrypted block structure without key #256 forensics dump-block + reconstruct-AAD. PRs feat(tooling): key-free forensic dump-block for encrypted SSTs #444 / feat(tooling): reconstruct block AAD for offline tag verification #445 (issue 🟡 for remaining increments).
• done bench(compare-rocksdb): add zstd level-22 compression matrix axis #385 / ci(benchmark): publish compare-rocksdb overlay reports to gh-pages #387 compare-rocksdb zstd-22 axis + gh-pages overlay reports; done bench(compare): add surrealkv as a third engine in compare-rocksdb #424 surrealkv as third engine. Merged.
• done test: triage ignored tests — wire up, document blockers, or remove dead ones #320 triage ignored tests; done test(verify): regression test for BlockVerifyError::DataReadError (short-read path) #315 DataReadError regression test. Merged.

[coderabbitai]

🔗 Related PRs

#229 - feat(compression): enable dictionary compression in pure Rust backend [merged]
#337 - refactor(block): collapse 4 Block I/O paths into a single BlockTransform enum (#248) [merged]
#401 - feat(compaction): parallel block compression [merged]
#409 - feat(tooling): repair_db — rebuild MANIFEST from on-disk SSTs [merged]
#421 - feat(manifest): incremental manifest via append-only edit log [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[polaz]

Roadmap actualized (2026-04-06)

Status changes

Issue	Old status	New status	Notes
#216	—	✅ done	Merged as #226, structured-zstd 0.0.7
#143	—	✅ done	Merged as #214, v4.3.0
#126	—	✅ done	upstream-monitor.yml live (Mon/Thu)

Blocker updates

• structured-zstd#25 (FastCOVER dict builder): now CLOSED → feat(compression): enable dictionary compression in pure Rust backend #218 is unblocked
• structured-zstd#72 (parallel block decompress): still OPEN → perf: read/write path optimization — batch multi_get + PinnableSlice + batch insert #143 batch decompress remains blocked
• perf(table): per-SST batch point-read for multi_get #223 (Table::batch_get): now unblocked (perf: read/write path optimization — batch multi_get + PinnableSlice + batch insert #143 dependency satisfied)

Summary

• 3/18 tasks complete, 15 open, 1 externally blocked
• ~41 working days remaining (was ~48-53)
• Added Status column and Summary table to issue body

[polaz]

Actualization — 2026-04-07

Merged: #227 (PR for #217)

Status changes

• perf(compression): cache pre-compiled Dictionary across block decompress calls #217 perf(compression): cache pre-compiled Dictionary across block decompress calls #217 → Done: ZstdDictionary caches ZSTD_DDict via Arc<OnceLock> for the C FFI backend, and FrameDecoder in thread-local storage keyed on xxh3-64 for the pure Rust backend. Per-block dictionary parsing eliminated.

Updated counts

• Phase 1: 1 → 2 done, 2 → 1 open
• Total done: 2 → 3
• Phase 1 remaining: 2d → 1d
• Total remaining: ~43d → ~42d

Next unblocked

• feat(compression): enable dictionary compression in pure Rust backend #218 (dict compression in pure backend) — was already unblocked by structured-zstd#25 closing
• All of Phase 2 (micro-opts) — fully independent