Storage capacity, admission control & tight-space compaction

[polaz]

Epic / tracker. Give the engine first-class storage-capacity awareness: introspect what is stored and how much more fits, gate writes (and compaction) before running out of space without ever breaking an SST, and add an opt-in tight-space compaction mode for embedded / blockchain deployments where the disk is small and "just provision more" is not an option.

Why

Today the engine has no disk-space or quota awareness: no way to ask "how full am I / how much more fits", no admission control, no read-only-on-full, no space-aware compaction. An out-of-space write fails mid-flush; the SST is never corrupted (atomic commit: a partial file is an unreferenced orphan), but liveness breaks (the memtable cannot evict). For small-disk deployments there is also no way to compact when nearly full.

Design invariants

• Never start an SST write that cannot finish. Admission control gates before touching disk, so the on-disk format stays intact.
• read-only is a computed predicate, not a latched bit: used + reserved > effective_limit, re-evaluated live. Raising the quota (runtime config), freeing disk, or a compaction reclaiming space clears it automatically on the next check — no sticky state to unstick.
• No deadlock when full: user writes are gated, but space-reclaiming compaction (Drop / Move / shrinking Merge) keeps an always-available emergency reserve, so the engine can always free space.
• Tight-space mode is opt-in. The default path stays simple (separate output file, atomic commit, no journal).

Sequence

1. Storage introspection API — stats, average K/V shape, remaining-capacity estimate, StorageStatus.
2. Storage quota + read-only admission control (computed predicate, reserved headroom).
3. Fs::available_space disk-free probe; effective_limit = min(quota, disk_free + used).
4. Compaction space admission (deadlock-free: Drop/Move priority, emergency reserve).
5. Tight-space opt-in resumable incremental-reclaim compaction (punch-hole consumed input + progress journal + resume).

Compaction throttling already exists (Config::compaction_rate_limit / RateLimiter); the tight mode reuses it so reads degrade but never stop during a slow reclaim.

Child issues are linked below as they are filed.

Child issues

1. Storage introspection: capacity, average K/V shape, remaining-capacity estimate #483 — Storage introspection: capacity, average K/V shape, remaining-capacity estimate
2. Storage quota and read-only admission control #484 — Storage quota and read-only admission control
3. Fs::available_space — filesystem free-space probe #485 — Fs::available_space filesystem free-space probe
4. Compaction space admission (deadlock-free) #486 — Compaction space admission (deadlock-free)
5. Tight-space resumable incremental-reclaim compaction (opt-in) #487 — Tight-space resumable incremental-reclaim compaction (opt-in)

[coderabbitai]

🔗 Related PRs

#163 - feat(config): per-level Fs routing for tiered storage [merged]
#308 - perf(table/scanner): bump compaction readahead 32 KiB → 2 MiB (#133 Phase 1c) [merged]
#389 - perf(compaction): I/O rate limiter (leaky token bucket) [merged]
#402 - feat(compaction): parallel sub-compaction across key ranges [merged]
#467 - docs(compaction): clarify major_compact watermark + clean subcompaction profile [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!