Factorized columnar storage types

[antiguru]

Trie-structured factorized columnar storage for Materialize's dataflow engine, plus the full compute renderer migration onto it.

Storage layer (mz-timely-util)

Trie dedupes repeated keys/values at each level of a K → V → (Time, Diff) hierarchy.

• Level<C, Rest> recursive trie type with Vecs<C, Strides> at each level.
• FactorizedColumns<A, B, C> for (Data, Time, Diff); KVUpdates<K, V, T, R> for K → V → (Time, Diff) with tuple leaf.
• KVUpdatesRepeats variant adds Repeats on leaf columns — 44-58x serialized size reduction on low-cardinality time/diff data.
• DD integration: FactBatch/FactCursor/FactMerger/FactBuilder, FactLayout, FactValSpine/Batcher/Builder, trie-aware chunker + internal merger.
• Cursor-based iteration via Index::cursor (Cursor-based sequential iteration for Index trait frankmcsherry/columnar#105) — 10x faster Repeats iteration.
• Galloping seek (2-12x speedup on large batches).

Benchmark (100k tuples, k=100, v=1000, t=5 distinct times, +1 diffs):

Metric	Plain	Repeats
Serialized size	1.6MB	36KB
form()	42M elem/s	18M elem/s
iter (cursor, serialized)	47G elem/s	556M elem/s

Compute renderer migration

Every ArrangementFlavor::Local / JoinedFlavor::Local producer and consumer now runs on the factorized spine.
Old DatumContainer-backed RowRowSpine is deleted; RowRowSpine is now an alias for FactValSpine<Row, Row, T, R>.

Tiers:

• Tier 1 — FactLocal variant threaded through render (as_collection, flat_map, scope, enter_region, leave_region, hydration logging).
• Tier 2 — TraceBundle::oks flipped to Fact agent; peek DiffGat bound loosened from = &'a Diff to Copy + Into<Diff> for the by-value Fact cursor; index-export bridges dropped.
• Tier 3 — reduce and top_k outputs emit FactLocal via RowRowReduceBuilder; Row's Columnar::Ref becomes DatumSeq so reduce's internal RowValSpine keys unify with the Fact cursor.
• Tier 4 — threshold, linear-join pre-arrange, and delta-join lookup slot emit/consume Fact; cross-product match in linear-join collapsed.
• Tier 5 — deleted the old ArrangementFlavor::Local and JoinedFlavor::Local; renamed FactLocal → Local and FactRowRow* → RowRow*; removed dead RowRowLayout and ArrangementSize for RowRowAgent; dropped enable_compute_factorized_arrangement dyncfg.

Residual DatumContainer-backed spines: RowValSpine (reduce's internal input arrangement) and key-only RowSpine.
Error spine stays on ColumnationStack since DataflowError isn't Columnar.

Depends on frankmcsherry/columnar#105 for Index::cursor.

🤖 Generated with Claude Code

[antiguru]

Factorized columns: PR partitioning plan

58 commits, ~7650 LoC above upstream/main.
Goal: split into self-contained PRs, each compiling, testable, and reviewable independently.

Summary

PR	Title	LoC	Deps	Risk
1	timely-util: factorized columnar trie types	~1900	—	low
2	timely-util: FactBatch + FactCursor + FactMerger + FactBuilder	~1500	PR 1	low
3	timely-util: FactColumn container + FactValBatcher alias	~500	PR 2	low
4	compute: gated FactLocal arrangement flavor	~1800	PR 3	medium
5	compute: trie-aware batcher for factorized arrangements	~700	PR 4	medium
6	compute: finalize tier-5 — rename + drop dyncfg	~300	PR 5 in prod	high
7	timely-util,ore,repr: perf tuning (64 KiB, inline, extend)	~120	PR 2	low
8	timely-util: variant-C serialization + sort-by-prefix	~700	PR 5 + PR 7	medium

Eight PRs.
PR 6 is the hinge: it changes the public RowRowSpine alias, so it must wait for PR 4/5 to bake in production behind the dyncfg.

---

PR 1 — Factorized columnar trie types

Rationale: Adds a new isolated module src/timely-util/src/columnar/factorized.rs (and submodules) defining Level<L, Rest>, Lists, KVUpdates, form, iter, for_each_cursor, push_flat, plus AsBytes/FromBytes for serialization.
No consumers yet — the module is entirely orthogonal to existing code.
Bundles in the columnar crate bump needed for the API shape.

Files:

• src/timely-util/src/columnar/factorized.rs + submodules (benches, tests_prop).
• Cargo.toml — columnar git-patch pointer.
• src/ore/src/overflowing.rs, src/repr/src/row.rs, src/repr/src/timestamp.rs — AsBytes/Index/Cursor updates required by the columnar API bump.

Commits to include (oldest first):

• 87f05b5653 add factorized columnar types
• 0d5aba7213 add form(), len, clear
• 03a6b10fa9 tests + benchmark tests
• d43431729c criterion benchmarks
• fbfe4d4268 throughput + length_in_words metrics
• 0e6315dff1 KVUpdates with Repeats/Lookbacks
• 5d337ddc8d for_each_cursor
• ed08c5a317 serialized iteration benchmarks
• 0d7aa7d47c generic Level<L, Rest>
• 3a3afc6861 columnar dep pointer
• 04de82a08e update AsBytes/Index impls for columnar API
• 0850913e31, 1969290371 bench extras (sort cost, radix sort)

Roll-out: always-on new code, no dyncfg.
Test plan: new proptests pass; criterion benches run (compile only).
Risk: low — isolated module.

---

PR 2 — FactBatch + FactCursor + FactMerger + FactBuilder

Rationale: The arrangement-layer types (DD's Batch/BatchReader/Cursor/Merger/Builder).
Depends on PR 1's trie types; still has no compute consumer, so can live in-tree for a release before PR 4 turns it on.

Files:

• src/timely-util/src/columnar/factorized/batch.rs (baseline shape).
• src/timely-util/src/columnar/factorized/container.rs, layout.rs.
• src/timely-util/src/columnar/factorized/tests_prop.rs (builder/cursor/merger oracles).
• src/timely-util/benches/factorized.rs updates.

Commits:

• c43753b7d1 factorized arrangement stack
• 9414182bc8 FactValBatcher alias
• 4bbc6467e8 arrangement bench stack
• c56d43070d 1M-element benches
• dc7a1feda7 reserve+extend in stash_updates
• 5d4de4a6c7 CI lint fixes
• 758c91da0d as-to-CastFrom, sibling-file module
• 65f40b8311 galloping seek
• 1b962e9e9b Vec bounds test
• 13cacab15e Row-like benchmarks

Roll-out: always-on, no consumers.
Test plan: proptests + benches compile.
Risk: low — nothing external consumes this yet.

---

PR 3 — FactColumn container + FactValBatcher type alias

Rationale: Adds FactColumn<K,V,T,R> (the timely wire container: Typed / Bytes / Align variants), plus the FactValBatcher = MergeBatcher<Vec<...>, FactTrieChunker, FactTrieInternalMerger> type alias that consumers will use.
Splits the container from the batch to keep reviews focused; container doc defines ContainerBytes / PushInto / DrainContainer plumbing.

Files:

• src/timely-util/src/columnar/factorized/column.rs.
• src/timely-util/src/columnar/factorized/chunker.rs (FactTrieChunker).
• src/timely-util/src/columnar/factorized/batcher.rs (FactTrieInternalMerger).

Commits:

• 0764d05243 FactColumn container + batcher alias.

Roll-out: always-on.
Test plan: reuses existing proptests.
Risk: low.

---

PR 4 — compute: gated FactLocal arrangement flavor (dyncfg off by default)

Rationale: Adds ArrangementFlavor::FactLocal alongside existing Local, routes render output through it when enable_compute_factorized_arrangement dyncfg is on.
Crucially keeps the existing arrangement path intact so the default build behaves exactly like today.
This is the A/B-testable landing; production can flip the flag on a cluster at a time.

Files:

• src/compute/src/typedefs.rs — FactRowRowSpine alias.
• src/compute/src/extensions/arrange.rs — ArrangementSize impl.
• src/compute/benches/arrange_row.rs — RowRowSpine vs FactRowRowSpine bench.
• src/compute/src/render/{context.rs, reduce.rs, threshold.rs, join/**} — gated FactLocal paths.
• src/compute-types/src/dyncfgs.rs — new dyncfg.

Commits:

• d49fd3c8d3 FactRowRowSpine aliases + ArrangementSize
• d37f4c87f2 arrange_row bench
• d1763fef3f docs
• a190d80fc4 thread FactLocal through render
• ffbd8028e4 emit FactLocal under dyncfg
• bb5dad48d2 columnar wire format for FactLocal
• 4ee97c7c2e threshold wiring
• 4279de2ebc linear_join pre-arrange
• eaa417902e docs
• 61d65d0bb9 Row Columnar Ref as DatumSeq
• bf0310b580 reduce output to FactLocal
• 90404d72d8 index export / delta-join bridge (flip flag on for integration tests)
• ed75ae8d35 peek DiffGat bound
• 8d41982fbb flip TraceBundle/delta-join/logging to Fact trace

Drop from this PR: 3fbafbbe85 wip direct swap + 35b3cdff3a its revert (wasted cycles; the final rename lands in PR 6).

Roll-out: dyncfg enable_compute_factorized_arrangement defaulting to false.
Test plan: existing SLTs pass with dyncfg off; targeted SLTs opt-in with dyncfg on including advent-of-code/2023/aoc_1204.slt --auto-index-selects.
Risk: medium — changes render-layer plumbing even in the off path.

---

PR 5 — compute: trie-aware batcher for factorized arrangements

Rationale: Currently the RowRowBatcher is a vanilla DD MergeBatcher<Vec<...>, ContainerChunker, VecInternalMerger> — flat tuples pass through chunker and all merge passes, only deduping at FactBuilder::done.
This PR swaps to MergeBatcher<..., FactTrieChunker, FactTrieInternalMerger> so dedup happens early in the pipeline.
Structurally important for the memory regression (OrderBy +57.5% RSS etc.) recorded in the tier-5 memo.

Files:

• src/timely-util/src/columnar/factorized/chunker.rs — FactTrieChunker.
• src/timely-util/src/columnar/factorized/batcher.rs — FactTrieInternalMerger, TrieMergeBuilder, ChainCursor.
• src/timely-util/src/columnar/factorized/tests_prop.rs — new proptests.

Commits:

• 8a8b8d930e trie-aware batcher for factorized arrangements.

Roll-out: still gated by the PR 4 dyncfg.
Test plan: proptests + arrange_row large-config microbench improves; aoc_1204 passes with dyncfg on.
Risk: medium — batcher correctness matters; mitigated by proptests.

---

PR 6 — compute: finalize tier-5 (rename FactRowRow* → RowRow*, drop dyncfg)

Rationale: Once PR 4 + PR 5 have baked in production with the dyncfg defaulted to true for at least one release, drop the dyncfg, rename FactRowRowSpine / FactRowRowBatcher back to RowRowSpine / RowRowBatcher, and remove the old DatumContainer-keyed RowRowLayout.

Files:

• src/compute/src/typedefs.rs — alias rename, remove old.
• src/compute/src/render/context.rs — ArrangementFlavor::FactLocal → Local; drop the old variant.
• src/compute/src/render/{reduce.rs, top_k.rs, threshold.rs, join/**} — consumer catch-up.
• src/compute-types/src/dyncfgs.rs — remove the dyncfg.

Commits:

• 8a8de48299 rename FactRowRow* → RowRow*
• 2f7bd04a30 migrate top_k
• 04720baf25 rename ArrangementFlavor::FactLocal → Local
• 7a4074ac33 fmt catch-up
• 3f5b459b69 drop dyncfg
• ba88e9fa8d style
• 043b1018bf docs close-out

Roll-out: always-on after landing — cannot be behind a flag (it IS the removal of the flag).
Gate on: PR 4 + PR 5 must have enable_compute_factorized_arrangement=true as the default in the prior release and observed healthy.
Test plan: full SLT + feature-benchmark suite; compare memory numbers against baseline.
Risk: high — public type alias change, widely used.

---

PR 7 — Perf tuning: 64 KiB chunk target, inline hints, extend overrides

Rationale: Small, local perf improvements that don't change semantics: shrink chunk-target byte budget (limits per-batcher RAM for workloads with many arrangements), add #[inline] hints on hot cursor/merger methods, override Push::extend on Overflows + Timestamps so Vec::extend bulk path runs.

Files:

• src/timely-util/src/columnar/factorized/batcher.rs — TARGET_BYTES = 64 KiB, inline hints, two-pass extend in seal_val_from_staging.
• src/timely-util/src/columnar/factorized/chunker.rs — pending_flush_target = 64 KiB.
• src/timely-util/src/columnar/factorized/batch.rs — inline hints on seek_key, copy_key, stash_updates.
• src/ore/src/overflowing.rs — Push<&Overflowing<T>>::extend override.
• src/repr/src/timestamp.rs — Push<D>::extend override on Timestamps.

Commits:

• f10afdabaa 64 KiB chunk target + inline hints
• 83313f42f4 Overflows extend override + batcher use
• e4a764671c Timestamps extend override

Also drop from this PR (noise): 4c896448d8 byte-based chunk target (2 MiB) is superseded by f10afdabaa; squash f10afdabaa onto trie-batcher PR 5 directly, keeping 64 KiB from the start.

Roll-out: always-on.
Test plan: 56/56 proptests pass; arrange_row bench deltas reported.
Risk: low.

---

PR 8 — Variant-C: serialized merger chunks + sort-by-prefix

Rationale: Two independent perf wins that both require the foundation (PR 4–7) to be in place.
Variant C switches Merger::Chunk from typed KVUpdates to serialized FactColumn::Align, letting the builder reuse one allocation across seals.
Sort-by-prefix adds a SortPrefix trait returning u128 so the chunker's sort compares cheap prefixes first before full Row::cmp.

Files:

• src/timely-util/src/columnar/factorized/batcher.rs — freeze_into_aligned, should_freeze.
• src/timely-util/src/columnar/factorized/batch.rs — FactBuilder reuse + push_borrowed_level_into per-iteration dedup.
• src/timely-util/src/columnar/factorized/chunker.rs — SortPrefix + prefix sort.
• src/timely-util/src/columnar/factorized/column.rs, factorized.rs — minor.
• src/repr/src/row.rs, src/repr/src/timestamp.rs — SortPrefix impls.
• src/timely-util/src/columnar/factorized/tests_prop.rs — new batcher proptests.

Commits:

• abbb23e8b5 + 0e73bcf869 (revert) — drop both (the dead-end first attempt).
• 8e9406ab0f serialize trie chunks, reuse builder allocations
• bef67e0056 end-to-end batcher proptests
• 365626bf7e dedup keys/vals per iteration (THE bug fix — never land 8e9406a without this)
• 4616ea187f 128-bit sort prefix

Critical invariant: FactBuilder::push_borrowed_level_into must dedup per iteration (not per chunk), because reduce output can emit a FactColumn::Typed chunk with duplicate K slots.
The regression test is test/sqllogictest/advent-of-code/2023/aoc_1204.slt run with --auto-index-selects.

Roll-out: always-on. These are internal implementation changes; no API surface.
Test plan: 62/62 factorized tests; aoc_1204 PASS; arrange_row shows −37% on n=100000/k=1000/v=10000.
Risk: medium — we've burned two attempts on this; land with the proptests + aoc_1204 regression test together.

---

Commit → PR mapping

Commit	PR
87f05b5653	1
0d5aba7213	1
03a6b10fa9	1
d43431729c	1
fbfe4d4268	1
0e6315dff1	1
5d337ddc8d	1
ed08c5a317	1
0d7aa7d47c	1
3a3afc6861	1
04de82a08e	1
0850913e31	1
1969290371	1
c43753b7d1	2
9414182bc8	2
4bbc6467e8	2
c56d43070d	2
dc7a1feda7	2
5d4de4a6c7	2
758c91da0d	2
65f40b8311	2
1b962e9e9b	2
13cacab15e	2
0764d05243	3
d49fd3c8d3	4
d37f4c87f2	4
d1763fef3f	4
a190d80fc4	4
ffbd8028e4	4
bb5dad48d2	4
4ee97c7c2e	4
4279de2ebc	4
eaa417902e	4
61d65d0bb9	4
bf0310b580	4
90404d72d8	4
ed75ae8d35	4
8d41982fbb	4
3fbafbbe85	drop (wip reverted in same branch)
35b3cdff3a	drop (the revert of above)
8a8b8d930e	5
4c896448d8	fold into 5 (2 MiB was the first cut; 64 KiB replaces)
f10afdabaa	fold into 5 (final 64 KiB + inline hints)
8a8de48299	6
2f7bd04a30	6
04720baf25	6
7a4074ac33	6
3f5b459b69	6
ba88e9fa8d	6
043b1018bf	6
83313f42f4	7
e4a764671c	7
abbb23e8b5	drop (buggy variant C, reverted)
0e73bcf869	drop (the revert of above)
8e9406ab0f	8
bef67e0056	8
365626bf7e	8
4616ea187f	8

Drops summary: 4 commits are transient dead-ends (two wip/revert pairs); net commit count after split ~ 54.

---

Open questions

1. Is PR 4's dyncfg gate worth the plumbing cost, or ship always-on behind CI coverage alone?
   The current branch already removed the dyncfg (commit 3f5b459b69), which merged PR 4 + PR 6 into one step — that's why the branch is hard to land.
   Splitting them back requires re-introducing the dyncfg, some Local vs FactLocal duplication, and extra #[cfg]-like branching in render.
   ~300 LoC of extra code for the release-window safety.
2. PR 1/2/3 can probably merge as a single larger PR if the reviewer is comfortable reading ~3900 LoC of net-new, consumer-free Rust.
   All proptests pass; it's pure library code.
   Collapsing 1+2+3 reduces ceremony to 6 PRs.
3. PR 8 could be split further: variant-C serialization (PR 8a) vs sort-by-prefix (PR 8b), since they're independent.
   Recommend keeping together — both change the chunker's internals, both rely on the same proptests, and landing them separately would require two rounds of feature-benchmark validation.
4. The arrange_row bench suite itself (in src/compute/benches/) touches the criterion_main! macro and should either ship with PR 4 or be extracted into a standalone "benches-only" PR 0 for easier review.

[antiguru]

Performance results

arrange_row microbench

Single-worker compute-only bench (src/compute/benches/arrange_row.rs). All numbers mean wall time, lower is better.

Tracked the evolution across the recent perf commits on this branch:

Config	pre-byte-target	2 MiB target	64 KiB target	64 KiB + variant-C + sort-prefix (HEAD)
n=10000 / k=100 / v=1000	2.40 ms	2.43 ms	2.45 ms	3.26 ms (+33% vs 64 KiB)
n=100000 / k=1000 / v=10000	35.2 ms	26.1 ms	37.65 ms	23.8 ms (−37% vs 64 KiB)
n=100000 / k=10 / v=100	18.3 ms	16.1 ms	18.89 ms	18.5 ms (−2%, within noise)

Takeaways:

• Going from a fixed 1024-leaf chunk target to a byte-based 2 MiB target wins −26% on the large k=1000/v=10000 config by amortizing allocator cost over fewer, larger chunks.
• Shrinking the byte target to 64 KiB (to bound per-batcher RAM when many arrangements share a worker) gives most of that back, but is required for memory-bounded workloads like ParallelIngestion.
• Variant-C (serialize Merger::Chunk to FactColumn::Align, reuse the builder target across seals) plus the 128-bit SortPrefix for Row recovers −37% on the large config.
• The small config regresses +33% because it only fires ~5 flushes total, so serialization+decode per chunk dominates over the amortization wins.
  Acceptable for maintained-arrangement workloads where the hot path is seal-heavy; worth watching on short-lived one-shot builds.

Feature benchmark (pending CI re-run)

Earlier feature-bench runs on this branch (before the 64 KiB / variant-C-done-right / sort-prefix commits) showed regressions across maintained workloads, for reference:

scenario	wallclock	clusterd RSS
GroupByMaintained	+47.8%	+74.6%
MinMaxMaintained	+37.7%	+35.7%
DifferentialJoin	+49.4%	—
FinishOrderByLimit	+45.8%	—
DeltaJoinMaintained	+10.2%	+27.8%
OrderBy	2.5× slower	+49.0%
CreateIndex	+85.6%	+25.8%
CustomerWorkload1	+44.1%	+47.7%
SkewedJoin	+26.1%	+15.4%
ParallelIngestion	+13.0%	+75.8% (before 64 KiB target)

Notes on those numbers:

• ParallelIngestion memory regression is directly from the then-2 MiB chunker pending buffer × 10 independent clusters; the 64 KiB change should cut that substantially.
• Wallclock regressions on maintained-arrangement scenarios (GroupByMaintained, MinMaxMaintained, DifferentialJoin, etc.) are what variant-C-done-right + sort-prefix target by reducing per-seal realloc churn and per-tuple Row::cmp cost.
• CI feature-benchmark needs a fresh run on the current HEAD to confirm the recovery; the microbench shape is consistent with the expected direction but can't amplify real workload effects.

Correctness regression covered

• test/sqllogictest/advent-of-code/2023/aoc_1204.slt run with --auto-index-selects exercises the FactBuilder per-iteration dedup path against WITH MUTUALLY RECURSIVE under indexed views.
  A prior variant-C attempt broke this (returned 20696 vs expected 978); the current code passes.

[antiguru]

Feature-benchmark update (CI run on current HEAD)

Earlier comment listed the pre-recent-perf-commits numbers.
The recent commits (64 KiB chunk target, Overflows/Timestamps extend overrides, variant-C serialization + builder reuse, 128-bit sort prefix) landed some wins and left some regressions.

Comparison vs main

Scenario	wallclock (then → now)	clusterd RSS (then → now)
Update	+25.9% → +23.7%	-4.3% → -0.8%
OrderBy	2.5× → +20.9%	+49.0% → +64.6%
MinMaxMaintained	+37.7% → +18.5%	+35.7% → +35.7%
DifferentialJoin	+49.4% → +41.6%	−20.5% → −30.1%
CustomerWorkload1	+44.1% → +21.0%	+47.2% → +31.0%
CreateIndex	+85.6% → +57.8%	+2.7% → +8.8%
GroupByMaintained	+47.8% → +23.0%	+74.6% → +61.4%
DeltaJoinMaintained	+10.2% → +10.1%	+27.8% → +9.5%
ParallelIngestion	+13.0% → −1.3%	+75.8% → +88.4%
FinishOrderByLimit	+45.8% → +36.4%	−1.9% → −10.2%

Plus workload_ddnet.yml:

metric	OLD	NEW	Δ
Data ingestion Mem (sum)	227.4	345.7	+52.0%
Mem avg (%)	18.5	32.0	+72.8%
Query p95 (ms)	41.6	48.7	+17.2%
Query p99 (ms)	93.4	104.6	+11.9%

Reading

• Wallclock regressions roughly halved across maintained-arrangement scenarios.
  GroupBy, MinMax, OrderBy, CustomerWorkload1, CreateIndex, FinishOrderByLimit all moved substantially in the right direction — consistent with fewer Row::cmp calls (sort prefix) and reduced per-seal realloc churn (builder reuse).
• ParallelIngestion wallclock is now −1.3% (was +13%).
  The 64 KiB chunk target bounded per-batcher pending RAM; 10 concurrent clusters no longer pay a 2 MiB × N tax.
• Memory regressions persist, and in some cases got slightly worse:
  • ParallelIngestion clusterd RSS went from +75.8% to +88.4% — unexpected; worth investigating whether variant-C's persistent FactColumn::Align output buffers (2-MiB-aligned, one per batcher) are the cost, or the sort-prefix Vec<(u128, tuple)> allocation.
  • OrderBy RSS went from +49% to +64.6%.
  • workload_ddnet data-ingestion memory +52% and avg memory +73% — same root cause suspected (per-batcher aligned buffers × N arrangements).
• DifferentialJoin and FinishOrderByLimit now use less clusterd RSS than main (−30% and −10% respectively).

Suspected remaining memory cost

Variant-C's freeze_into_aligned rounds every serialized chunk up to a TARGET_WORDS = 1 << 18 (2 MiB) boundary via alloc_aligned_zeroed.
Even with the 64 KiB chunker flush target, every emitted output chunk rounds up to ≥ 2 MiB.
For workloads with many concurrent arrangements (ParallelIngestion: 10 clusters, ddnet: many maintained indexes), this is the likely culprit.

Plan for follow-up: either scale TARGET_WORDS down (e.g. to match the 64 KiB flush budget) or use a non-aligned backing storage for small chunks and only spill to aligned bytes above a threshold.

Microbench (arrange_row) — unchanged from prior comment

One-config summary: n=100000/k=1000/v=10000 at 23.8 ms (−37% vs the 64 KiB-without-variant-C baseline).
Small config n=10000/k=100/v=1000 regressed +33% as expected for seal-light workloads.