Factorized columnar storage types#36118

Draft

antiguru wants to merge 60 commits intoMaterializeInc:mainfrom

antiguru:worktree-factorized-columns

Commits on Apr 22, 2026

feat(timely-util): add factorized columnar types with push_flat and iter

antiguru
and
claude
committed
feat(timely-util): add form(), len, clear, is_empty for factorized columns
antiguru
committed
test(timely-util): add strides, roundtrip, and benchmark tests for factorized columns

antiguru
and
claude
committed
bench(timely-util): add criterion benchmarks for factorized columns

antiguru
and
claude
committed
bench(timely-util): add throughput and length_in_words metrics to factorized benchmarks

antiguru
and
claude
committed
feat(timely-util): add KVUpdates types with Repeats/Lookbacks leaf compression

antiguru
and
claude
committed
feat(timely-util): add for_each_cursor using Index::cursor for fast Repeats iteration

antiguru
and
claude
committed
bench(timely-util): add serialized iteration benchmarks for factorized columns

antiguru
and
claude
committed
chore: point columnar dep at antiguru/columnar#feature/cursor-index

antiguru
and
claude
committed
feat(timely-util): add AsBytes/FromBytes for Level, refactor to generic Level<L, Rest>

antiguru
and
claude
committed
bench(timely-util): add sort cost and builder pipeline benchmarks

antiguru
and
claude
committed
bench(timely-util): add smart radix sort (skip uniform bytes, datatoad approach)

antiguru
and
claude
committed
feat(timely-util): add factorized arrangement stack (batch, cursor, merger, builder)

antiguru
and
claude
committed
feat(timely-util): add FactValBatcher type alias for standard Vec-based merge batcher

antiguru
and
claude
committed
feat(timely-util): add FactColumn container with ContainerBytes, batcher alias

antiguru
and
claude
committed
bench(timely-util): add arrangement stack benchmarks

antiguru
and
claude
committed
bench(timely-util): add 1M-element configs to arrangement benchmarks

antiguru
and
claude
committed
perf(timely-util): reserve+extend in stash_updates for 15-29% merge speedup

antiguru
and
claude
committed
fix: update AsBytes/Index impls for columnar crate API changes

antiguru
and
claude
committed
fix: address CI lints in overflowing, container, and mod

antiguru
and
claude
committed
fix(timely-util): replace `as` conversions with `CastFrom`, switch to sibling-file module

antiguru
and
claude
committed
perf(timely-util): galloping seek_key/seek_val, 2-12x speedup on large batches

antiguru
and
claude
committed
test(timely-util): validate FactBatch bounds for Row-like variable-length keys

antiguru
and
claude
committed
bench(timely-util): add row-like (Vec<u8> keyed) arrangement benchmarks

antiguru
and
claude
committed
feat(compute): add FactRowRowSpine aliases + ArrangementSize impl

antiguru
and
claude
committed
bench(compute): add arrange_row comparing RowRowSpine vs FactRowRowSpine

antiguru
and
claude
committed
docs(plan): update factorized-arrangement plan to reflect current state

antiguru
and
claude
committed
feat(timely-util): trie-aware batcher for factorized arrangements

antiguru
and
claude
committed
wip: attempt RowRowSpine = FactValSpine direct swap

antiguru
and
claude
committed
Revert "wip: attempt RowRowSpine = FactValSpine direct swap"
antiguru
committed
feat(compute): thread FactLocal arrangement flavor through render

antiguru
and
claude
committed
feat(compute): emit FactLocal arrangements under dyncfg flag

antiguru
and
claude
committed
feat(compute): use columnar wire format for FactLocal arrangements

antiguru
and
claude
committed
feat(compute): wire threshold to FactLocal via FactColBuilder

antiguru
and
claude
committed
feat(compute): route linear_join pre-arrange through FactLocal under dyncfg

antiguru
and
claude
committed
docs(plan): update migration plan with reduce block + wrapper path

antiguru
and
claude
committed
refactor(repr): make Row's Columnar Ref a DatumSeq, not &RowRef

antiguru
and
claude
committed
feat(compute): migrate reduce output to FactLocal

antiguru
and
claude
committed
feat(compute): bridge FactLocal at index export / delta-join + flip flag on

antiguru
and
claude
committed
refactor(compute): loosen peek DiffGat bound for Fact cursor

antiguru
and
claude
committed
refactor(compute): flip TraceBundle/delta-join/logging to Fact trace; drop bridges

antiguru
and
claude
committed
refactor(compute): rename ArrangementFlavor::FactLocal -> Local; drop old Local

antiguru
and
claude
committed
refactor(compute): migrate top_k to FactRowRow* aliases
antiguru
committed
refactor(compute): rename FactRowRow* → RowRow*; drop dead RowRowSpine layout
antiguru
committed
style(compute): cargo fmt catch-up for tier-5 migration
antiguru
committed
chore(compute): drop enable_compute_factorized_arrangement dyncfg
antiguru
committed
style(compute): shorten overlong RowRowEnter lines via local alias
antiguru
committed
docs(plan): close out factorized-migration plan after tier 5
antiguru
committed
perf(timely-util): byte-based chunk target (2 MiB) for trie batcher

antiguru
and
claude
committed
perf(timely-util): serialize trie chunks to FactColumn::Align in batcher

antiguru
and
claude
committed
Revert "perf(timely-util): serialize trie chunks to FactColumn::Align in batcher"
antiguru
committed
perf(timely-util): shrink factorized chunk target to 64 KiB + inline hot paths

antiguru
and
claude
committed
perf(ore,timely-util): bulk extend override for Overflows; use in trie builder

antiguru
and
claude
committed
perf(repr): add extend override to Timestamps Push impl

antiguru
and
claude
committed
perf(timely-util): serialize trie chunks, reuse builder allocations

antiguru
and
claude
committed
test(timely-util): add end-to-end batcher proptests for variant-C

antiguru
and
claude
committed
fix(timely-util): dedup keys/vals per iteration in FactBuilder
antiguru
committed
perf(timely-util): 128-bit K sort prefix for factorized trie chunker

antiguru
and
claude
committed
perf(timely-util): size freeze_into_aligned to exact trie footprint
antiguru
committed
perf(timely-util): implement ContainerBuilder::relax for FactTrieChunker
antiguru
committed