[Tracking] `TestTableBuilder` framework and cross-product integration testing

[DrakeLin]

Motivation

Writing a kernel integration test that needs a Delta table today usually means hand-rolling JSON commits, parquet, and protocol/metadata consistency by hand. There is no systematic way to test feature interactions across log states, table features, and snapshot load paths, so most of the cross-product (LogState x FeatureSet x DataLayout x TableConfig x VersionTarget) is untested in OSS.

The goal is to replace ad-hoc test setup with a composable builder (TestTableBuilder in test-utils/) and use it to land a default cross-product sweep.

Ideal end state

A developer writes one parameterized test that runs across every meaningful Delta table state by applying a standard rstest_reuse template:

#[apply(default_sweep)]
fn test_my_feature(
    log_state: LogState,
    feature_set: FeatureSet,
    layout_config: (DataLayoutConfig, TableConfig),
    version_target: VersionTarget,
) {
    let (data_layout, table_config) = layout_config;
    let table = test_table(log_state, feature_set, data_layout, table_config);
    let snap = build_snapshot!(version_target, table.table_root(), engine);
    // assertions
}

The axes:

Axis	What it controls	Examples
LogState	shape of _delta_log	commits-only, checkpoint at end/middle, multi-checkpoint, CRC at last, stale/missing _last_checkpoint hint, post-cleanup, log compaction range, catalog log tail, schema-history ops
FeatureSet	enabled features	empty, CM=id/name, DV, RT, V2 checkpoint, ICT+CDF, types, catalog-managed, all-common combo
DataLayoutConfig	data file organization	unpartitioned-flat, partitioned-by-primitive, clustered-by-primitive; null distributions; nested schema depth
TableConfig	table properties	checkpoint stats JSON / struct / off, numIndexedCols, dataSkippingStatsColumns
VersionTarget	how the snapshot is loaded	Latest, AtVersion(v), IncrementalToLatest{from}, IncrementalFrom{from,to}, AtTimestamp(ts)

DataLayoutConfig and TableConfig are bundled into a single layout_config axis in the default sweep (rather than crossed): the sweep round-trips each pairing with no predicate, so the stats config can't affect the version or row-count assertions, and crossing the two would add cases without adding coverage. The standalone per-axis templates (data_layout_sweep, table_config_sweep) keep them independent.

Invalid combinations are eliminated by construction: mutually exclusive features live in separate FeatureSet rows, dependency-required features are auto-enabled, and LogState shapes that require a specific feature (schema history, catalog tail) run as their own targeted tests rather than entries in the default sweep.

Two use cases

1. Full sweep: curated baseline run as part of the integration suite. The current default_sweep is 17 LogState x 3 FeatureSet x 11 (DataLayout, TableConfig) x 5 VersionTarget = 2,805 cases. Every combo valid by construction; failure attribution is per-axis.
2. Per-axis sweep: a developer adding a feature picks one of the per-axis templates (log_state_sweep, feature_set_sweep, data_layout_sweep, table_config_sweep, version_target_sweep) and supplies their own #[values(...)] for the remaining axes. Drop to hand-rolled add_commit for corrupt/malformed cases.

Catalog-managed tables are an orthogonal modifier with their own targeted suite. When catalogManaged is on, snapshot loading uses with_log_tail(commits) rather than the standard load path, and the builder produces all three catalog-tail states (none / k staged + matching tail / k staged + mixed materialization). These run paired with checkpoint and CRC variants to cover the staged -> ratified boundary.

Project plan

Default cross-product (v1) landed in #2544. Future axes (catalog tail, schema history, log compaction) layer on top.

Foundation (merged)

PR	Item
#2282	core framework + initial VersionTarget (Latest, AtVersion, IncrementalToLatest)
#2283	FeatureSet methods + initial TableConfig covering writeStatsAsJson x writeStatsAsStruct
#2320	data generation
#2321	partition support + initial DataLayoutConfig (Unpartitioned, PartitionedAllTypes, ClusteredAllTypes)
#2284	multi-checkpoint support
#2521	LastCheckpointHintState (present / missing / stale)
#2521	LogState::with_cleanup_commits_before(n) (post-cleanup variants)
#2522	CRC via Snapshot::write_checksum
#2544	Default cross-product test suite v1 over (LogState x FeatureSet x DataLayoutConfig x TableConfig x VersionTarget), exercises read end-to-end
#2657	sparse null injection in data generation (null-distribution half; nesting depth via #2648)
#2652	VersionTarget::AtTimestamp(ts) + VersionTarget::IncrementalFrom { from, to }
#2653	numIndexedCols + dataSkippingStatsColumns TableConfig knobs; bundled DataLayoutConfig x TableConfig into one layout_config axis

In review

PR	Item
#2455	catalog-managed minimal: catalogManaged flag + single staged commit threaded through with_log_tail
#2831	Add + Remove cases: LogState::with_removed_file() + Remove-action read coverage across delta-only / checkpoint / post-cleanup log states. DV-aware reads split into a follow-up

Open work

Roughly one item per two working days, in priority order.

Target	Item
Jun 29	Read predicates (one per kernel pruning path: partition prune, min/max prune, null-count prune, DV-aware) wired into the default sweep.
Jul 1	DV cases: DV-aware reads in the sweep, gated on the deletionVectors feature (the same hook plain removes use in #2831, deleting via a DV when the feature is on). Add + Remove landed in #2831.
Jul 3	DomainMetadata / SetTxn / ICT (get_in_commit_timestamp) read assertions. Requires public get_in_commit_timestamp first.
Jul 7	LogState::catalog_tail sub-axis (none / k staged + matching tail / k staged + mixed materialization + tail) + load helper that calls with_log_tail(...) automatically + targeted catalog-managed x (catalog_tail x checkpoint x CRC) suite. (in review: #2455)
Jul 9	Schema-history axis (ADD / DROP / RENAME / ALTER) with feature gating + targeted schema-history x CM-enabled suite.
Jul 13	Log compaction file support in LogState.
Jul 15	Extensible table-state invariant validation (assert_table_invariants): generalize the row-ID uniqueness assertion (landed in #2794) into a reusable aggregator of self-gating checks — rowId uniqueness now, DomainMetadata-survives-checkpoint and replay/CRC next. Tracked in #2809.
ongoing	Migrate existing hand-rolled integration tests onto the builder.

[scottsand-db]

Looks great! Some comments:

1. catalog-managed basic integration -- what is basic? what is non-basic?
2. Schema-history axis - ADD / DROP / RENAME / ALTER with feature gating -- lower priority, come later
3. LogState catalog_tail sub-axis -- very important, come before ALTER
4. Log compaction file support in LogState -- less important, come later
5. Default cross-product test suite exercising read, read-with-predicate, scan-metadata, round-trip-checkpoint -- can't we have this very soon? and we just update it as we add new dimensions?
6. Missing: tables that have had log cleanup maintenance operations performed on them. Overall, the theme is here: prioritize the most common table states.
7. What are the read predicates that you are using?

[scottsand-db]

I think also: DomainMetadata tets, SetTxn tests, and ICT tests (get the value)

[DrakeLin]

Test numIndexedStatsCols > 2, query for those stats

[scottsand-db]

Add + Removes + DVs