Problem
scan_metadata_from reshapes cached scan-file metadata into checkpoint-like Add actions, but the transform preserves only JSON stats and raw partitionValues.
At kernel/src/scan/mod.rs:975-978, it uses get_scan_metadata_transform_expr() with restored_add_schema().
get_scan_metadata_transform_expr() (kernel/src/scan/log_replay.rs:923-934) preserves stats at line 928 but omits the optional stats_parsed and partitionValues_parsed columns.
The resulting checkpoint information then unconditionally declares:
has_stats_parsed: false,
has_partition_values_parsed: false,
at kernel/src/scan/mod.rs:1048-1051.
When the current scan needs typed stats or partition values, log replay consequently reparses them:
Expression::parse_json(...) at kernel/src/scan/log_replay.rs:896
Expression::map_to_struct(...) at kernel/src/scan/log_replay.rs:909
Cached scan metadata may already contain these parsed columns. In addition to repeated work, dropping stats_parsed can reduce pruning when JSON stats is null.
Proposed optimization
- Accept the actual cached scan-file schema, including optional parsed columns.
- Preserve
stats_parsed and partitionValues_parsed when reshaping cached rows into Add actions.
- Verify that their schemas are compatible with the current scan’s physical stats and partition schemas.
- Build
CheckpointReadInfo dynamically:
- Set the parsed flags when compatible parsed columns are preserved.
- Include those columns in
checkpoint_read_schema.
- Fall back to parsing JSON stats or mapping raw partition values when parsed columns are absent or incompatible.
Testing
Add coverage where:
- A previous scan emits
stats_parsed.
- A previous scan emits
partitionValues_parsed.
scan_metadata_from reuses compatible parsed columns.
- Incompatible parsed schemas fall back safely.
- Cached data with optional parsed columns satisfies evaluator input-schema validation.
Problem
scan_metadata_fromreshapes cached scan-file metadata into checkpoint-like Add actions, but the transform preserves only JSONstatsand rawpartitionValues.At
kernel/src/scan/mod.rs:975-978, it usesget_scan_metadata_transform_expr()withrestored_add_schema().get_scan_metadata_transform_expr()(kernel/src/scan/log_replay.rs:923-934) preservesstatsat line 928 but omits the optionalstats_parsedandpartitionValues_parsedcolumns.The resulting checkpoint information then unconditionally declares:
at
kernel/src/scan/mod.rs:1048-1051.When the current scan needs typed stats or partition values, log replay consequently reparses them:
Expression::parse_json(...)atkernel/src/scan/log_replay.rs:896Expression::map_to_struct(...)atkernel/src/scan/log_replay.rs:909Cached scan metadata may already contain these parsed columns. In addition to repeated work, dropping
stats_parsedcan reduce pruning when JSONstatsis null.Proposed optimization
stats_parsedandpartitionValues_parsedwhen reshaping cached rows into Add actions.CheckpointReadInfodynamically:checkpoint_read_schema.Testing
Add coverage where:
stats_parsed.partitionValues_parsed.scan_metadata_fromreuses compatible parsed columns.