Skip to content

Reuse parsed stats in scan_metadata_from #3307

Description

@dengsh12

Problem

scan_metadata_from reshapes cached scan-file metadata into checkpoint-like Add actions, but the transform preserves only JSON stats and raw partitionValues.

At kernel/src/scan/mod.rs:975-978, it uses get_scan_metadata_transform_expr() with restored_add_schema().

get_scan_metadata_transform_expr() (kernel/src/scan/log_replay.rs:923-934) preserves stats at line 928 but omits the optional stats_parsed and partitionValues_parsed columns.

The resulting checkpoint information then unconditionally declares:

has_stats_parsed: false,
has_partition_values_parsed: false,

at kernel/src/scan/mod.rs:1048-1051.

When the current scan needs typed stats or partition values, log replay consequently reparses them:

  • Expression::parse_json(...) at kernel/src/scan/log_replay.rs:896
  • Expression::map_to_struct(...) at kernel/src/scan/log_replay.rs:909

Cached scan metadata may already contain these parsed columns. In addition to repeated work, dropping stats_parsed can reduce pruning when JSON stats is null.

Proposed optimization

  1. Accept the actual cached scan-file schema, including optional parsed columns.
  2. Preserve stats_parsed and partitionValues_parsed when reshaping cached rows into Add actions.
  3. Verify that their schemas are compatible with the current scan’s physical stats and partition schemas.
  4. Build CheckpointReadInfo dynamically:
    • Set the parsed flags when compatible parsed columns are preserved.
    • Include those columns in checkpoint_read_schema.
  5. Fall back to parsing JSON stats or mapping raw partition values when parsed columns are absent or incompatible.

Testing

Add coverage where:

  • A previous scan emits stats_parsed.
  • A previous scan emits partitionValues_parsed.
  • scan_metadata_from reuses compatible parsed columns.
  • Incompatible parsed schemas fall back safely.
  • Cached data with optional parsed columns satisfies evaluator input-schema validation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions