Skip to content

Defer parquet scan column planning - #173

Merged
platypii merged 1 commit into
masterfrom
scan-plan-performance
Aug 21, 2026
Merged

Defer parquet scan column planning#173
platypii merged 1 commit into
masterfrom
scan-plan-performance

Conversation

@platypii

Copy link
Copy Markdown
Collaborator

Summary

parquetScan is a lazy API, but it previously built column read plans eagerly for every candidate row group.

This refactors planning into two stages:

  • parquetPlanGroups selects candidate row groups and row ranges.
  • parquetPlanGroup builds column chunk and byte-range plans for one group.
  • parquetPlan composes both stages, preserving existing parquetRead behavior.
  • parquetScan now performs only group planning during initialization and creates a one-column plan when readColumn is called.

This avoids inspecting and planning unused columns while keeping row-group pruning, page ranges, offset indexes, and scan caching behavior unchanged.

Performance

Local benchmark using a 170 MB Parquet file with 129 row groups:

Case master This PR
Scan initialization 0.0334 ms 0.0101 ms
Initialize + read first range 0.375 ms 0.341 ms
Read one column across all groups 33.28 ms 34.14 ms

Scan initialization is approximately 3.3× faster. Full-column read performance is effectively unchanged within benchmark noise.

The existing full-file parquetReadObjects benchmark improved from a median of 1,111 ms to 1,025 ms, though it does not exercise the parquetScan path directly.

@platypii
platypii merged commit 0620905 into master Aug 21, 2026
6 checks passed
@platypii
platypii deleted the scan-plan-performance branch August 21, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant