Skip to content

Add lazy physical parquet scans - #172

Merged
platypii merged 1 commit into
masterfrom
parquet-scan
Aug 20, 2026
Merged

Add lazy physical parquet scans#172
platypii merged 1 commit into
masterfrom
parquet-scan

Conversation

@platypii

Copy link
Copy Markdown
Collaborator

Summary

Adds parquetScan, a lazy, column-oriented API for query engines that need physical row ranges and selective column reads.

What changed

  • Exposes candidate row ranges using absolute, half-open file coordinates.
  • Reads individual columns lazily for a candidate range or exact subrange.
  • Uses row-group statistics, bloom filters, and page indexes for conservative pruning.
  • Keeps pruning separate from exact value filtering.
  • Enables offset-index range reads by default for scans.
  • Automatically makes pruning-filter columns available without eagerly reading them.
  • Caches covering column reads for contained subranges.
  • Shares metadata, index preparation, and physical-plan execution with existing read APIs.
  • Adds public TypeScript types and exports for the scan API.

Performance

Scan preparation reads metadata and requested indexes only; column data remains lazy. Exact subranges use offset indexes where available to avoid reading complete column chunks. Existing row-oriented reads retain eager, parallel range fetching.

Validation

  • Added coverage for lazy reads, exact subranges, cache reuse, pruning, absolute coordinates, and API validation.
  • Verified offset-index reads transfer less data than full-column reads.
  • Verified filter-only columns are not unnecessarily fetched by parquetReadColumn.
  • Tests, lint, generated types, fixture comparisons, and differential checks against master pass.

Follow-up opportunity

The scan currently caches the most recent requested range per column. Retaining page-aligned decoded coverage and parsed offset indexes could avoid repeated I/O for adjacent, separately requested subranges.

@platypii
platypii merged commit 43226ac into master Aug 20, 2026
6 checks passed
@platypii
platypii deleted the parquet-scan branch August 20, 2026 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant