Skip to content

Push down Parquet scans with delete-aware ranges - #43

Merged
platypii merged 1 commit into
masterfrom
parquet-scan-pushdown
Aug 21, 2026
Merged

Push down Parquet scans with delete-aware ranges#43
platypii merged 1 commit into
masterfrom
parquet-scan-pushdown

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Summary

  • use hyparquet scan candidates for row-group, bloom-filter, and page-index pruning in both materialized and prepared reads
  • retain absolute row positions so position deletes remain correct, and apply equality deletes before exposing deferred payload columns
  • update to hyparquet 1.29.1 and reuse scan metadata/ranges instead of adding a separate callback path
  • use a 128 KiB initial footer fetch for Parquet files below 8 MiB while retaining hyparquet's default for larger files

Closes #41.

Performance

Real HypAware cache

hypaware query overview --json --days 7 --include-local-only against a local cache containing 1,374 Parquet files. Implementations alternated each iteration, with two warmups per implementation and 11 measured runs:

master this PR change
median runtime 2,206.0 ms 2,220.1 ms +0.64%
output 6,462 bytes 6,462 bytes identical SHA-256

A separate seven-run confirmation measured +1.12%. The non-delete many-file path is therefore effectively at parity, with a repeatable approximately 1% cost.

Selective reads and deletes

workload master this PR result
one-file materialized read 1,077.8 ms 1,089.3 ms 1.1% slower
page-index selective read 0.463 ms 0.442 ms 4.8% faster
count with one position delete 33.45 ms 0.51 ms 65x faster
selective payload with one position delete 48.78 ms 5.74 ms 8.5x faster

The count-with-delete read volume fell from 2.491 MiB to 0.125 MiB. The selective payload query fell from 2.491 MiB to 0.388 MiB.

Footer range traffic

A read-only snapshot of 974 production Parquet files was evaluated using their actual file and footer lengths:

policy bytes fetched second requests
hyparquet 512 KiB default 48.1 MiB 0
adaptive 128/512 KiB policy 36.2 MiB 0

This is a 24.8% reduction in initial metadata traffic without another range request. The largest observed footer on a file below 8 MiB was 81.2 KiB. On the 160 raw AI gateway log files alone, the reduction was 22.5%.

Validation

  • npm test — 59 files, 702 tests passed
  • npm run lint
  • npx tsc

Use hyparquet scan candidates for prepared and materialized reads, preserve Iceberg position/equality deletes, and adapt initial footer fetches for small Parquet files.
@platypii
platypii merged commit 8bb46be into master Aug 21, 2026
6 checks passed
@platypii
platypii deleted the parquet-scan-pushdown branch August 21, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Position deletes use filtered row indices under pushed WHERE

1 participant