Skip to content

perf: bypass query plans for JSON reads - #3314

Draft
chiinlquah wants to merge 1 commit into
delta-io:mainfrom
chiinlquah:stack/perf/direct-json-read-fast-path
Draft

perf: bypass query plans for JSON reads#3314
chiinlquah wants to merge 1 commit into
delta-io:mainfrom
chiinlquah:stack/perf/direct-json-read-fast-path

Conversation

@chiinlquah

@chiinlquah chiinlquah commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

What changes are proposed in this pull request?

Read JSON files in PlanBasedJsonHandler through the typed ReadBytes I/O operation and decode
the returned bytes directly with Arrow JSON. This avoids constructing and executing a generic
single-node query plan for JSON scans while preserving the generic query-plan path for other plan
shapes.

The Arrow JSON byte decoder is shared with SyncJsonHandler. A focused test wraps the plan
executor and rejects QueryPlan operations, proving that JSON reads take only the typed I/O path.

In a local CommitRange benchmark over 1,000 one-action commit files (50 warmups, 10 measured
iterations, release/O3 native build), the median fell from 3.616 s with generic driver-local plan
execution to 0.788 s with direct JSON, a 4.59x speedup. Both variants launched zero Spark jobs.
The remaining cost includes three scans per commit in the benchmarked CommitRange call path.

How was this change tested?

  • cargo test -p delta_kernel --features declarative-plans engine::plans::json::tests::test_read_json_files
  • Repository pre-commit checks (formatting, Clippy, and tests)

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.79245% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.34%. Comparing base (048ffbe) to head (0adc981).

Files with missing lines Patch % Lines
kernel/src/engine/plans/json.rs 89.74% 1 Missing and 3 partials ⚠️
kernel/src/engine/arrow_utils/mod.rs 76.92% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3314   +/-   ##
=======================================
  Coverage   90.34%   90.34%           
=======================================
  Files         250      250           
  Lines       89131    89172   +41     
  Branches    89131    89172   +41     
=======================================
+ Hits        80524    80563   +39     
- Misses       5672     5673    +1     
- Partials     2935     2936    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant