[Enhancement] Expression-driven on-demand lazy column loading for parquet scanner#74886
Conversation
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05b7e36091
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6c7b91e35
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
47a52c8 to
42cc009
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42cc009dc4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f9bf5bc0c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
6f9bf5b to
c414f36
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e4f0f8005
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0871e538c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
62218c1 to
6225939
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Signed-off-by: tqqq <dirtysalt1987@gmail.com>
Signed-off-by: tqqq <dirtysalt1987@gmail.com>
Signed-off-by: tqqq <dirtysalt1987@gmail.com>
Signed-off-by: tqqq <dirtysalt1987@gmail.com>
Signed-off-by: tqqq <dirtysalt1987@gmail.com>
Signed-off-by: tqqq <dirtysalt1987@gmail.com>
Signed-off-by: tqqq <dirtysalt1987@gmail.com>
6225939 to
558d693
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 558d693a1b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 2 / 2 (100.00%) file detail
|
[BE Incremental Coverage Report]✅ pass : 464 / 526 (88.21%) file detail
|
|
@Mergifyio backport branch-4.1 |
|
@mergify backport branch-4.1 |
|
@mergify backport branch-4.1 |
✅ Backports have been createdDetails
Cherry-pick of b083761 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
☑️ Command
|
1 similar comment
☑️ Command
|
…quet scanner (StarRocks#74886) Signed-off-by: tqqq <dirtysalt1987@gmail.com> Co-authored-by: wanpengfei-git <wanpengfei91@163.com> Signed-off-by: Oliver Layer <o.layer@celonis.com>
Why I'm doing:
For multi-branch OR queries with
count(*), StarRocks reads far more IO than Trino:The root cause: compound conjuncts are evaluated at the scanner level after all lazy columns have been backfilled, so expression short-circuit never prevents IO. And predicate-only columns are forced into the output chunk schema, so
read_lazy_columns()must backfill them regardless of whether expression evaluation ever reached them.What I'm doing:
Two changes:
1. Evaluate compound conjuncts inside GroupReader
Previously
scanner_ctxs(multi-slot OR expressions) were evaluated at the scanner level afterGroupReader::get_next()returned a fully-materialized chunk. Now they are evaluated insideget_next()whileLazyMaterializationContextis still attached asMissingColumnProvider.This allows
ColumnRefto triggermaterialize_slot()on-demand when the expression evaluator reaches a branch whose date-guard matches the current batch. Branches whose date-guard is all-false are short-circuited at the AND level — their payload columns are never loaded.2. Skip physical IO for predicate-only untriggered columns
FE already distinguishes output columns from predicate columns via the
isOutputColumnflag. Inread_lazy_columns(), predicate-only columns that were never triggered during expression evaluation are null-filled instead of read from disk — they cost zero IO and their values are never accessed by downstream operators.For
count(*), all lazy columns are predicate-only, so only the 1-2 branches per batch that are actually reached incur physical IO.3. Separate active/lazy IO ranges when lazy loading is active
SharedBufferedInputStreamsupports two modes: merged (active+lazy ranges coalesced into one buffer) and separated (independent buffers). In merged mode, reading the active column would fetch the entire merged range — including all lazy column data — defeating the physical IO savings. This PR forces separated mode whenparquet_late_materialization_enableis on and lazy columns exist, so that untriggered lazy columns' data is never physically fetched.Compromise
Predicate-only untriggered columns are null-filled rather than removed from the output chunk. The output chunk schema (
_tuple_desc) includes all referenced columns, andreorder_chunk()downstream requires all slots to exist. The null-fill costs trivial memory but zero IO.A follow-up PR will introduce an
output_tuple_desc(filtered tois_output_column()slots) for output chunk creation, eliminating the null-fill entirely.What type of PR is this:
Does this PR entail a change in behavior?
Checklist:
Bugfix cherry-pick branch check: