Skip to content

feat!: add element-at expression - #3120

Open
DrakeLin wants to merge 3 commits into
delta-io:mainfrom
DrakeLin:stack/kernel-partition-timezone-scans
Open

feat!: add element-at expression#3120
DrakeLin wants to merge 3 commits into
delta-io:mainfrom
DrakeLin:stack/kernel-partition-timezone-scans

Conversation

@DrakeLin

@DrakeLin DrakeLin commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

🥞 Stacked PR

Use this link to review incremental changes.


What changes are proposed in this pull request?

Add an ElementAt expression for row-wise Map<String, String> lookup with dynamic keys and rightmost-duplicate semantics. Arrow and DataFusion share the behavior, predicate handling remains conservative, and unsupported FFI visitors receive an explicit unknown node.

This PR affects the following public APIs

Add ElementAtExpression, Expression::element_at, and the ElementAt expression variant.

How was this change tested?

Unit, executor, serialization, predicate, and FFI tests cover dynamic and missing keys, nulls, duplicate keys, and expression traversal.

@DrakeLin DrakeLin changed the title feat: carry reader timezone in scan partition options feat: preserve reader timezone in parsed scan partitions Aug 13, 2026
@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.13973% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.19%. Comparing base (728aeb9) to head (3400443).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...src/engine/arrow_expression/evaluate_expression.rs 96.60% 6 Missing and 3 partials ⚠️
kernel/src/timestamp_timezone.rs 90.12% 2 Missing and 6 partials ⚠️
kernel/src/checkpoint/checkpoint_shape.rs 94.02% 1 Missing and 3 partials ⚠️
kernel/src/scan/log_replay.rs 96.66% 4 Missing ⚠️
ffi/src/expressions/kernel_visitor.rs 0.00% 3 Missing ⚠️
kernel/src/expressions/mod.rs 94.33% 3 Missing ⚠️
ffi/src/test_ffi.rs 0.00% 2 Missing ⚠️
kernel/src/plans/proto/convert.rs 94.28% 2 Missing ⚠️
kernel/src/scan/scan_plan.rs 90.90% 0 Missing and 2 partials ⚠️
ffi/src/expressions/engine_visitor.rs 97.95% 1 Missing ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3120      +/-   ##
==========================================
+ Coverage   90.08%   90.19%   +0.11%     
==========================================
  Files         248      250       +2     
  Lines       85653    86691    +1038     
  Branches    85653    86691    +1038     
==========================================
+ Hits        77162    78195    +1033     
+ Misses       5684     5676       -8     
- Partials     2807     2820      +13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions github-actions Bot added the breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump. label Aug 13, 2026
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

Benchmark results: ❌ Fail (a benchmark is ≥1.15x slower)

Summary: 🚀 1  ·  ✅ 12  ·  ☑️ 1  ·  🚧 0  ·  ❌ 1

Per-benchmark results (15 rows)
Test Change Base PR
clustered/readMetadataLatestPredicate/serial ✅ 1.00x 96.6±2.16ms 96.9±2.16ms
crcLatest/snapshotLatest ❌ 1.28x slower 9.0±0.10ms 11.5±0.47ms
crcMissing/snapshotLatest ✅ 1.03x faster 28.1±0.84ms 27.4±0.78ms
crcSlightlyStale/snapshotLatest 🚀 1.30x faster 12.7±0.40ms 9.8±0.09ms
crcVeryStale/snapshotLatest ✅ 1.02x faster 18.6±0.99ms 18.2±0.83ms
partitioned/readMetadataLatestPredicate/serial ✅ 1.02x faster 60.2±1.40ms 59.3±1.61ms
v1Checkpoint/readMetadataLatest/serial ☑️ 1.02x slower 12.7±0.31ms 12.9±0.27ms
v1Checkpoint/snapshotLatest ✅ 1.01x faster 870.4±34.71µs 864.3±29.58µs
v2Checkpoint/readMetadataLatest/parallel2 ✅ 1.00x 8.9±0.22ms 8.9±0.19ms
v2Checkpoint/readMetadataLatest/serial ✅ 1.00x 15.2±0.32ms 15.2±0.27ms
v2Checkpoint/snapshotLatest ✅ 1.00x 829.5±31.93µs 829.7±35.21µs
wideSchemaJsonStats/readMetadataLatestPredicate/serial ✅ 1.03x faster 74.4±0.66ms 72.0±1.71ms
wideSchemaJsonStats/snapshotLatest ✅ 1.00x 2.4±0.09ms 2.4±0.08ms
wideSchemaStructStats/readMetadataLatestPredicate/serial ✅ 1.03x faster 36.4±0.92ms 35.5±0.70ms
wideSchemaStructStats/snapshotLatest ✅ 1.00x 2.2±0.07ms 2.2±0.06ms

Legend: 🚀 ≥1.15x faster  · ✅ faster or unchanged  · ☑️ ≤1.03x slower  · 🚧 1.03x-1.15x slower  · ❌ ≥1.15x slower
Commit: 3400443 · Trigger: auto-push · Tags: base · Updated: 2026-08-14 10:16 PDT

@DrakeLin
DrakeLin force-pushed the stack/kernel-partition-timezone-scans branch from 502ee37 to ea8e321 Compare August 13, 2026 23:51
@DrakeLin DrakeLin changed the title feat: preserve reader timezone in parsed scan partitions feat!: preserve reader timezone in parsed scan partitions Aug 13, 2026
@DrakeLin
DrakeLin force-pushed the stack/kernel-partition-timezone-scans branch 5 times, most recently from 859810e to f545df3 Compare August 14, 2026 04:27
@DrakeLin DrakeLin changed the title feat!: preserve reader timezone in parsed scan partitions feat!: add element-at expression Aug 14, 2026
@DrakeLin
DrakeLin force-pushed the stack/kernel-partition-timezone-scans branch 3 times, most recently from f70c857 to 206b81c Compare August 14, 2026 07:32
@DrakeLin
DrakeLin force-pushed the stack/kernel-partition-timezone-scans branch from 206b81c to a04cf06 Compare August 14, 2026 16:20
@DrakeLin
DrakeLin force-pushed the stack/kernel-partition-timezone-scans branch from a04cf06 to 3400443 Compare August 14, 2026 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant