Skip to content

feat: add geometry arrow stats conversion - #3286

Draft
thinh2 wants to merge 1 commit into
delta-io:mainfrom
thinh2:thinh-bui_data/change-a-geometry-arrow-json
Draft

feat: add geometry arrow stats conversion#3286
thinh2 wants to merge 1 commit into
delta-io:mainfrom
thinh2:thinh-bui_data/change-a-geometry-arrow-json

Conversation

@thinh2

@thinh2 thinh2 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

What changes are proposed in this pull request?

This PR adds the Arrow foundation for geometry support in the default engine.
Kernel GeometryType values now map to physical Arrow Binary fields with GeoArrow WKB
extension metadata, while geography remains unsupported in Arrow conversion for now.

It also teaches parse_json_impl to handle geometry stats encoded as WKT strings. The JSON path
decodes geometry leaves through a string-first schema, converts valid WKT values to WKB bytes, and
keeps invalid geometry stats isolated to null cells in the parsed stats batch.

How was this change tested?

cargo test -p delta_kernel --lib --features 'arrow-59 default-engine-base geo-type-in-dev test-utils' geometry -- --nocapture
cargo test -p delta_kernel --lib --features 'arrow-58 default-engine-base geo-type-in-dev test-utils' geometry -- --nocapture
cargo test -p delta_kernel --lib --features 'arrow-59 default-engine-base geo-type-in-dev test-utils' parse_json_safe_cast -- --nocapture
cargo clippy -p delta_kernel --benches --tests --features 'arrow-59 default-engine-base geo-type-in-dev test-utils' -- -D warnings
cargo check -p delta_kernel --no-default-features --features geo-type-in-dev
cargo nextest run --workspace --all-features

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.61039% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.28%. Comparing base (6dd5a96) to head (2757179).

Files with missing lines Patch % Lines
kernel/src/engine/arrow_utils/mod.rs 84.78% 14 Missing ⚠️
kernel/src/engine/arrow_conversion/mod.rs 94.44% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3286      +/-   ##
==========================================
- Coverage   90.28%   90.28%   -0.01%     
==========================================
  Files         251      253       +2     
  Lines       88562    88705     +143     
  Branches    88562    88705     +143     
==========================================
+ Hits        79960    80084     +124     
- Misses       5718     5732      +14     
- Partials     2884     2889       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pub(crate) const GEOARROW_EXTENSION_METADATA_KEY: &str = "ARROW:extension:metadata";
pub(crate) const GEOARROW_WKB_EXTENSION_NAME: &str = "geoarrow.wkb";

pub(crate) fn geometry_geoarrow_metadata(geometry: &GeometryType) -> HashMap<String, String> {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

manually constructed the ArrowField metadata matching the the GeoArrow specification.


use geoarrow_array::array::WktArray;
use geoarrow_array::cast::to_wkb;
use geoarrow_arrow_array_58::StringArray as GeoStringArray;

@thinh2 thinh2 Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pinned an arrow_array v58 for wkt_to_wkb_bytes conversion.

@thinh2 thinh2 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comments for reviewers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant