feat(python/sedonadb-geopandas): add column assignment - #1195
Open
jiayuasu wants to merge 7 commits into
Open
Conversation
Support gdf[key] = value for same-frame Series, broadcastable scalars, and geometry values. Assignment validates its input — a Series from another frame, a bare expression, and array-likes are rejected with actionable messages, since none records a usable row provenance — and geometry assignment preserves the active-geometry bookkeeping: a geometry column becomes active only when the frame had none, replacing the active geometry with a non-geometry clears it, and destination CRS is stamped only onto CRS-less geometry. The scalar surface accepts Python scalars, NumPy scalars (0-d arrays included), Arrow scalars, masked values, and pandas NA; NumPy and pandas temporal scalars are rejected for now because representing them faithfully needs dedicated unit and timezone handling, which arrives as its own change. Raises the sedonadb floor to 0.4.1 for DataFrame.mutate.
…lization gaps Read any geometry-typed column back as a GeoSeries rather than only the active one, so a freshly assigned geometry column supports .area and .buffer() immediately. Keep an explicitly cleared active geometry cleared through materialization by applying GeoPandas' own constructor rule instead of the engine heuristic. Inherit the destination CRS when a CRS-less same-frame geometry Series replaces a column that has one, mirroring the scalar path, and construct replacements for geography columns with the destination's spatial kind so they stay geography. Preserve NumPy scalar dtypes (and uint64 range) by rebuilding them as typed Arrow scalars instead of .item() promotion, classify Arrow-wrapped nulls and NaN as missing exactly like their bare payloads, broadcast typed-null nested Arrow scalars through one-element typed arrays, reject pandas NaT with the same temporal deferral as Timestamp and Timedelta (and disclose the deferral in the README), and recognize Shapely 1.x multipart geometries as single values before the sequence check.
…CRS, and scalar edge cases Preserve the no-active-geometry marker through materialization whatever the remaining columns are named: the GeoDataFrame constructor auto-activates a geometry column literally named 'geometry', and there is no public spelling for a frame with geometry columns but none active, so the marker is cleared directly. Apply the destination CRS unconditionally for missing and bare-Shapely geography replacements — the geography constructors synthesize CRS84, which the CRS-absence guard mistook for a CRS the value carried. Exempt np.void from the typed Arrow scalar conversion, which broke its previously working bytes broadcast. Classify the one-element-null Arrow array spelling of a typed-null nested scalar as missing explicitly, instead of through a pandas coincidence that a pandas-less install would lose.
…lls, and structured scalars Route a typed spatial null (an invalid GeoArrow scalar carrying its own kind and CRS) through the value path so its metadata survives, instead of synthesizing a destination-kind null stamped with the destination CRS; typed non-spatial nulls still mean missing geometry. Strip the CRS84 that geography constructors synthesize when the destination column is explicitly CRS-less, so replacements do not silently gain a CRS. Clear the no-active-geometry marker in place during materialization rather than reconstructing the frame, which let the GeoDataFrame constructor coerce an unrelated all-null column named geometry to geometry dtype. Broadcast structured NumPy scalars as typed Arrow structs preserving field names and dtypes (rejecting dtypes with no Arrow mapping) instead of flattening them to lossy lists, keeping the bytes fallback for plain voids only. The pandas-free nested-null classification now has a test that actually blocks the pandas import.
…s, honor GeoArrow metadata Strip a synthesized CRS with the value-preserving st_setsrid(0) instead of st_setcrs(NULL), which null-propagates its CRS argument and erased every row while leaving the schema looking correct; the regression test now asserts materialized values, not just the schema. Recognize GeoArrow-typed scalars from their extension name rather than resolving them — the scalar resolver drops the planar/spherical edge type and rejects non-WKB storage outright — and resolve WKB-typed scalars, valid and null alike, through their one-element-array spelling, which honors the complete extension metadata; a null of non-WKB storage is rebuilt from its own kind and CRS, and a valid one keeps the resolver's error. Handle structured masked records before the generic mask check, which itself raises on the mask dtype: masked fields become struct nulls and a fully masked record is missing. The no-pandas regression test now builds its Arrow inputs before blocking the import (PyArrow's lazy pandas shim otherwise fails first) and was mutation-checked.
…o every destination Recognize GeoArrow-typed scalars before the new-column fast path, so a spherical WKB scalar assigned to a fresh or non-geometry column keeps its geography kind (and activates a geometry-less frame with the right semantics) and non-WKB nulls resolve there too; destination CRS inheritance still applies only when replacing an existing geometry column, since a new destination has no metadata to inherit. Serialize a non-WKB GeoArrow null's own CRS with its canonical to_json() form — stringifying the wrapper produced StringCrs(...), which ST_SetCRS rejects as PROJJSON. Unwrap 0-d structured masked containers to their record form before the generic mask check, which itself raises on structured dtypes, covering direct and Literal-wrapped assignment for all three mask states.
…askedRecords recursion Rebuild a large-WKB GeoArrow scalar's type on Binary storage — keeping its CRS and edge metadata — before the one-element-array resolution, since SedonaDB's WKB importer rejects LargeBinary storage outright; the typed routing had regressed what previously normalized through the plain scalar path. Unwrap 0-d structured masked containers through the base MaskedArray view: MaskedRecords' own [()] returns another 0-d MaskedRecords and recursed until RecursionError, while the base view yields the record form for the field/mask normalizer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
This is the first slice of the column assignment / arithmetic / dissolve work drafted in #1184, which grew too large to review as one change. It is being split into four focused PRs — assignment, dissolve, arithmetic, temporal support — that build on each other, so each can be reviewed on its own terms.
This PR adds
gdf[key] = valueto the experimentalsedonadb-geopandaspackage:Seriesassignment. ASeriesread from the same frame can be assigned back (renamed or replacing a column). ASeriesfrom a different frame, a bare SedonaDB expression, and array-likes are rejected with actionable messages: none of them records a usable row provenance, and each previously failed obscurely or would silently write wrong values. Because assignment rebinds the frame, aSeriesread before an assignment is stale and is rejected the same way.pandas.NAbroadcast to every row, via a shared scalar classifier/normalizer (is_scalar/normalize_scalar).TypeErrorbeats a subtly wrong stored value.sedonadbfloor to 0.4.1, the first released version withDataFrame.mutate, which assignment is built on.Testing
80 tests (58 new), run with warnings promoted to errors against both a source-built current main and the released sedonadb 0.4.1 in a clean environment (covering pandas 2.x and 3.x). Ruff lint/format clean.