Skip to content

fix(python/sedonadb): serialize pyogrio reader lifecycles - #1187

Draft
jiayuasu wants to merge 1 commit into
apache:mainfrom
jiayuasu:fix/pyogrio-reader-lifecycle
Draft

fix(python/sedonadb): serialize pyogrio reader lifecycles#1187
jiayuasu wants to merge 1 commit into
apache:mainfrom
jiayuasu:fix/pyogrio-reader-lifecycle

Conversation

@jiayuasu

Copy link
Copy Markdown
Member

What

  • add an ExternalFormatSpec capability for formats that cannot safely consume multiple file readers concurrently
  • disable concurrent file reads for pyogrio and serialize schema inference and the complete reader lifecycle within each physical scan
  • drop the Arrow FFI stream before closing its owning Python context and releasing the scan permit
  • add regression tests for scan-local serialization and deterministic Python-context cleanup

Why

#1051 serialized pyogrio context entry and exit, but active Arrow streams could still overlap. Subsequent wheel runs continued to abort during multi-file and partitioned pyogrio reads, as tracked in #1072.

An unchanged CPython 3.12 comparison repeatedly ran the two affected tests in fresh processes and reproduced SIGABRT at iteration 72 during test_read_ogr_partitioned: run 32625781579.

This change keeps one scan-local permit from reader creation through consumption and native cleanup. Pyogrio multi-file scans therefore trade file-level parallelism for stability. Other external formats retain concurrent reads by default, and independent physical scans do not share a process-wide lock.

Verification

  • cargo fmt --all -- --check
  • cargo test -p sedona-datasource -p sedonadb --lib (9 + 11 tests passed)
  • cargo clippy -p sedona-datasource -p sedonadb --lib --no-deps -- -D warnings
  • local pyogrio and external-read modules (22 tests passed)
  • pinned CPython 3.12 stress run:
    • full pyogrio module: 18/18 passed
    • 500 same-process repetitions and 500 fresh-process repetitions
    • 2,002 executions of the two affected tests with no abort, hang, timeout, or test failure
    • run 32628586406
  • all five core wheel platforms passed on the combined verification branch: run 32627022432

Closes #1072.

Follow-up to #924 and #1051. Replacing the Python reader with the native GDAL reader remains tracked in #1092.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python wheel builds are failing with handler timeout

1 participant