Conversation
…r PostGIS ingestion
… debian trixie and gdal
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates geospatial ingestion/transformation away from in-memory geopandas/pandas to a PostGIS-native pipeline: ingestion streams data into PostGIS via ogr2ogr (GDAL), and transformations are built once as parameterized SQL (SQLAlchemy Core) for both preview and process paths. It also bumps the repo’s Python pin to 3.13 and updates the Airflow image build/dev tooling to a Debian Trixie-based setup that can ship newer GDAL.
Changes:
- Replace geopandas-based ingestion with
ogr2ogrsubprocess calls (file/URL/FTP/DB/OGC services) and update unit tests accordingly. - Introduce SQL-native transformation builder (
build_transformation_select) plus process (CTAS) and preview (bounded rows + GeoJSON) paths. - Upgrade Python pin to 3.13 and update Airflow tooling (local base image build + GDAL install + compose args + workflow).
Reviewed changes
Copilot reviewed 39 out of 42 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| pyrightconfig.json | Update static-analysis Python version to 3.13. |
| pyproject.toml | Update workspace requires-python to 3.13. |
| Makefile | Add Airflow base-image build target and related variables. |
| libs/data_manipulation/tests/test_transformation.py | Replace geopandas transformation tests with SQL compilation/assertion tests. |
| libs/data_manipulation/tests/test_ingestion.py | Replace geopandas ingestion tests with ogr2ogr command-building tests. |
| libs/data_manipulation/tests/test_column_actions.py | Refocus tests on SQL-level filtering/selection helpers. |
| libs/data_manipulation/src/data_manipulation/utils.py | Remove GeoDataFrame detection helper (geopandas dependency removal). |
| libs/data_manipulation/src/data_manipulation/transformation/transform.py | Remove in-memory transformation pipeline. |
| libs/data_manipulation/src/data_manipulation/transformation/transform_projection.py | Remove in-memory CRS relabel helper. |
| libs/data_manipulation/src/data_manipulation/transformation/transform_geom_point.py | Remove in-memory point-geometry builder. |
| libs/data_manipulation/src/data_manipulation/transformation/transform_encoding.py | Remove in-memory encoding transform. |
| libs/data_manipulation/src/data_manipulation/transformation/transform_columns.py | Remove in-memory rename/cast helpers (superseded by SQL). |
| libs/data_manipulation/src/data_manipulation/transformation/sql_transform.py | Add canonical SQL-native transformation builder + preview + CTAS process path. |
| libs/data_manipulation/src/data_manipulation/transformation/filter_sql.py | Factor out a reusable build_filter_clause helper and reuse it. |
| libs/data_manipulation/src/data_manipulation/ingestion.py | Implement ogr2ogr-based ingestion for all sources and supporting helpers. |
| libs/data_manipulation/src/data_manipulation/constants.py | Add default OGC SRS constant to avoid SRID 0 ingestion. |
| libs/data_manipulation/src/data_manipulation/init.py | Update public exports to new ingestion/SQL-transform APIs. |
| libs/data_manipulation/README.md | Document the new ogr2ogr + SQL-native design and GDAL requirements. |
| libs/data_manipulation/pyproject.toml | Bump Python pin; drop geopandas/pyarrow; add pyproj. |
| libs/data_manipulation/.python-version | Bump to Python 3.13. |
| docker/Dockerfile.airflow | Switch to Trixie-based Airflow base image; install newer GDAL via micromamba. |
| docker/compose.airflow.yaml | Wire in AIRFLOW_BASE_IMAGE build arg and bump Airflow version defaults. |
| docker/airflow-base/scripts/docker/keys/postgres.asc | Add upstream key material for building the Trixie Airflow base image. |
| docker/airflow-base/scripts/docker/keys/microsoft.asc | Add upstream key material for building the Trixie Airflow base image. |
| docker/airflow-base/scripts/docker/keys/mariadb.asc | Add upstream key material for building the Trixie Airflow base image. |
| docker/airflow-base/README.md | Document local build of Trixie Airflow base image. |
| ARCHITECTURE.md | Update architecture notes to reflect server-side SQL transformations and ogr2ogr ingestion. |
| apps/elt/pyproject.toml | Bump Python pin; update Airflow/provider versions. |
| apps/elt/dags/utils.py | Remove pandas dependency and add raw-cursor helper to return dict rows. |
| apps/elt/dags/task_groups/transformation.py | Replace read/transform/write with DB-native transform_staging_to_final CTAS. |
| apps/elt/dags/task_groups/ingestion.py | Add passing decrypted Basic Auth to OGC ingestion (GDAL config). |
| apps/elt/.python-version | Bump to Python 3.13. |
| apps/backend/src/api/routes/ingestion/staging.py | Replace geopandas preview logic with read_transformed_preview + SRID detection helper. |
| apps/backend/pyproject.toml | Bump Python pin; align Airflow client version; drop chardet. |
| apps/backend/Dockerfile | Bump base Python image to 3.13 and update uv python selection. |
| apps/backend/.python-version | Bump to Python 3.13. |
| .python-version | Bump repo default to Python 3.13. |
| .github/workflows/build-airflow-trixie-image.yml | Add workflow to build/push the Trixie Airflow base image. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+200
to
+201
| command_string = " ".join(command) | ||
| logger.debug(f"Running command: {command_string}") |
Comment on lines
+324
to
+326
| command_string = " ".join(command) | ||
| logger.debug(f"Running command: {command_string}") | ||
| # -------- |
Comment on lines
+398
to
+400
| command_string = " ".join(command) | ||
| logger.debug(f"Running command: {command_string}") | ||
|
|
Comment on lines
+263
to
+271
| start = time.time() | ||
|
|
||
| ingest_file_with_ogr2ogr(str(temp_file_path), table_name, engine, schema) | ||
|
|
||
| # Calculate the end time and time taken | ||
| end = time.time() | ||
| length = end - start | ||
|
|
||
| print("It took", length, "seconds.") |
Comment on lines
+6
to
+9
| AIRFLOW_VERSION ?= 3.2.2 | ||
| AIRFLOW_PYTHON_VERSION ?= 3.13.5 | ||
| AIRFLOW_BASE_IMAGE ?= georchestra/airflow-base:$(AIRFLOW_VERSION)-trixie | ||
| export AIRFLOW_VERSION |
| docker build \ | ||
| --build-arg BASE_IMAGE=debian:trixie-slim \ | ||
| --build-arg AIRFLOW_VERSION=3.2.2 \ | ||
| --build-arg AIRFLOW_PYTHON_VERSION=3.14.0 \ |
Comment on lines
+34
to
+35
| - name: Push airflow | ||
| run: docker push georchestra/airflow-base:latest |
Comment on lines
+37
to
+41
| - name: Push release | ||
| if: github.ref_type == 'tag' | ||
| run: | | ||
| docker tag georchestra/airflow-base:latest georchestra/airflow-base:${{ steps.version.outputs.VERSION }} | ||
| docker push georchestra/airflow-base:${{ steps.version.outputs.VERSION }} |
Comment on lines
+32
to
+40
| url = engine.url | ||
| pg_conn_parts = [ | ||
| f"host={url.host}", | ||
| f"port={url.port or 5432}", | ||
| f"dbname={url.database}", | ||
| f"user={url.username}", | ||
| f"password={url.password}", | ||
| ] | ||
| return "PG:" + " ".join(part for part in pg_conn_parts if part.split("=", 1)[1]) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.