Skip to content

Commit 131ae73

Browse files
mihiarcclaude
andcommitted
Release v0.1.1 - OGC GeoParquet compliance
This release fixes a critical issue where the package was producing invalid GeoParquet files that couldn't be read by standard geospatial tools. Major changes: - Complete rewrite to produce OGC GeoParquet v1.0.0 compliant files - Added GeoPandas dependency for proper GeoParquet generation - Geometry now stored as WKB (binary) instead of WKT (text) - CRS information properly preserved in geo metadata - Removed legacy EnhancedGDBConverter entirely - All output files can now be read by GeoPandas, DuckDB Spatial, QGIS, etc. Breaking changes: - Removed use_legacy_converter parameter (all output is now GeoParquet) - GDBConverter is now an alias for GeoParquetConverter This ensures full interoperability with the geospatial ecosystem and follows the official GeoParquet specification. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent d5f40ca commit 131ae73

14 files changed

Lines changed: 1194 additions & 988 deletions

CHANGELOG.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,30 @@ All notable changes to the ESRI Converter project will be documented in this fil
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [0.1.1] - 2025-06-22
9+
10+
### Added
11+
- **OGC GeoParquet Compliance**: Now produces valid GeoParquet files according to the OGC GeoParquet v1.0.0 specification
12+
- New `GeoParquetConverter` class replacing the old non-compliant converter
13+
- GeoPandas dependency for proper GeoParquet generation
14+
- Verification scripts for testing GeoParquet compliance
15+
16+
### Changed
17+
- Complete rewrite of the converter to produce OGC-compliant GeoParquet files
18+
- Geometry storage changed from WKT text to WKB binary format
19+
- CRS information is now properly preserved in geo metadata
20+
- Updated documentation to reflect GeoParquet compliance
21+
22+
### Removed
23+
- Legacy `EnhancedGDBConverter` that produced non-standard parquet files
24+
- `use_legacy_converter` parameter (all output is now GeoParquet compliant)
25+
26+
### Fixed
27+
- Output files can now be read by standard GeoParquet readers (GeoPandas, DuckDB Spatial, QGIS, etc.)
28+
- Added required geo metadata to parquet file headers
29+
- Geometry data now stored in standard `geometry` column
30+
- Fixed interoperability with the broader geospatial ecosystem
31+
832
## [0.1.0] - 2025-01-28
933

1034
### Added

CLAUDE.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
ESRI Converter is a modern Python package for converting ESRI Geodatabase (GDB) files to OGC-compliant GeoParquet format. Built for 2025 with performance and developer experience in mind, using GeoPandas, Polars, Rich, PyArrow, Fiona, and Shapely. The package now produces valid GeoParquet files that can be read by all standard geospatial tools.
8+
9+
## Key Commands
10+
11+
### Development Setup
12+
```bash
13+
# Create and activate virtual environment
14+
uv venv
15+
source .venv/bin/activate
16+
17+
# Install in development mode
18+
uv pip install -e ".[dev]" # Development dependencies
19+
uv pip install -e ".[all]" # All optional dependencies
20+
```
21+
22+
### Code Quality
23+
```bash
24+
# Format code
25+
uv run black esri_converter/
26+
27+
# Lint code
28+
uv run ruff check esri_converter/
29+
30+
# Type check
31+
uv run mypy esri_converter/
32+
33+
# Run tests (when implemented)
34+
uv run pytest
35+
uv run pytest --cov=esri_converter
36+
```
37+
38+
### Documentation
39+
```bash
40+
# Install docs dependencies
41+
python scripts/docs.py install
42+
43+
# Serve docs locally (http://127.0.0.1:8000)
44+
python scripts/docs.py serve
45+
46+
# Build documentation
47+
python scripts/docs.py build
48+
```
49+
50+
### Publishing
51+
```bash
52+
# Full test publish workflow
53+
python scripts/publish.py full-test
54+
55+
# Full live publish workflow (requires PYPI_LIVE_TOKEN)
56+
python scripts/publish.py full-live
57+
```
58+
59+
## Architecture Overview
60+
61+
### Module Structure
62+
```
63+
esri_converter/
64+
├── __init__.py # Public API exports
65+
├── api.py # High-level conversion functions
66+
├── exceptions.py # Custom exception hierarchy
67+
├── converters/ # Core conversion engines
68+
│ └── geoparquet_converter.py # GeoParquetConverter (OGC-compliant output)
69+
└── utils/
70+
├── formats.py # Format information and recommendations
71+
└── validation.py # Input validation functions
72+
```
73+
74+
### Core API Functions
75+
- `convert_gdb_to_parquet()` - Single GDB conversion with detailed metrics
76+
- `convert_multiple_gdbs()` - Batch conversion with aggregate results
77+
- `discover_gdb_files()` - Find GDB files in directories
78+
- `get_gdb_info()` - Non-destructive GDB inspection
79+
80+
### Key Design Patterns
81+
1. **Streaming Architecture**: Chunk-based processing for large datasets (default: 15,000 records/chunk)
82+
2. **Rich UI Integration**: Beautiful console output with progress bars, tables, and trees
83+
3. **Comprehensive Error Handling**: Custom exception hierarchy with context preservation
84+
4. **Lazy Evaluation**: Uses Polars for memory-efficient operations
85+
86+
### Exception Hierarchy
87+
```
88+
ESRIConverterError
89+
├── UnsupportedFormatError
90+
├── ValidationError
91+
├── ConversionError
92+
│ ├── SchemaError
93+
│ └── MemoryError
94+
└── FileAccessError
95+
```
96+
97+
## Development Notes
98+
99+
1. **Python Version**: Requires Python 3.10+
100+
2. **Package Manager**: Always use `uv` for Python operations
101+
3. **Testing**: Currently no unit tests - test scripts exist but formal test suite needs implementation
102+
4. **Type Safety**: Project uses strict mypy configuration
103+
5. **Code Style**: Ruff for linting and formatting with 100-character line length
104+
6. **Main Converter**: `GeoParquetConverter` (aliased as `GDBConverter` for compatibility)
105+
7. **Default Output**: Creates "geoparquet_output" directory if not specified
106+
107+
## Important Considerations
108+
109+
- The CLI is commented out in v0.1.0 - focus is on Python API
110+
- GDAL warnings are suppressed for cleaner output
111+
- Schema normalization handles inconsistent field types across chunks
112+
- Memory-efficient processing adapts based on dataset characteristics
113+
- Rich console output provides detailed progress and performance metrics
114+
- **GeoParquet Compliance**: All output files are OGC GeoParquet v1.0.0 compliant
115+
- **GeoPandas Dependency**: Required for proper GeoParquet output with WKB geometry storage
116+
- **Geometry Storage**: Uses WKB (Well-Known Binary) format for optimal performance

README.md

Lines changed: 46 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@ Modern tools for converting ESRI proprietary formats to open source formats. Bui
44

55
## 🚀 Features
66

7+
- **OGC GeoParquet Compliant**: Produces valid GeoParquet files readable by all standard tools
78
- **Large-Scale Processing**: Handle multi-GB GDB files with streaming and chunking
8-
- **Modern Stack**: Built with Polars, Rich, and PyArrow for maximum performance
9+
- **Modern Stack**: Built with GeoPandas, Polars, Rich, and PyArrow for maximum performance
910
- **Beautiful UI**: Rich progress bars, tables, and visual feedback
1011
- **Memory Efficient**: Process datasets larger than available RAM
1112
- **Robust Error Handling**: Comprehensive validation and error recovery
@@ -28,7 +29,7 @@ pip install esri-converter[duckdb,dev]
2829
## 🔧 Requirements
2930

3031
- Python 3.10+
31-
- Modern dependencies: Polars, Rich, Fiona, PyArrow, Shapely
32+
- Modern dependencies: GeoPandas, Polars, Rich, Fiona, PyArrow, Shapely
3233

3334
## 🎯 Quick Start
3435

@@ -37,10 +38,15 @@ pip install esri-converter[duckdb,dev]
3738
```python
3839
from esri_converter.api import convert_gdb_to_parquet
3940

40-
# Convert a single GDB file
41+
# Convert a single GDB file to OGC-compliant GeoParquet
4142
result = convert_gdb_to_parquet("data.gdb")
4243
print(f"Converted {result['total_records']:,} records")
4344
print(f"Output size: {result['output_size_mb']:.1f} MB")
45+
46+
# The output files are valid GeoParquet files that can be read by:
47+
# - GeoPandas: gpd.read_parquet("output.parquet")
48+
# - DuckDB Spatial: SELECT * FROM 'output.parquet'
49+
# - QGIS, ArcGIS Pro, and other GIS tools
4450
```
4551

4652
### Advanced Usage
@@ -84,7 +90,7 @@ print(f"Successfully converted {results['gdbs_converted']}/{results['total_gdbs'
8490

8591
#### `convert_gdb_to_parquet()`
8692

87-
Convert a File Geodatabase to GeoParquet format.
93+
Convert a File Geodatabase to OGC GeoParquet format.
8894

8995
**Parameters:**
9096
- `gdb_path` (str | Path): Path to the .gdb file
@@ -213,16 +219,16 @@ print(f"Estimated output size: {sizes['parquet']:.1f} MB")
213219

214220
```
215221
esri_converter/
216-
├── __init__.py # Main package exports
217-
├── api.py # Clean API functions
218-
├── exceptions.py # Custom exceptions
222+
├── __init__.py # Main package exports
223+
├── api.py # Clean API functions
224+
├── exceptions.py # Custom exceptions
219225
├── converters/
220226
│ ├── __init__.py
221-
│ └── gdb_converter.py # Core conversion logic
227+
│ └── geoparquet_converter.py # OGC GeoParquet converter
222228
└── utils/
223229
├── __init__.py
224-
├── formats.py # Format information
225-
└── validation.py # Input validation
230+
├── formats.py # Format information
231+
└── validation.py # Input validation
226232
```
227233

228234
### Key Components
@@ -232,6 +238,36 @@ esri_converter/
232238
3. **Utilities** (`utils/`): Validation, format info, and helper functions
233239
4. **Exception Handling** (`exceptions.py`): Comprehensive error types
234240

241+
## 🗺️ GeoParquet Compliance
242+
243+
ESRI Converter produces **OGC GeoParquet v1.0.0** compliant files that are compatible with the entire geospatial ecosystem.
244+
245+
### What is GeoParquet?
246+
247+
GeoParquet is an open standard that adds geospatial capabilities to Apache Parquet files. Our output files:
248+
249+
- ✅ Can be read by GeoPandas, DuckDB Spatial, QGIS, and other GIS tools
250+
- ✅ Include proper geo metadata according to the specification
251+
- ✅ Store geometries as WKB (Well-Known Binary) for optimal performance
252+
- ✅ Preserve CRS (Coordinate Reference System) information
253+
- ✅ Support all geometry types (Point, LineString, Polygon, etc.)
254+
255+
### Verifying GeoParquet Output
256+
257+
```python
258+
import geopandas as gpd
259+
260+
# Read the converted GeoParquet file
261+
gdf = gpd.read_parquet("output/my_layer.parquet")
262+
263+
# The file contains:
264+
# - Geometry column with proper spatial data
265+
# - CRS information preserved from source
266+
# - All attributes from the original GDB
267+
print(f"CRS: {gdf.crs}")
268+
print(f"Bounds: {gdf.total_bounds}")
269+
```
270+
235271
## 🔧 Technical Details
236272

237273
### Performance Optimizations

esri_converter/__init__.py

Lines changed: 16 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,12 @@
1212
- Robust error handling and type safety
1313
1414
Example:
15-
>>> from esri_converter import GDBConverter
16-
>>> converter = GDBConverter()
17-
>>> converter.convert_gdb("data.gdb", output_format="geoparquet")
15+
>>> from esri_converter import convert_gdb_to_parquet
16+
>>> result = convert_gdb_to_parquet("data.gdb")
17+
>>> print(f"Converted {result['total_records']} records")
1818
"""
1919

20-
from importlib.metadata import version, PackageNotFoundError
20+
from importlib.metadata import PackageNotFoundError, version
2121

2222
try:
2323
__version__ = version("esri-converter")
@@ -26,29 +26,26 @@
2626
__version__ = "unknown"
2727

2828
# Core API functions
29-
from .api import (
30-
convert_gdb_to_parquet,
31-
convert_multiple_gdbs,
32-
discover_gdb_files,
33-
get_gdb_info
34-
)
29+
from .api import convert_gdb_to_parquet, convert_multiple_gdbs, discover_gdb_files, get_gdb_info
3530

3631
# Core converter classes
37-
from .converters.gdb_converter import EnhancedGDBConverter
38-
from .converters.gdb_converter import EnhancedGDBConverter as GDBConverter # Alias for backward compatibility
39-
40-
# Utility functions
41-
from .utils.formats import list_supported_formats, get_format_info
42-
from .utils.validation import validate_gdb_file, validate_output_path
32+
from .converters.geoparquet_converter import GeoParquetConverter
33+
from .converters.geoparquet_converter import (
34+
GeoParquetConverter as GDBConverter, # Alias for backward compatibility
35+
)
4336

4437
# Core exceptions
4538
from .exceptions import (
39+
ConversionError,
4640
ESRIConverterError,
4741
UnsupportedFormatError,
4842
ValidationError,
49-
ConversionError,
5043
)
5144

45+
# Utility functions
46+
from .utils.formats import get_format_info, list_supported_formats
47+
from .utils.validation import validate_gdb_file, validate_output_path
48+
5249
__all__ = [
5350
"__version__",
5451
# Core API functions
@@ -58,7 +55,7 @@
5855
"get_gdb_info",
5956
# Converters
6057
"GDBConverter",
61-
"EnhancedGDBConverter",
58+
"GeoParquetConverter",
6259
# Utilities
6360
"list_supported_formats",
6461
"get_format_info",
@@ -75,4 +72,4 @@
7572
__author__ = "Your Name"
7673
__email__ = "your.email@example.com"
7774
__license__ = "MIT"
78-
__description__ = "Modern tools for converting ESRI proprietary formats to open source formats"
75+
__description__ = "Modern tools for converting ESRI proprietary formats to open source formats"

0 commit comments

Comments
 (0)