All notable changes to ValidateX are documented here. This project follows Semantic Versioning.
- Airflow Integration fix:
ValidateXOperatornow correctly computes quality score viaresult.compute_quality_score()and resolves failed columns fromresult.resultslist (previously used non-existent attributes) - 20 unit tests for
ValidateXOperatorcovering CSV, Parquet, gate pass/fail, HTML report generation, unsupported formats, and XCom output USER_GUIDE.md: comprehensive step-by-step guide covering all engines, CLI, YAML config, Airflow, CI/CD, custom expectations, and troubleshooting- PySpark data quality example (
examples/spark_data_quality.py): end-to-end e-commerce orders dataset with 15+ checks and HTML/JSON reports MANIFEST.in: ensures README, LICENSE, CHANGELOG, and templates are bundled in source distributions
ValidateXOperator.execute()bug:result.score→result.compute_quality_score()ValidateXOperator.execute()bug:result.columns.items()→result.resultslistValidateXOperator.execute()bug:result.passed_count/result.failed_count→len(passed)/len(failed)from filteredresult.results- Airflow integration no longer raises
ImportErrorat module load time; error is deferred to operator instantiation so non-Airflow users are unaffected
python_requiresupdated from>=3.8to>=3.9(aligns with pandas 2.x)requirements.txtcleaned up: removedpysparkas hard dependency (it is an optional extra —pip install "validatex[spark]")- Added
sqlalchemytorequirements.txtcore deps - Expanded PyPI classifiers: Python 3.12, OS Independent, Scientific/Engineering
- Push-Down SQL Native Validation engine (SQLAlchemy-based)
- Data Drift Detection with Population Stability Index (PSI)
- Apache Airflow
ValidateXOperatorintegration - Sequential / time-series expectations (
increasing,decreasing) - Conditional / cross-row expectations (
null_when,not_null_when,satisfy) - 20 advanced column expectations (email, URL, UUID, ISO date, phone, JSON, etc.)
- YAML / JSON declarative suite configuration
- CLI interface (
validatex validate,profile,run,init,list-expectations) - Data profiler with auto-suggest (
DataProfiler)
- Minor packaging fix for PyPI metadata
- Initial public release
- 36 column-level expectations
- 5 table-level expectations
- 4 aggregate / cross-column expectations
- Pandas and PySpark dual-engine support
- Severity modeling (Critical / Warning / Info)
- Weighted data quality score (0–100)
- Column health summary with mini bar charts
- Modern dark-theme HTML reports
- JSON report export