Describe the bug
In report.py. there is this code:
df_stats_detailed = (
df_stats_detailed.withColumnRenamed("source_dq_row_count", "total_records")
.withColumnRenamed("source_dq_status", "status")
.withColumnRenamed("source_dq_actual_row_count", "valid_records")
.withColumnRenamed("source_dq_error_row_count", "failed_records")
.withColumn(
"success_percentage",
(safe_cast(ansi_enabled, "valid_records", "double") / safe_cast(ansi_enabled, "total_records", "double")) * 100,
)
)
which does not protect against a zero denominator in the same way that the querydq calculation does (it uses nullif).
To Reproduce
Steps to reproduce the behavior:
Run a unit test with source_dq_row_count = 0
Expected behavior
The df_stats_detailed calculation should protect against division by zero in the same way that querydq does.
Note
For tests to test this: The code for success_percentage uses nullif(greatest(...), 0) which returns NULL instead, then coalesce(..., lit(0)) catches it. This should be verified with a test that exercises the both-zero case.
Describe the bug
In report.py. there is this code:
which does not protect against a zero denominator in the same way that the
querydqcalculation does (it usesnullif).To Reproduce
Steps to reproduce the behavior:
Run a unit test with source_dq_row_count = 0
Expected behavior
The df_stats_detailed calculation should protect against division by zero in the same way that querydq does.
Note
For tests to test this: The code for success_percentage uses nullif(greatest(...), 0) which returns NULL instead, then coalesce(..., lit(0)) catches it. This should be verified with a test that exercises the both-zero case.