Skip to content

[BUG] Divide By Zero Unhandled in df_stats_detailed Calculation #314

Description

@yankomo

Describe the bug
In report.py. there is this code:

            df_stats_detailed = (
                df_stats_detailed.withColumnRenamed("source_dq_row_count", "total_records")
                .withColumnRenamed("source_dq_status", "status")
                .withColumnRenamed("source_dq_actual_row_count", "valid_records")
                .withColumnRenamed("source_dq_error_row_count", "failed_records")
                .withColumn(
                    "success_percentage",
                    (safe_cast(ansi_enabled, "valid_records", "double") / safe_cast(ansi_enabled, "total_records", "double")) * 100,
                )
            )

which does not protect against a zero denominator in the same way that the querydq calculation does (it uses nullif).

To Reproduce
Steps to reproduce the behavior:
Run a unit test with source_dq_row_count = 0

Expected behavior
The df_stats_detailed calculation should protect against division by zero in the same way that querydq does.

Note
For tests to test this: The code for success_percentage uses nullif(greatest(...), 0) which returns NULL instead, then coalesce(..., lit(0)) catches it. This should be verified with a test that exercises the both-zero case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions