From PR #1573 review: We are relying on parquet readers to perform type widening (e.g., reading INT as LONG, FLOAT as DOUBLE) when reading checkpoint statistics.
Problem
The kernel's schema comparison code (can_widen_to) assumes that parquet readers will handle type widening:
- Integer widening: byte → short → int → long
- Float widening: float → double
However, this is not currently documented as a requirement in the parquet reader contract/interface.
Request
1. Document in the `ParquetHandler` trait what type widening behavior is expected/required
2. Clarify whether this is a hard requirement or best-effort behavior
3. Consider adding tests to verify type widening behavior in the default parquet implementation
Related
- PR #1573: feat: Add function to check if schema supports parsed stats")
From PR #1573 review: We are relying on parquet readers to perform type widening (e.g., reading INT as LONG, FLOAT as DOUBLE) when reading checkpoint statistics.
Problem
The kernel's schema comparison code (
can_widen_to) assumes that parquet readers will handle type widening:- Integer widening: byte → short → int → long
- Float widening: float → double
However, this is not currently documented as a requirement in the parquet reader contract/interface.
Request
Related