Skip to content

Prediction time columns serialize inconsistently across formats #42

Description

@fegue

birdnet prediction outputs are not serialized consistently across save methods.

Problem

Two friction points:

  1. Parquet

    • start_time and end_time are written as Arrow halffloat
    • Python reads them correctly
    • R Arrow reads raw float16 bit patterns instead of seconds
  2. CSV

    • start_time and end_time are serialized as "HH:MM:SS.xx" strings
    • in-memory outputs use numeric seconds

Expected

Saved outputs should be predictable across formats and languages.

  • Parquet should store time columns in an interoperable numeric type
  • CSV should either preserve numeric seconds or be explicitly treated as a human-readable export

Observed

In memory:

arr = prediction.to_structured_array()
arr["start_time"][:3]
# array([0., 3., 6.], dtype=float16)

Parquet schema

prediction.to_arrow_table().schema
# start_time: halffloat
# end_time: halffloat

R read-back

arrow::read_parquet("test.parquet", as_data_frame = TRUE)
# start_time
# 0
# 16896
# 17920

Expected values were 0, 3, 6.

CSV output:

00:00:00.00
00:00:03.00
00:00:06.00

Likely cause

start_time / end_time inherit float16 and are passed to Arrow as pa.float16(). That produces Parquet halffloat, which is not read correctly by R Arrow.

Proposed fix

Upcast start_time and end_time to at least float32 before Arrow table creation.

Possibly also upcast confidence when it is float16, for consistency.

Why this matters

The same prediction currently becomes:

  • numeric seconds in memory
  • formatted strings in CSV
  • correct numeric values in Python Parquet
  • corrupted-looking values in R Parquet

That makes cross-language round-tripping unreliable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions