[FIX] Added functions to scan and repair NetCDF files before loading

[mari-lage]

Description:

The purpose of this pull-request is to solve issue #85.

Some Seaglider basestation NetCDF files could not be opened with xarray.open_dataset(). The error raised was:
ValueError: unable to decode time units 'seconds since 1970-1-1 00:00:00'
This occurred during CF (Climate and Forecast) time decoding in xarray.

The issue was traced to specific variables (e.g. compass_timeouts_times_truck) that had:

• Data stored as strings/characters (e.g. b'1759139459.653,')
• Metadata indicating numeric time (e.g.: units = "seconds since 1970-1-1 00:00:00")

Because this variable with time units does not contain numerical values, the error was raised and the file could not be opened.

Modification:

I implemented a pre-processing repair step that scans all NetCDF files before loading.
For each file, it now disables automatic time decoding: xr.open_dataset(..., decode_times=False)
Then it identifies problematic variables, e.g., those that have a units attribute containing "since", or have a string/byte dtype (S, U, or O). For those variables, it now removes the misleading metadata and saves the .nc file. The original file is backed up as pxxxx_original.nc in an additional folder. All modifications are logged in a repair_log.txt file. The fix is conservative, i.e., there is no attempt to convert the string values into real timestamps, and only the metadata that is incorrect is removed. This ensures no accidental data corruption, full reversibility (as the original files are preserved), and minimal assumptions about the intended meaning of the data.

Checklist:

• I have followed the coding conventions.
• I have updated or added tests to cover my changes.
• I have updated the documentation if needed.
• I have run pytest to check that all tests pass.
• I have run pre-commit run --all-files to lint and format the code.

Related Issues / Pull Requests:

Link any related issues, pull requests, or discussions:

• Fixes Seaglider basestation NetCDF files could not be opened #85

[tillmrtz]

This is where I changed decode_timedelta = False to decode_times=False. I don't really know what other changes I have made but it worked for all datasets I tried (approx. 8 different ones). I wanted to implement for another issue, however it's taking some more time and better to implement sooner than later.

[mari-lage]

Ok, I removed decode_timedelta = False now, and it works for both datasets I have with the incorporate changes to identify the time unit problem.

[tillmrtz]

Changes for handling the time data looks fine to me.

[tillmrtz]

Issue with date format resolved hopefully