Conversation
There seems to be transient issue with requesting geomag data in that sometimes chunks return nan values when there are data, so trying retries.
Add guards to prevent unintended or redundant writes to HDF5 dataset attributes. Introduce detection of file mode and can_write check, plus _metadata_needs_write and _values_equal helpers (handling numpy scalars/arrays) to compare in-memory metadata with on-disk attributes before writing. Import read_attrs_to_dict and to_numpy_type helpers. Update TransferFunctionGroup to construct EstimateDataset with write_metadata=False to avoid side-effect writes when only reading, and add minor formatting cleanups.
Add guards to skip unnecessary HDF5 attribute writes when metadata is unchanged and to avoid writes on read-only files. Implemented _can_write, _values_equal and _metadata_needs_write helpers on dataset classes and BaseGroup to compare existing attrs (including numpy arrays / numpy scalars) to expected values before create/modify calls. Update write_metadata paths to use attr.modify only when values differ and to skip entirely if file is not writable. Also tidy some dataset slicing formatting and adjust imports. Add tests (tests/groups/test_metadata_write_guards.py) to assert unchanged metadata is not rewritten for BaseGroup, ChannelDataset, FeatureChannelDataset, FCChannelDataset and EstimateDataset.
Introduce repack_hdf5 in mth5.utils.h5_tools to copy HDF5 file-level attributes, groups, datasets, and their attributes into a new file (mirroring h5repack behavior). The function validates input existence, prevents accidental overwrite unless overwrite=True, and returns the output Path. Add tests to verify structure/data/attributes are copied, missing source raises FileNotFoundError, and overwrite behavior raises FileExistsError unless allowed. Uses h5py and pathlib.
Add a guard in BaseGroup before writing metadata that inspects the HDF5 file mode (via getattr on self.hdf5_group.file.mode). If the file is not opened writable (no 'w' or '+'), the code now logs a warning and returns early to avoid attempting writes on read-only files and raising errors.
Update .github/workflows/tests.yml to run pytest with coverage only when matrix.python-version is 3.13; for other Python versions the job runs tests without coverage. Also add an if condition to the Codecov upload step so coverage is uploaded only for the 3.13 job, avoiding duplicate or unnecessary coverage collection/uploads from other matrix entries.
kujaku11
changed the title
Update to Geomag clients to make sure requested data is complete.
INTERMAGNET Support
Contributor
|
🚀 Version bumped to v0.6.8 and changelog updated. Ready to merge! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There is a known issue with hdf5 when rewriting metadata attributes the file size will increase because new locations are created but the old ones are not deleted. one solution is to check metadata against old medadata and only write if anything changed. This PR applies that logic so there are no unnecessary rewrites to limit file size increase. Another method is to create a new file from the existing file which if bloated will reduce the file size. that is also included in mth5/utils.
tests are updated as is the action to only tun codecov on one python version which speeds up the tests.