Add mid/high level interface for HDF5 Dimension Scale

[fergu]

Opening this as a stub issue for adding a mid/high level interface to the Dimension Scale functions. I meant to do this a while back when the low level library calls were added, but I never got around to it. See also #720. I will also add a short blurb on what HDF5 dimension scales are useful for in a reply to this issue for anyone who is unfamiliar.

Here are some current ideas for things to implement. This is currently closer to a train of thought rather than anything set in stone, so I'd welcome feedback on how this could/should be changed:

High-level interface:

• Add a subtype of the HDF5.Dataset type called HDF5.Scale for cases where a Dataset is a dimension scale. This seems like the easiest way to add this functionality to the existing interface without breaking changes, but I'm open to suggestions.
• Add checks for existing dimension scales during reading of an HDF5.Dataset and returning them alongside the dataset. Alternatively, add a function to check if a dataset has scales (HDF5.has_scales or HDF5.dimensions_with_scales?) and read the scale given a dataset, a la HDF5.read_dimension_scale( ds::HDF5.Dataset, dim::Integer)
• Add a method to write dimension scales using the Dictionary interface as well as the existing write_dataset calls.

Mid-level implementations of low level library calls:

• Add HDF5.is_scale(), which should return true if a supplied HDF5.Dataset (or HDF5.Scale) is a dimension scale
• Add HDF5.attach_scale() and HDF5.detach_scale() to attach/detach a dimension scale from a supplied HDF5.Dataset
• Add HDF5.set_label() and HDF5.get_label() to assign a label to a dimension scale. This might be ambiguous without the HDF5.Scale datatype though, so it might be best to exclude this if HDF5.Scale also isn't a good idea.
• Add HDF5.set_scale() to convert an existing HDF5.Dataset to a dimension scale. Interestingly, there does not appear to be a corresponding unset_scale library call, though it seems like the only thing that makes a dataset a scale is the attributes, so HDF5.unset_scale() might just need to delete those attributes.
• Add HDF5.get_num_scales() to return the number of scales attached to a given HDF5.Dataset
• Add HDF5.dimensions_with_scales() to return a Vector{Integer} of dimensions with attached scales
• Add HDF5.get_scale_name() to return the name (label) attached to a scale, if there is one
• Add HDF5.is_attached() to return true if a supplied HDF5.Dataset (or HDF5.Scale) is attached to a dataset as a scale.

[fergu]

For a bit of context on what dimension scales do for anyone unfamiliar, in order to help the discussion (this blurb is written by me, not from HDF5 docs or anything, so take it with a grain of salt and all that)

HDF5 Dimension Scales are basically just a way to attach coordinate information to a given axis of a dataset. If I am given a new HDF5 file that someone else made, and I want to know the coordinate information associated with an axis of a dataset, I can just query the dimension scale associated with that axis, and it will give me a dataset with the corresponding coordinate information. This is much smoother than trying to infer what other dataset in the file is meant to be the coordinate data for that axis based on names or context or an email from the creator of the file. This additionally allows multiple datasets to share a single dimension scale, with all of those datasets pointing to a single piece of data in the file (as opposed to copies of identical data scattered around the file). In other words, these are another tool in helping make HDF5 files "self-describing".

Practically, dimension scales are just regular HDF5 Datasets with some extra attributes added to track where they are being used. You can write an HDF5 dataset to file, and then use the HDF5 library function h5ds_set_scale() to specify that the dataset is a scale. This adds a few attributes to the new scale to indicate things like the "name" of the scale (which is different than the path of the scale), and a list of datasets that a given scale is attached to. The handy thing about that last point is that attaching a scale to a dataset (using h5ds_attach_scale()) also adds a link to the scale as an attribute of the target dataset. You can read that attribute/link, and it will return an HDF5.Dataset (well, currently an hid_t, but that's what this task aims to fix) directly, without having to find a name/parse a path or anything like that.

[bertini97]

Hey, this would be great. Any chance it's being worked on?

[mkitti]

I have to admit that this is the first time that I'm thinking about it.

My initial impression is that this could be implemented or prototyped in another package where HDF5.jl is a dependency and it does need to be implemented in HDF5.jl initially. I guess the question then is whether the low-level functions are a private or public API.

[fergu]

The dimension scale API is public (https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_s__u_g.html). HDFGroup has it as part of a "high level API", alongside a few other more specific use cases like images or tables, or various optimizations (https://support.hdfgroup.org/documentation/hdf5/latest/_r_m.html).

There's probably a reasonable discussion to be had about whether the high level API should be part of this package since HDFGroup does split it out from the main API. Though, the high level API is part of a standard HDF5 installation so it's not packaged separately either.

If this functionality were to be implemented in a separate package with HDF5.jl as a dependency, what would the eventual pathway be there? It seems like if every high level API were to be split into its own package, the result would be lots of versions of HDF5.jl for a user to have to install/using in order to pull in those specific functionalities.

[mkitti]

The dimension scale API is public (https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_s__u_g.html). HDFGroup has it as part of a "high level API", alongside a few other more specific use cases like images or tables, or various optimizations (https://support.hdfgroup.org/documentation/hdf5/latest/_r_m.html).

The question is not about the C library but rather if the Julia HDF5.API module should be used by external packages directly.

There's probably a reasonable discussion to be had about whether the high level API should be part of this package since HDFGroup does split it out from the main API. Though, the high level API is part of a standard HDF5 installation so it's not packaged separately either.

This is a separate question about the high level C API, specifically libhdf5_hl. I would lean towards automatically generating bindings via an approach similar to https://github.com/mkitti/LibHDF5.jl/blob/main/gen/bind_generator.jl

If this functionality were to be implemented in a separate package with HDF5.jl as a dependency, what would the eventual pathway be there? It seems like if every high level API were to be split into its own package, the result would be lots of versions of HDF5.jl for a user to have to install/using in order to pull in those specific functionalities.

The main advantage of a separate package is that development can occur on this feature independently of the development cycle of this package. It's been more than a year since we've managed a release of this package, and there's quite some development that needs to occur.

I'm not sure if having smaller packages for high level features is necessarily a bad thing. If we needed to at the end of the day, we could just create a meta (i.e. HDF5Complete.jl) package that depends on all of them if needed. We could alternatively merge them into this package at a later date once these are mature.

Another advantage of a separate package is increased flexibility to bring in new dependencies. If I understand correctly, it might be useful to integrate a package like Unitful.jl together with this feature. I would be somewhat reluctant to do that with HDF5.jl but a HDF5DimensionScales.jl would have full flexibility to do so.

[bertini97]

I'm not sure I follow. Using the low level API this package provides, I was able to correctly assign a dimension scale. I think we just need to implement a high level API on top of that. Where is the need for another package?

On a side note, lest I forget, I think iteration should be aware of whether a dataset is a scale through isscale and ignore it.

[mkitti]

If you create a pull request into HDF5.jl, it might take a year for us to release a version of this package that includes these changes since we already have many breaking changes lined up. I would probably not be able to merge your changes until version 0.18.1 which might get released towards the end of the year if not next year. Also the changes would not be available to those who want to use version 0.17.x of HDF5.jl. The problem is that the development cycles for HDF5.jl are quite long.

Since you likely do not depend on breaking changes since the last release, you could create a new package that only requires 0.17 rather than the upcoming 0.18. You could release this code tomorrow, and then release an updated version next week. If you really wanted to, we could host the new package in this repository under a subdirectory. With a new package you can release the code much sooner than it would get released here.

[fergu]

That's a good point, I can definitely see where you're coming from and I think it makes sense. I'll give some thought to how this might be implemented in its own package.

Just to take this discussion a bit further (and to be clear, I'm not necessarily advocating for this - just thinking out loud) - given the complexity of HDF5, would splitting even this package out into a few sub packages make things any more maintainable from your perspective?

For example, have HDF5.jl basically be a metapackage. Then you might have HDF5Core.jl (essentially, whatever minimal implementation is needed to interface with an HDF5 file), HDF5HighLevel.jl - so on and so forth. Maybe things could even be split out a bit more granularly than this (along the general splits in the HDF5 docs) if the interplay between different aspects isn't too intertwined - e.g., HDF5Compression.jl, HDF5DimensionScales.jl, HDF5Image.jl etc etc.

Though, like I said - just kind of thinking out loud. This kind of idea might be a lot of work for maybe not a ton of benefit.

[mkitti]

That is roughly the general plan. Compression has already been largely spun out.