CI breakage with new zarr

[martindurant]

xref: zarr-developers/VirtualiZarr#677 (comment)

cc @d-v-b

We are seeing several test failures recently, presumably as a consequence of the dtype refactor in zarr.

Example output:

ValueError: No Zarr data type found that matches {'name': '<i1', 'object_codec_id': None}

Other errors to do with missing variables are probably a direct consequence of this.

[d-v-b]

that's the output of ZDType.to_json(zarr_format=2), which returns a dict with two keys: "name", which is the value that goes under the "dtype" key in .zarray, and "object_codec_id", which only matters for the object dtypes. You can probably unbreak your tests by pulling out the "name" value.

[d-v-b]

if you can show me some code I can probably say more

[martindurant]

Minimal reproducer:

file .zarray

{"chunks": [3], "compressor": null, "dtype": "<i1", "fill_value": 0, "filters": null, "order": "C", "shape": [3], "zarr_format": 2}

code

>>> import zarr
>>> z = zarr.open_array(".", zarr_version=2)
ValueError: No Zarr data type found that matches {'name': '<i1', 'object_codec_id': None}

[d-v-b]

Where did this metadata document come from? Because it's assigning endianness to a single-byte datatype. Zarr python 2.x would convert "<i1" to "|i1" before saving metadata, which is what zarr python 3.1 assumes.

>>> store = {}
>>> z = zarr.create(store=store, dtype='>i1', shape=(1,), mode='w')
>>> json.loads(store['.zarray'])["dtype"]
# '|i1'

[d-v-b]

If it helps unbreak things, we could be more permissible about the JSON values we consume, e.g., including ">i1", "<i1", even if the endianness is meaningless

[martindurant]

I can do "|" for that particular case, but the previous version was valid before, and this is presumably how numpy repr'd the dtype.

Here is another example, which loads with v3.0.8 for example, but not now:

{"chunks": [3], "compressor": null,
"dtype": [["var", "O"], ["xyz", ">i2", [2]]],
"fill_value": 0, "filters": null, "order": "C", "shape": [3], "zarr_format": 2}

[d-v-b]

this is presumably how numpy repr'd the dtype.

unfortunately no

>>> np.dtype('<i1').str
'|i1'

Zarr python 3.x didn't properly support structured data types before 3.1. Your example data can be read but not written:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr==3.0.10",
# ]
# ///


import zarr
import json

from zarr.core.buffer.cpu import Buffer

meta_dict = {"chunks": [3], "compressor": None,
"dtype": [["var", "O"], ["xyz", ">i2", [2]]],
"fill_value": 0, "filters": None, "order": "C", "shape": [3], "zarr_format": 2}
store = {'.zarray': Buffer.from_bytes(json.dumps(meta_dict).encode('utf-8'))}
z = zarr.open(store)
z[:] # this works
z[:] = z[:] # this errors

As of 3.1, for zarr v2 we also disallow structured data types that contain the numpy object dtype, because this is ambiguous. I do think we could add support for structured data types that specify arrays though.

[martindurant]

Your example data can be read but not written

That is kerchunk's niche, so totally fine

[d-v-b]

I think it's pretty central to the zarr data types that they support reading and writing. So I don't see how we can fit a "read only" data type like "[["var", "O"], ["xyz", ">i2", [2]]]" into that model.

The problem is specifically the "O" part. We can't write object arrays without knowing which special "object codec" to use, and the zarr v2 structured dtype representation doesn't support declaring an object codec.

But I think this could be solved on kerchunk's end by defining a special zarr data type that handles structured dtypes that contain object dtypes. You would need to register this data type, and define logic for constructing an instance of that data type from JSON like "[["var", "O"], ["xyz", ">i2", [2]]]", which should be rejected as unparseable by all the other data types. You would encounter an error at runtime if you tried to write, but reading should work. let me know if you want more advice on how to do this.

[martindurant]

I don't see how we can fit a "read only" data type

I'm not saying that you should have a special case, only that kerchunk doesn't care about the write part.

defining a special zarr data type that handles structured dtypes that contain object dtypes

I definitely don't know how to do this.
Note also, that ideally kerchunk does not need importing to read the metadata it generates. The import is sometimes necessary anyway to get the various codecs, but most datasets work with fsspec/zarr alone.

[d-v-b]

If this has a high impact, then I think we could consider defining a StructuredRO (structured read-only) data type, that does everything the structured data type does, except with the additional support for fields that are annotated as dtype "O". Users would encounter an error when trying to write these arrays, but reading should be fine

[martindurant]

It's hard for me to comment on the impact: kerchunk is trying to allow for the full flexibility of possible data types (in FITS tables, for this example). Simple types will account for >90% of cases for sure, but we kerchunk has for long been trying to account for the long tail of non-simple cases.

[d-v-b]

if I did start implementing something in zarr to address this, how would you recommend I test it for kerchunk compatibility?

[martindurant]

I would say the two above cases would take us a long way there. It should be possible to open those arrays and get an equivalent of NaNs out when there is no real data.

-edit-
I suppose there's no need to bother with the byteorder flag, it might only exist in one hard-coded string in one place in the kerchunk tests.