Zarr datasets catalog through a datasetScan

[aulemahal]

Hi!

I am exploring publishing Zarr datasets through our THREDDS server. Adding a Zarr through a <dataset ...> entry in the catalog works. The OpenDAP and DAP4 services still have difficulties with my data, but that seems mostly because of an issue with scalar variables , which is unrelated to TDS.

However, I have a large collection that is organized in structured folders. Instead of adding <dataset ...> entries for each Zarr dataset, I would like to use the datasetScan construct like we already do for nc files. I was unable to do that as it seems the scan requires dataset to be a file, whereas Zarr datasets are folders. What I tried is simply adding a filter entry for *.zarr .

Is there a way to circumvent that ?

Using zipped folders does work. But while accessing them through opendap fails (see linked issue), I was hoping to access our Zarr datasets using the HTTP file server directly and that is very slow for zipped dataset (as well as requires more code on the client side).

HTTP access example in Python:

import xarray
import zarr

# A zipped dataset
store = zarr.storage.FSStore(url="zip::https://thredds.server/thredds/fileServer/path/to/dataset.zarr.zip", mode="r")
ds = xr.open_zarr(store)

# A directory dataset
ds = xr.open_zarr("https://thredds.server/thredds/fileServer/path/to/dataset.zarr")

So the lines above do work. But the zipped version is very slow. And my question is more oriented on the catalog side: even though the last line works correctly, the dataset is not discoverable through the html and xml catalogs.

[lesserwhirls]

This is a very timely question, as I am planning on working on the issue of exposing Zarr datasets using a datasetScan-like construct (if not an enhanced datasetScan) this week. The scalar issue with our netCDF-Java support is what I am looking at first so that it can make it into the next release of netCDF-Java.

[lesserwhirls]

I've extended datasetScan to expose zarr (directory) datasets for both local and object store systems, and that will be in the next release of the TDS (this week). These datasets will get two catalog entries each - one as a catalogRef, which simply exposes the underlying directory structure like the current behavior, and a second, which is a direct access dataset. The HTTPServer access point on a zarr dataset can be used directly with xarray.

[FObersteiner]

I've just experimented with zarr a little bit, and since this all seems somewhat experimental, here are some comments.

The approach described in the OP seems to work basically, however

• additional dependencies were needed on the Python side; aiohttp, fsspec, requests (I did not use xarray[complete], this might cover those dependencies as well)
• FSStore class is a zarr-python v2 feature, with zarr-python v3 you'd have to use something like zarr.storage.FsspecStore.from_url, but I didn't get this to work (yet)
• reading from a zip archive on the TDS via the TDS fileServer works fine with mentioned dependency additions
• reading from a .zarr directory does not work for me (might be my TDS config, or just lack of know-how ^^)