Supporting variable-length chunks for Zarr v3

[glwagner]

Is there a plan to support variable-length (per-axis irregular) chunks in Zarr v3?

The two relevant extensions are rectangular (ZEP 0003) and rectilinear (zarr-extensions) — both currently experimental. Reference impl: zarrs (Rust) has both.

Today Zarr.jl rejects anything but regular:

Zarr.jl/src/metadata3.jl

Lines 214 to 223 in 03270b2

	# Chunk Grid
	chunk_grid = d["chunk_grid"]
	if chunk_grid["name"] == "regular"
	chunks = Int.(chunk_grid["configuration"]["chunk_shape"])
	if length(shape) != length(chunks)
	throw(ArgumentError("Shape has rank $(length(shape)) which does not match the chunk_shape rank of $(length(chunks))"))
	end
	else
	throw(ArgumentError("Unknown chunk_grid of name, $(chunk_grid["name"])"))
	end

This will be useful for distributed output in Oceananigans with variably-sized partitions, eg when the simulation is divided into regions that communicate via MPI, but those regions have different sizes (because of continents) in order to balance the load between different nodes.

cc @simone-silvestri, @vopikamm, @giordano

[asinghvi17]

DiskArrays already supports rectangular and rectilinear grids as described in the links you posted, so in principle I don't see a problem. But we'd need to make sure the plumbing is done right.

[glwagner]

Claude noticed that, and says this:

Why this is more tractable than it looks

DiskArrays already has IrregularChunks and a GridChunks built on a per-axis ChunkVector, so the iteration / lookup layer that Zarr.jl already delegates to does not need to change. The work is mostly on the Zarr-side metadata and a few arithmetic helpers.

Sketch of changes

• Replace the chunks::NTuple{N,Int} field in MetadataV3 with a chunk-grid representation that can be either regular or irregular.
• Extend the chunk_grid parser in src/metadata3.jl to accept rectangular and/or rectilinear; mirror in JSON.lower.
• Generalize trans_ind / boundint / getchunkarray in ZArray.jl to ask the grid for per-chunk extents instead of doing arithmetic on a constant chunk size. The codec pipeline itself works off size(buffer) and shouldn't need changes.
• Sharding (ShardingCodec inner grid) probably stays regular for the first cut.