feat(zarr): open a Zarr store from a local folder, and make the selector work - #1469
Merged
Conversation
…tor work GeoLibre could only render a Zarr store behind a URL, and the Zarr panel's Selector (JSON) field did nothing to a layer already on the map. **Add Zarr Layer dialog.** Add Data → Zarr Layer now opens a dialog rather than the on-map panel, mirroring Add NetCDF / HDF: pick a remote store or a folder on disk, choose a variable, and set each non-spatial dimension. The on-map panel is unchanged and still reachable from Controls → Zarr Layer. The dimension pickers offer the dimension's real **coordinate values**, read from the store with zarrita, not indices: `@carbonplan/zarr-layer` matches a selector by value, so on a `month` axis of 1-12 December is `12` and an index would be off by one. An axis that cannot be enumerated (no coordinate array, or more values than a dropdown should hold) falls back to a typed value. **Local folders.** A Zarr store is a directory tree, and the renderer accepts any zarrita `Readable`, so `ZarrDirectoryStore` maps a store key onto a path under the chosen folder. Reading the folder is the host's job: the desktop app uses the Tauri folder dialog (whose `recursive: true` grant covers the picked subtree), and a Chromium browser uses `showDirectoryPicker()`. Firefox and Safari implement neither, so the button explains that rather than opening a dialog that cannot deliver. Note the issue expected this to be desktop-only; the File System Access API does grant recursive read on a picked directory, so the web build gets it too where the browser supports it. A local cube also binds to the Time Slider: its URL is only an identifier, so the CF `units`/`calendar` are read out of the folder instead of walked over HTTP (`addZarrRasterLayer` gains `store` and `readTimeAttributes` for this). `readZarrStoreMetadata` lists a store's renderable arrays from v2 `.zmetadata`, a v3 consolidated `zarr.json`, or — where the transport can list, which means local disk — a bounded walk of the nodes. Coordinate arrays are left out by rank, and a pyramid's variable is offered once while keeping the path its coordinates are siblings of. **Selector.** Fixed upstream in maplibre-gl-components 0.28.2, which applies a Selector (JSON) edit to the live layers; 0.28.3 additionally reads a store-backed layer's spatial metadata through the store, so a local store no longer logs three failed requests for its own pseudo-URL. Fixes #1457
Contributor
📝 WalkthroughWalkthroughAdds local-folder Zarr loading across browser and Tauri environments, exposes directory-store and local-reader APIs, integrates local stores and CF time metadata with rendering, and wires folder picking into the desktop runtime. ChangesZarr support
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🔍 PR preview
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/geolibre-desktop/src/components/layout/AddZarrDialog.tsx`:
- Around line 93-96: Extract the duplicated leading-dimension computation from
the top-level leadingDims memo and loadCoordinates into a shared helper such as
getLeadingDims accepting a ZarrStoreVariable. Replace both inline dims/shape
checks and slicing expressions with calls to that helper, preserving the current
[] fallback and slicing behavior.
- Around line 132-138: Update applyLoadedStore to guard against store.variables
being empty before accessing first.name or calling loadCoordinates. Surface the
existing friendly no-variables error message, matching the NetCDF dialog’s
errorNoVariables behavior, and leave the normal variable-loading path unchanged.
In `@apps/geolibre-desktop/src/i18n/locales/it.json`:
- Line 661: Update the folderHelp translation in
apps/geolibre-desktop/src/i18n/locales/it.json at lines 661-661 to mention
zarr.json alongside .zmetadata and .zgroup. Apply the same guidance update to
apps/geolibre-desktop/src/i18n/locales/ja.json at lines 647-647,
apps/geolibre-desktop/src/i18n/locales/ka.json at lines 661-661,
apps/geolibre-desktop/src/i18n/locales/ko.json at lines 647-647,
apps/geolibre-desktop/src/i18n/locales/nl.json at lines 661-661, and
apps/geolibre-desktop/src/i18n/locales/pt.json at lines 661-661.
In `@packages/plugins/src/plugins/zarr-coordinates.ts`:
- Around line 119-127: Update createHttpZarrStore to stop passing headers
through the deprecated FetchStore overrides option. Use zarrita’s custom fetch
handler to merge the supplied headers into each request while preserving the
normalized URL and default behavior when headers are absent; verify this works
with the pinned zarrita 0.7.1 API.
In `@packages/plugins/src/plugins/zarr-directory-store.ts`:
- Around line 150-151: Update localZarrStoreUrl so each picked folder produces a
unique local-store URL by incorporating an opaque per-folder identifier, while
preserving the encoded folder name for display. Ensure callers provide or derive
that identifier consistently so duplicate folder names do not share
control-state keys or overwrite one another.
In `@packages/plugins/src/plugins/zarr-store-metadata.ts`:
- Around line 99-141: Update the error-selection logic in readZarrStoreMetadata
so a store with consolidated metadata but no renderable arrays throws the “No
renderable (2-D or higher) arrays found” error even when options.listEntries is
absent. Distinguish consolidatedV2 or rootV3 presence from the non-consolidated
HTTP fallback before entering the !options.listEntries branch, and add
regression coverage for a coordinate-only .zmetadata read without listEntries.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: d235013e-2b6d-44c1-af71-1622a61d29b7
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (29)
apps/geolibre-desktop/package.jsonapps/geolibre-desktop/src/components/layout/AddZarrDialog.tsxapps/geolibre-desktop/src/components/layout/TopToolbar.tsxapps/geolibre-desktop/src/i18n/locales/ar.jsonapps/geolibre-desktop/src/i18n/locales/de.jsonapps/geolibre-desktop/src/i18n/locales/en.jsonapps/geolibre-desktop/src/i18n/locales/es.jsonapps/geolibre-desktop/src/i18n/locales/fr.jsonapps/geolibre-desktop/src/i18n/locales/hi.jsonapps/geolibre-desktop/src/i18n/locales/id.jsonapps/geolibre-desktop/src/i18n/locales/it.jsonapps/geolibre-desktop/src/i18n/locales/ja.jsonapps/geolibre-desktop/src/i18n/locales/ka.jsonapps/geolibre-desktop/src/i18n/locales/ko.jsonapps/geolibre-desktop/src/i18n/locales/nl.jsonapps/geolibre-desktop/src/i18n/locales/pt.jsonapps/geolibre-desktop/src/i18n/locales/ru.jsonapps/geolibre-desktop/src/i18n/locales/tr.jsonapps/geolibre-desktop/src/i18n/locales/zh.jsonapps/geolibre-desktop/src/lib/zarr-directory-picker.tsdocs/plugin-api.mdpackages/plugins/package.jsonpackages/plugins/src/index.tspackages/plugins/src/plugins/maplibre-components.tspackages/plugins/src/plugins/zarr-coordinates.tspackages/plugins/src/plugins/zarr-directory-store.tspackages/plugins/src/plugins/zarr-store-metadata.tstests/zarr-directory-store.test.tstests/zarr-store-metadata.test.ts
- Give each picked folder a unique `local-zarr:` identifier. The Zarr control keys per-URL state (and the host refcounts in-flight adds) by this string, so two folders both named `data.zarr` collided and the second add could patch the first layer. A per-call sequence number goes in the query string, leaving the path — which the layer is named from — as the folder name. - Stop blaming consolidation when a *consolidated* store simply holds no renderable array: the "consolidate the store's metadata" advice now fires only when there really is no consolidated document. Covered by a test. - Set request headers through zarrita's `fetch` handler rather than the `overrides` option, which the installed @zarrita/storage marks deprecated. - Name `zarr.json` alongside `.zmetadata`/`.zgroup` in the folder-picker help, in every locale: it is the Zarr v3 root marker, and listing only the v2 ones would have a user reject a valid v3 folder. - Extract the duplicated leading-dimension slicing in the dialog into a shared `leadingDimsOf` helper. Not changed: the empty-`variables` guard in `applyLoadedStore`. `readZarrStoreMetadata` returns only when it found at least one renderable array and otherwise throws, so the array is never empty there.
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/geolibre-desktop/src/i18n/locales/es.json (1)
647-649: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAdd the Zarr kind header strings under the namespace the dialog reads.
AddDataDialog.tsxresolves the kind header throughaddData.kind.zarr.labelandaddData.kind.zarr.description, but these additions define onlyaddData.zarr.titleandaddData.zarr.description. Consequently, the Zarr entry point can show an untranslated key or fallback text for its header. Add localizedkind.zarr.label/descriptionentries while retainingaddData.zarrfor the dedicated dialog content.
apps/geolibre-desktop/src/i18n/locales/es.json#L647-L649: addaddData.kind.zarr.labeland.description.apps/geolibre-desktop/src/i18n/locales/fr.json#L647-L649: addaddData.kind.zarr.labeland.description.apps/geolibre-desktop/src/i18n/locales/hi.json#L647-L649: addaddData.kind.zarr.labeland.description.apps/geolibre-desktop/src/i18n/locales/id.json#L633-L635: addaddData.kind.zarr.labeland.description.apps/geolibre-desktop/src/i18n/locales/it.json#L647-L649: addaddData.kind.zarr.labeland.description.apps/geolibre-desktop/src/i18n/locales/ja.json#L633-L635: addaddData.kind.zarr.labeland.description.apps/geolibre-desktop/src/i18n/locales/ka.json#L647-L649: addaddData.kind.zarr.labeland.description.apps/geolibre-desktop/src/i18n/locales/ko.json#L633-L635: addaddData.kind.zarr.labeland.description.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/geolibre-desktop/src/i18n/locales/es.json` around lines 647 - 649, Add localized addData.kind.zarr.label and addData.kind.zarr.description entries, using the existing Zarr translations where appropriate, while retaining the existing addData.zarr entries for dialog content. Apply this in apps/geolibre-desktop/src/i18n/locales/es.json (647-649), fr.json (647-649), hi.json (647-649), id.json (633-635), it.json (647-649), ja.json (633-635), ka.json (647-649), and ko.json (633-635).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@apps/geolibre-desktop/src/i18n/locales/es.json`:
- Around line 647-649: Add localized addData.kind.zarr.label and
addData.kind.zarr.description entries, using the existing Zarr translations
where appropriate, while retaining the existing addData.zarr entries for dialog
content. Apply this in apps/geolibre-desktop/src/i18n/locales/es.json (647-649),
fr.json (647-649), hi.json (647-649), id.json (633-635), it.json (647-649),
ja.json (633-635), ka.json (647-649), and ko.json (633-635).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 944d102e-8b88-499c-8aeb-d9135043504e
📒 Files selected for processing (22)
apps/geolibre-desktop/src/components/layout/AddZarrDialog.tsxapps/geolibre-desktop/src/i18n/locales/ar.jsonapps/geolibre-desktop/src/i18n/locales/de.jsonapps/geolibre-desktop/src/i18n/locales/en.jsonapps/geolibre-desktop/src/i18n/locales/es.jsonapps/geolibre-desktop/src/i18n/locales/fr.jsonapps/geolibre-desktop/src/i18n/locales/hi.jsonapps/geolibre-desktop/src/i18n/locales/id.jsonapps/geolibre-desktop/src/i18n/locales/it.jsonapps/geolibre-desktop/src/i18n/locales/ja.jsonapps/geolibre-desktop/src/i18n/locales/ka.jsonapps/geolibre-desktop/src/i18n/locales/ko.jsonapps/geolibre-desktop/src/i18n/locales/nl.jsonapps/geolibre-desktop/src/i18n/locales/pt.jsonapps/geolibre-desktop/src/i18n/locales/ru.jsonapps/geolibre-desktop/src/i18n/locales/tr.jsonapps/geolibre-desktop/src/i18n/locales/zh.jsonpackages/plugins/src/plugins/zarr-coordinates.tspackages/plugins/src/plugins/zarr-directory-store.tspackages/plugins/src/plugins/zarr-store-metadata.tstests/zarr-directory-store.test.tstests/zarr-store-metadata.test.ts
…ialog The previous commit pointed Add Data -> Zarr Layer at a new dialog, which orphaned the on-map Zarr panel: `openZarrLayerPanel` had exactly one caller, so the panel became unreachable rather than relocated. Put it back, and move the local-folder support to where it belongs. The panel now grows a "Browse folder" button of its own, upstream in maplibre-gl-components 0.29.0, so every consumer of that package gets local Zarr rather than only GeoLibre. The host supplies the picker, because reading a folder needs a filesystem API a map plugin cannot have: `pickZarrDirectory` is registered through `setZarrLocalStoreProvider` at startup, and only where a folder dialog exists (the desktop app, or a browser with the File System Access API), so elsewhere the panel shows no button at all. `AddZarrDialog` and the modules that existed only to serve it are gone: `zarr-store-metadata` and `zarr-coordinates` (with the zarrita dependency and the `addData.zarr` catalog in all 16 locales), since the panel's own Fetch button now lists a folder's variables through the store. The directory store stays, as does its metadata reader: a local cube's URL is an identifier rather than an address, so the Time Slider reads its CF units out of the folder, keyed by the identifier the picker minted. What that trades away is the dialog's per-dimension pickers, which offered a dimension's real coordinate values. Slicing is the Selector (JSON) field again — which at least now applies to a layer already on the map (0.28.2). Fixes #1457
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/plugins/src/plugins/maplibre-components.ts`:
- Around line 2093-2101: Update the Zarr layer lifecycle around
zarrLocalStoreProvider to track each local URL by layer and remove its reader
from zarrLocalStoreReaders when the layer is removed. Apply the cleanup in both
the control-removal and store-removal paths, while preserving the shared URL
used by the layer and reader lookup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 54121662-efa3-473e-b3b1-eda455cf7bfa
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (10)
apps/geolibre-desktop/package.jsonapps/geolibre-desktop/src/components/layout/TopToolbar.tsxapps/geolibre-desktop/src/hooks/usePlugins.tsapps/geolibre-desktop/src/lib/zarr-directory-picker.tsdocs/plugin-api.mdpackages/plugins/package.jsonpackages/plugins/src/index.tspackages/plugins/src/plugins/maplibre-components.tspackages/plugins/src/plugins/zarr-directory-store.tstests/zarr-directory-store.test.ts
- Release a local folder's reader once no layer is backed by it. Every pick added an entry to `zarrLocalStoreReaders` and nothing ever removed one, so repeated add/remove cycles retained both the memory and the capability to read those folders. Prune on each of the two Zarr removal paths, and again before a new folder is taken — which also covers a folder the user browsed to and never added, since no layer removal would account for that one.
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/plugins/src/plugins/maplibre-components.ts (1)
2089-2091: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winPrune readers when unregistering the provider.
This branch disables future folder picking but leaves unused entries in
zarrLocalStoreReaders. If the integration is deactivated after a folder was selected and its layer removed, the directory reader—and its filesystem capability—remains retained until another provider call happens. CallpruneZarrLocalStoreReaders()before returning; live-layer readers will still be preserved.Based on learnings, plugin-owned runtime resources must be fully removed when the integration is deactivated.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/plugins/src/plugins/maplibre-components.ts` around lines 2089 - 2091, Update the !provider branch in the provider-unregistration flow to call pruneZarrLocalStoreReaders() before returning, after clearing zarrLocalStoreProvider. Preserve live-layer readers while removing unused directory readers and their retained filesystem capabilities when the integration is deactivated.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@packages/plugins/src/plugins/maplibre-components.ts`:
- Around line 2089-2091: Update the !provider branch in the
provider-unregistration flow to call pruneZarrLocalStoreReaders() before
returning, after clearing zarrLocalStoreProvider. Preserve live-layer readers
while removing unused directory readers and their retained filesystem
capabilities when the integration is deactivated.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: ec980000-ea83-4b9a-beea-bdeed744166e
📒 Files selected for processing (1)
packages/plugins/src/plugins/maplibre-components.ts
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1457.
GeoLibre could only render a Zarr store behind a URL, and the Zarr panel's Selector (JSON) field did nothing to a layer already on the map.
Local folders
A Zarr store is a directory tree, and the renderer accepts any zarrita
Readable, soZarrDirectoryStorejust maps a store key onto a path under the chosen folder. The Zarr panel grows a "Browse folder" button next to the Zarr URL, upstream inmaplibre-gl-components0.29.0 — so every consumer of that package gets local Zarr, not only GeoLibre, and Add Data → Zarr Layer still opens the same panel it always did.Reading the folder is the host's job, since it needs a filesystem API a map plugin cannot have: the desktop app uses the Tauri folder dialog (whose
recursive: truegrant covers the picked subtree), and a Chromium browser usesshowDirectoryPicker(). Firefox and Safari implement neither, so the button simply does not appear rather than opening a dialog that cannot deliver.The issue expected this to be desktop-only. The File System Access API does grant recursive read on a picked directory, so the web build gets it too where the browser supports it.
A local cube also binds to the Time Slider: its recorded URL is an identifier rather than an address, so the CF
units/calendarare read out of the folder instead of walked over HTTP.Selector
Fixed upstream across three releases, all consumed here:
_update*(); the selector was read only at add time. Debounced and serialized, since re-slicing fetches chunks andsetSelectoris async.<url>/zarr.jsonand friends, firing three failed requests at a pseudo-URL and detecting nothing. This also silently affected Cloud NetCDF.fetchVariablesdid not reconcile the panel's chosen variable, so Add Layer could ask a fresh store for the previous store's variable.Verification
Driven in the real app with Playwright, in both light and dark themes:
air_temperature_nc4.nc(real NCEP/NCAR reanalysis, 2920 steps, genuine CFhours since 1800-01-01) — picked, variables fetched through the store, rendered, and bound to the Time Slider showing Jan 2013 → 2014.One caveat: Playwright cannot open the OS folder dialog, so
showDirectoryPickerwas stubbed with a fake directory handle served from the real data directory. Everything below the handle — store reads, metadata, chunk decode, render — ran on real bytes, but the picker call itself was not exercised in-browser and the Tauri path was not driven end to end.Summary by CodeRabbit
addZarrLayer, updated selector semantics, and documented local-folder browsing behavior.