Conversation
The index cached nine full-grid arrays: float64 copies of the cell centers, the corner mesh, per-cell (min, max) of the corners along each axis, and a seam mask. Only the corner mesh is a source of truth. The rest were read by `sel` for a narrow band of rows. Drop the cached centers, per-cell bounds, and seam mask. `sel` now derives per-cell bounds from the corner band it needs. The 1D per-row lat bounds come from 1D reductions of the corner rows. On a 3298x4500 curvilinear grid (RTOFS global 1/12 degree) this cuts retained index memory from 0.96 GB to 0.24 GB and peak memory during grid detection from 1.7 GB to 1.3 GB. `sel` for a tile costs ~6 ms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #287 +/- ##
=======================================
Coverage 81.42% 81.43%
=======================================
Files 31 31
Lines 6542 6538 -4
=======================================
- Hits 5327 5324 -3
+ Misses 1215 1214 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CurvilinearCellIndexcached nine full-grid arrays: float64 copies of the cell centers (X,Y), the corner mesh, per-cell(min, max)of the corners along each axis, and a seam mask. Only the corner mesh is a source of truth.selread the others for a narrow band of rows.This PR keeps the corner mesh and drops the rest:
_lat_mask_for_rowsderives per-cell bounds and the seam mask from the corner band it needs, using the existing_cell_min_maxhelper.row_y_min,row_y_max) come from 1D reductions of the corner rows.selreads the grid shape from the corner mesh.equalscompares corner meshes.The constructor signature does not change.
Context
The tiles service for
esip-workshop-2026OOM-killed on the root request for a global 1/12° RTOFS dataset (3298 × 4500 cells, float32 lat/lon). Grid detection alone retained 0.96 GB and peaked at 1.7 GB on a 2 GiB pod that already held another dataset.Measured on a synthetic grid of that shape:
selfor one tilePeak is still dominated by the float64 casts in
Curvilinear.from_datasetand the two-passinfer_interval_breaks. Keeping the input dtype for the corner mesh would roughly halve both numbers again. I left that for a separate PR so a precision regression is easy to bisect.Tests
tests/test_grids.py: 390 passed.test_property_global_render_no_transparent_tileon HEALPix,test_rectilinear_triangular_equivalency). Neither exercises the curvilinear index, and replaying the same falsifying examples againstmainfails identically, so they are pre-existing.ruff,ty, and pre-commit pass.[This is Claude Code on behalf of Deepak Cherian]
🤖 Generated with Claude Code