Usage Guide

This guide covers how to use the overview generation tool, including CLI options, group structure configurations, and customization.

Quick Start

# 1. Export your Arraylake token
export ARRAYLAKE_TOKEN='ema_<my-token>'

# 2. Initialize the overview level (creates empty arrays)
python src/create_overviews.py \
    --arraylake-repo-name your-org/your-repo \
    --downsample-factor 4 \
    --initialize

# 3. Process tiles (computes and writes downsampled data)
python src/create_overviews.py \
    --arraylake-repo-name your-org/your-repo \
    --downsample-factor 4

CLI Reference

python src/create_overviews.py --help

Usage: create_overviews.py [OPTIONS]

  Create overviews at a specific downsample factor for an Arraylake dataset.

Options:
  --arraylake-repo-name TEXT      Name of the Arraylake repo containing the
                                  dataset to create overviews for.  [required]
  --downsample-factor INTEGER     Downsample factor (e.g., 16 for 16x
                                  downsampling). Determines output group name.
                                  [required]
  --data-bbox <FLOAT FLOAT FLOAT FLOAT>...
                                  Bounding box where data exists (min_x,
                                  min_y, max_x, max_y). If not specified,
                                  processes all tiles (full extent).
  --initialize                    Initialize the overview level (create empty
                                  arrays with fill values). Run this once
                                  before processing.
  --default-resampling [mean|median|mode|nearest|min|max]
                                  Default resampling method for all variables.
                                  [default: mean]
  --resampling-override TEXT      Override resampling for specific variables.
                                  Format: 'variable=method'. Can be specified
                                  multiple times. Methods: mean, median, mode,
                                  nearest, min, max.
  --variables TEXT                Specific variables to create overviews for.
                                  If not specified, processes all data
                                  variables.
  --chunk-size INTEGER            Chunk size for overview arrays (spatial
                                  dimensions).  [default: 1800]
  --batch-size INTEGER            Jobs per batch before committing.  [default:
                                  100]
  --limit INTEGER                 Limit the number of jobs (for testing).
  --source-factor INTEGER         Source level factor to compute from (e.g., 4
                                  to compute 16x from 4x). If not specified,
                                  automatically selects the best available
                                  source level.
  --debug                         Enable debug logging.
  --skip-existing                 Skip jobs that have already been completed
                                  (tracked via logs/overview_completed_*.txt).
                                  Useful for resuming interrupted runs.
  --native-group TEXT             Group path where native resolution data
                                  lives.  [default: /]
  --overview-template TEXT        Template for overview group names. Use
                                  {factor} as placeholder.  [default:
                                  {factor}x]
  --x-coord TEXT                  Name of X coordinate array (auto-detected if
                                  not specified).
  --y-coord TEXT                  Name of Y coordinate array (auto-detected if
                                  not specified).
  --time-coord TEXT               Name of time coordinate array (auto-detected
                                  if not specified).

Group Structure Options

Use --native-group and --overview-template to configure the hierarchy.

Parent/Child Model (default)

Native data lives at the root (/), overviews in subgroups:

/                        # Root: multiscales metadata
├── var1, var2, ...      # Native resolution data (GeoZarr-compliant)
├── 4x/                  # 4x overview (GeoZarr-compliant)
│   └── var1, var2, ...
└── 16x/                 # 16x overview (GeoZarr-compliant)
    └── var1, var2, ...

# Default options:
--native-group /
--overview-template {factor}x

To put all overviews in a overviews/ group, use:

--native-group /
--overview-template overviews/{factor}x

Note on xarray DataTree: When opening as a DataTree, xarray attempts to align dimensions with the same name across the hierarchy. Since each level has different coordinate values, this can cause issues. Consider using different dimension names per level (e.g., latitude_4x).

Sibling Model

Each level (including native) lives in its own group:

/                        # Root: multiscales metadata only
├── 0/                   # Native resolution (GeoZarr-compliant)
│   └── var1, var2, ...
├── 1/                   # First overview (GeoZarr-compliant)
│   └── var1, var2, ...
└── 2/                   # Second overview (GeoZarr-compliant)
    └── var1, var2, ...

--native-group 0
--overview-template {factor}x

Resampling Methods

Method	Composable	Description	Use Case
`mean`	Yes	Average of pixels in block	Continuous data
`min`	Yes	Minimum value in block	Conservative estimates
`max`	Yes	Maximum value in block	Peak detection
`median`	No	Median of pixels in block	Noisy continuous data
`mode`	No	Most common value in block	Categorical data
`nearest`	No	Top-left pixel of block	Preserving exact values

Composable methods can be computed from a previous overview level (e.g., 16x from 4x). Non-composable methods should be computed from native resolution for accurate results.

# Set default for all variables
--default-resampling mean

# Override for specific variables
--resampling-override "land_cover=mode"

Mixed Resampling Methods

If your dataset has variables with different composability requirements, you'll need to run processing multiple times with different source factors:

# 1. Initialize once (creates all variable arrays)
python src/create_overviews.py \
    --arraylake-repo-name org/repo \
    --downsample-factor 16 \
    --initialize

# 2. Process composable variables from 4x (faster)
python src/create_overviews.py \
    --arraylake-repo-name org/repo \
    --downsample-factor 16 \
    --source-factor 4 \
    --variables temperature precipitation humidity \
    --default-resampling mean

# 3. Process non-composable variables from native (accurate)
python src/create_overviews.py \
    --arraylake-repo-name org/repo \
    --downsample-factor 16 \
    --source-factor 1 \
    --variables land_cover soil_type \
    --default-resampling mode

How Processing Works

Two-Stage Workflow

Stage 1: Initialize (--initialize)

Creates the overview group with empty arrays filled with fill values
Establishes the complete grid structure
Updates multiscales metadata on root group
Run once per overview level

Stage 2: Process (default)

Computes and writes downsampled data
Can filter by bounding box (--data-bbox)
Parallelizes across Coiled workers

Job Execution

Job Generation: Dataset is divided into spatial tiles (based on --chunk-size). One job per (tile × variable) combination. Each job processes all time steps for that tile/variable.
Filtering: If --data-bbox is provided, only tiles intersecting that region are processed.
Distributed Execution: Jobs run on Coiled workers. Each worker:
- Forks the Icechunk repository
- Reads source data (from native or a previous overview level)
- Iterates over time steps to manage memory
- Downsamples using xarray's coarsen operation
- Writes results to the forked repo
Batch Commit: After each batch (--batch-size jobs):
- Collects all successful forks
- Merges them into the main branch
- Commits atomically
Pipelining: While one batch commits, the next batch starts processing.
Fault Tolerance: Completed batches are already committed. Use --skip-existing to resume interrupted runs.

Building Multi-Level Pyramids

Run the tool multiple times with increasing factors:

# Build 4x from native
python src/create_overviews.py --arraylake-repo-name org/repo --downsample-factor 4 --initialize
python src/create_overviews.py --arraylake-repo-name org/repo --downsample-factor 4

# Build 16x from 4x (auto-selected, or specify --source-factor 4)
python src/create_overviews.py --arraylake-repo-name org/repo --downsample-factor 16 --initialize
python src/create_overviews.py --arraylake-repo-name org/repo --downsample-factor 16

# Build 64x from 16x
python src/create_overviews.py --arraylake-repo-name org/repo --downsample-factor 64 --initialize
python src/create_overviews.py --arraylake-repo-name org/repo --downsample-factor 64

The tool automatically selects the best source level (highest existing factor that evenly divides the target). Override with --source-factor if needed.

Customization

Coiled Configuration

Worker resources are configured in src/coiled_app.py:

REGION = "us-west-2"              # AWS region for workers
NUM_WORKERS = 50                  # Number of parallel workers
KEEP_ALIVE = "30m"                # Keep workers alive between batches
SPOT_POLICY = "spot_with_fallback"
CPU_PER_WORKER = 2
MEMORY_PER_WORKER = "8 GiB"

Auto-Detection

Coordinate names are auto-detected in priority order:

GeoZarr spatial:dimensions metadata
CF conventions (axis attribute, standard_name)
Common patterns (longitude/lon/x, latitude/lat/y)

Override with --x-coord, --y-coord, --time-coord if needed.

Tuning Parameters

Parameter	Default	Trade-offs
`--chunk-size`	1800	Larger = fewer tiles, more memory per job
`--batch-size`	100	Larger = faster, but more work at risk if interrupted

FilesExpand file tree

USAGE.md

Latest commit

History