Skip to content

Commit 9078824

Browse files
bbestbbest
authored
Merge pull request #6 from CalCOFI/data-access-helpers-docs
Add Data Access + Matching Helpers pages; supersede API stub
2 parents 2664231 + 7230f60 commit 9078824

4 files changed

Lines changed: 512 additions & 16 deletions

File tree

‎_quarto.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,10 @@ book:
4949
- reports.qmd
5050
- apps.qmd
5151
- maps.qmd
52-
- api.qmd
5352
- db.qmd
53+
- data-access.qmd
54+
- helpers.qmd
55+
- api.qmd
5456
- portals.qmd
5557
- status.qmd
5658
- refs.qmd

‎api.qmd‎

Lines changed: 48 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,66 @@
11
# API
22

3-
The raw interface to the Application Programming Interface (API) is available at:
3+
::: {.callout-important}
4+
## Superseded by direct querying + `calcofi4r` helpers
5+
6+
The Postgres-backed Plumber API at [api.calcofi.io](https://api.calcofi.io) is
7+
**being phased out**. CalCOFI data is now published as versioned, public
8+
**Parquet** files that you query directly with DuckDB — no API server, no
9+
credentials, no rate limits.
10+
11+
- To query the data directly (SQL, R, Python), see **[Data Access](data-access.qmd)**.
12+
- For biological ↔ environmental matching (the old `zooplankton_biomass`,
13+
`itis_ichthyodata` and `ichthyodata` endpoints), see
14+
**[Matching Helpers](helpers.qmd)** — the `calcofi4r` functions
15+
`cc_match_zooplankton_biomass()`, `cc_match_ichthyo_by_taxon()` and
16+
`cc_match_ichthyo_by_name()`.
17+
18+
The endpoint notes below are retained as a historical reference and a map from
19+
each API endpoint to its replacement.
20+
:::
21+
22+
The raw interface to the legacy Application Programming Interface (API) is
23+
available at:
424

525
- [api.calcofi.io](https://api.calcofi.io)
626

7-
Here we will provide more guidance on how to use the API functions with documented input arguments, output results and examples of use.
27+
## Endpoint → replacement
28+
29+
| Legacy endpoint | Replacement |
30+
|---|---|
31+
| `/variables` | `calcofi4r::cc_list_measurement_types()` |
32+
| `/timeseries` | direct SQL aggregation — see [Data Access](data-access.qmd) |
33+
| `/cruises` | `calcofi4r::cc_read_cruise()` |
34+
| `/raster` | direct SQL + `calcofi4r::pts_to_rast_idw()` |
35+
| `/cruise_lines`, `/cruise_line_profile` | direct SQL against `ctd_cast` / `ctd_thin` |
36+
| `zooplankton_biomass` | `calcofi4r::cc_match_zooplankton_biomass()` ([Matching Helpers](helpers.qmd)) |
37+
| `itis_ichthyodata` | `calcofi4r::cc_match_ichthyo_by_taxon()` ([Matching Helpers](helpers.qmd)) |
38+
| `ichthyodata` | `calcofi4r::cc_match_ichthyo_by_name()` ([Matching Helpers](helpers.qmd)) |
39+
40+
## Legacy endpoint reference
841

9-
## `/variables`: get list of variables for timeseries
42+
### `/variables`: get list of variables for timeseries
1043

11-
Get list of variables for use in `/timeseries`
44+
Get list of variables for use in `/timeseries`.
1245

13-
## `/species_groups`: get species groups for larvae
46+
### `/species_groups`: get species groups for larvae
1447

15-
Not yet working. Get list of species groups for use with variables `larvae_counts.count` in `/timeseries`
48+
Not yet working. Get list of species groups for use with variables `larvae_counts.count` in `/timeseries`.
1649

50+
### `/timeseries`: get time series data
1751

18-
## `/timeseries`: get time series data
52+
### `/cruises`: get list of cruises
1953

20-
## `/cruises`: get list of cruises
54+
Get list of cruises with summary stats as CSV table for time (`date_beg`).
2155

22-
Get list of cruises with summary stats as CSV table for time (`date_beg`)
56+
### `/raster`: get raster map of variable
2357

24-
## `/raster`: get raster map of variable
58+
Get raster of variable.
2559

26-
Get raster of variable
60+
### `/cruise_lines`: get station lines from cruises
2761

28-
## `/cruise_lines`: get station lines from cruises
62+
Get station lines from cruises (with more than one cast).
2963

30-
Get station lines from cruises (with more than one cast)
64+
### `/cruise_line_profile`
3165

32-
## `/cruise_line_profile`
33-
Get profile at depths for given variable of casts along line of stations
66+
Get profile at depths for given variable of casts along line of stations.

‎data-access.qmd‎

Lines changed: 291 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,291 @@
1+
# Data Access
2+
3+
CalCOFI [database](db.qmd) releases are published as **Parquet** files on a
4+
public Google Cloud Storage (GCS) bucket. You can query them directly with
5+
[DuckDB](https://duckdb.org) — from R, Python, the DuckDB CLI, or any DuckDB
6+
client — with **no credentials, no API server, and no full download**. DuckDB
7+
reads only the columns and row groups a query actually touches, straight over
8+
HTTPS.
9+
10+
This page covers querying the release Parquet directly. For convenience-wrapped
11+
biological ↔ environmental matching, see [Matching Helpers](helpers.qmd).
12+
13+
## Where the data lives
14+
15+
Each release is a versioned folder:
16+
17+
```
18+
gs://calcofi-db/ducklake/releases/{version}/
19+
├── catalog.json # table list, row counts, "partitioned" flag
20+
├── relationships.json # primary keys + foreign keys
21+
├── RELEASE_NOTES.md
22+
└── parquet/
23+
├── {table}.parquet # single-file tables
24+
└── {table}/cruise_key=.../*.parquet # hive-partitioned tables
25+
```
26+
27+
- Public HTTPS base:
28+
`https://storage.googleapis.com/calcofi-db/ducklake/releases/{version}/parquet`
29+
- `{version}` is e.g. `v2026.05.14`; the current release pointer is at
30+
[`releases/latest.txt`](https://storage.googleapis.com/calcofi-db/ducklake/releases/latest.txt)
31+
and the table list at `releases/{version}/catalog.json`.
32+
- Most tables are **single-file** (`{table}.parquet`). The large CTD tables
33+
`ctd_thin` and `ctd_summary` are **hive-partitioned** by `cruise_key`
34+
(`catalog.json` flags these with `"partitioned": true`).
35+
36+
## Setup: the `httpfs` extension
37+
38+
DuckDB reads remote Parquet through its `httpfs` extension. Spatial queries
39+
(e.g. distances between casts and tows) also need `spatial`. Both are one-time
40+
installs, then loaded per session:
41+
42+
```sql
43+
INSTALL httpfs; LOAD httpfs;
44+
INSTALL spatial; LOAD spatial; -- only if you use ST_* functions
45+
```
46+
47+
## Single-file tables
48+
49+
A single-file table is just an HTTPS URL handed to `read_parquet()`:
50+
51+
```sql
52+
SELECT species_id, scientific_name, common_name, worms_id
53+
FROM read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/species.parquet')
54+
WHERE scientific_name = 'Sardinops sagax';
55+
```
56+
57+
Joins work the same way — name each table's URL. The biological hierarchy is
58+
`ichthyo` → `net` → `tow` → `site` (and `ichthyo` → `species`); the
59+
environmental hierarchy is `bottle_measurement` → `bottle` → `casts`:
60+
61+
```sql
62+
SELECT i.ichthyo_uuid, i.life_stage, i.tally, sp.scientific_name, t.time_start
63+
FROM read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/ichthyo.parquet') i
64+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/species.parquet') sp ON i.species_id = sp.species_id
65+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/net.parquet') n ON i.net_uuid = n.net_uuid
66+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/tow.parquet') t ON n.tow_uuid = t.tow_uuid
67+
WHERE sp.scientific_name = 'Sardinops sagax'
68+
LIMIT 10;
69+
```
70+
71+
## Hive-partitioned tables
72+
73+
`ctd_thin` and `ctd_summary` are partitioned into one folder per `cruise_key`.
74+
Reading them needs an **s3-style glob** (plain HTTPS URLs can't glob), so
75+
configure DuckDB's anonymous s3 access — GCS is s3-compatible:
76+
77+
```sql
78+
INSTALL httpfs; LOAD httpfs;
79+
SET s3_region = 'auto';
80+
SET s3_endpoint = 'storage.googleapis.com';
81+
SET s3_url_style = 'path';
82+
SET s3_access_key_id = '';
83+
SET s3_secret_access_key = '';
84+
85+
SELECT cruise_key, count(*) AS n
86+
FROM read_parquet(
87+
's3://calcofi-db/ducklake/releases/v2026.05.14/parquet/ctd_thin/**/*.parquet',
88+
hive_partitioning = true)
89+
GROUP BY cruise_key
90+
ORDER BY cruise_key;
91+
```
92+
93+
Because `cruise_key` is the partition column, filtering on it lets DuckDB
94+
**prune** whole partitions — a single-cruise query never opens the other ~96
95+
folders.
96+
97+
## From R
98+
99+
Use `DBI` + `duckdb` directly:
100+
101+
```r
102+
library(DBI)
103+
con <- dbConnect(duckdb::duckdb())
104+
dbExecute(con, "INSTALL httpfs; LOAD httpfs;")
105+
106+
base <- "https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet"
107+
d <- dbGetQuery(con, sprintf(
108+
"SELECT scientific_name, common_name, worms_id
109+
FROM read_parquet('%s/species.parquet')
110+
WHERE common_name ILIKE '%%sardine%%'", base))
111+
```
112+
113+
Or let the [`calcofi4r`](https://calcofi.io/calcofi4r) package register every
114+
release table as a view for you — it handles `httpfs`, the partitioned tables,
115+
and local caching:
116+
117+
```r
118+
# remotes::install_github("calcofi/calcofi4r")
119+
library(calcofi4r)
120+
121+
con <- cc_get_db() # latest release, tables as views
122+
DBI::dbListTables(con)
123+
124+
# lazy dbplyr against the remote parquet
125+
library(dplyr)
126+
tbl(con, "species") |> filter(scientific_name == "Sardinops sagax")
127+
128+
# or a one-off SQL query
129+
cc_query("SELECT count(*) FROM ichthyo")
130+
```
131+
132+
## From Python
133+
134+
```python
135+
import duckdb
136+
137+
con = duckdb.connect()
138+
con.sql("INSTALL httpfs; LOAD httpfs;")
139+
140+
base = "https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet"
141+
df = con.sql(f"""
142+
SELECT scientific_name, common_name, worms_id
143+
FROM read_parquet('{base}/species.parquet')
144+
WHERE common_name ILIKE '%sardine%'
145+
""").df()
146+
```
147+
148+
## Reproducibility
149+
150+
Because every query is plain SQL against immutable, versioned, public Parquet,
151+
a CalCOFI result is **reproducible by anyone** — pin the `{version}` and re-run
152+
the SQL.
153+
154+
The [Integrated App](https://app.calcofi.io/int) builds on this: its data
155+
**download bundle** ships a `query/` folder alongside the data:
156+
157+
```
158+
data/original/{bio,env}.csv ← query/{bio,env}.sql
159+
data/integrated/integrated_*.csv ← query/integrated_*.sql
160+
query/manifest.json release version, filters, GCS source URLs,
161+
per-file row counts + md5 checksums
162+
query/REPRODUCE.md DuckDB-CLI / Python / R re-run snippets
163+
```
164+
165+
Each `*.sql` file is fully interpolated, GCS-URL-based, and copy-paste runnable
166+
(prefixed with the `INSTALL`/`LOAD` it needs). Re-run `query/integrated_*.sql`
167+
in DuckDB and you get back exactly the rows in the matching `.csv` — the
168+
`manifest.json` md5 checksums let you confirm it. The same SQL is what
169+
[`calcofi4r::cc_match_bio_env()`](helpers.qmd) executes and attaches as
170+
`attr(x, "sql")`, so the app, the package, and a hand-written query are all the
171+
**same single source of truth**.
172+
173+
## Worked example: sardine larvae + temperature {#sec-worked-example}
174+
175+
The recurring example through these pages is *Pacific sardine
176+
(`Sardinops sagax`) larvae matched to CTD-bottle temperature, Q1 2018, with
177+
relaxed (5 km / 72 hr) matching*.
178+
179+
::: {.callout-note}
180+
Q1 **2018** — not a more recent year — because CTD-bottle environmental data in
181+
the current release ends 2021-05, while net-tow biological data runs later. Q1
182+
2018 has ample overlap of both.
183+
:::
184+
185+
Done as **direct SQL**, the query is a temporal interval join plus a spatial
186+
`ST_Distance_Sphere` filter. After the `INSTALL`/`LOAD` setup above, run the
187+
query below. This block is character-for-character what
188+
[`calcofi4r::cc_match_ichthyo_by_name()`](helpers.qmd) returns as
189+
`attr(d, "sql")` — the two produce the **identical 13 rows**. The
190+
[Integrated App](https://app.calcofi.io/int) download bundle builds
191+
`query/integrated_*.sql` with the same `cc_match_bio_env()` engine (its
192+
filter set is taxa + quarters + dates rather than `life_stage`, so a
193+
sardine / Q1 2018 bundle returns the egg + larva superset — same matching
194+
mechanics, same reproducibility):
195+
196+
```sql
197+
WITH bio AS (
198+
SELECT
199+
i.ichthyo_uuid::VARCHAR AS bio_id,
200+
t.time_start AS bio_datetime,
201+
s.longitude AS bio_lon,
202+
s.latitude AS bio_lat,
203+
n.std_haul_factor * i.tally / nullif(n.prop_sorted, 0) AS bio_value,
204+
sp.scientific_name,
205+
sp.common_name,
206+
sp.worms_id,
207+
i.life_stage,
208+
i.tally
209+
FROM read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/ichthyo.parquet') i
210+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/species.parquet') sp ON i.species_id = sp.species_id
211+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/net.parquet') n ON i.net_uuid = n.net_uuid
212+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/tow.parquet') t ON n.tow_uuid = t.tow_uuid
213+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/site.parquet') s ON t.site_uuid = s.site_uuid
214+
WHERE i.tally IS NOT NULL
215+
AND i.measurement_type IS NULL
216+
AND t.time_start IS NOT NULL
217+
AND s.longitude IS NOT NULL
218+
AND s.latitude IS NOT NULL
219+
AND sp.scientific_name IN ('Sardinops sagax')
220+
AND i.life_stage IN ('larva')
221+
AND t.time_start >= TIMESTAMP '2018-01-01'
222+
AND t.time_start <= TIMESTAMP '2018-03-31'
223+
),
224+
env AS (
225+
SELECT
226+
bm.bottle_measurement_id AS env_id,
227+
c.datetime_utc AS env_datetime,
228+
c.lon_dec AS env_lon,
229+
c.lat_dec AS env_lat,
230+
bm.measurement_value AS env_value,
231+
b.depth_m AS env_depth_m,
232+
bm.measurement_type AS measurement_type
233+
FROM read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/bottle_measurement.parquet') bm
234+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/bottle.parquet') b ON bm.bottle_id = b.bottle_id
235+
JOIN read_parquet('https://storage.googleapis.com/calcofi-db/ducklake/releases/v2026.05.14/parquet/casts.parquet') c ON b.cast_id = c.cast_id
236+
WHERE bm.measurement_type = 'temperature'
237+
AND bm.measurement_value IS NOT NULL
238+
AND c.datetime_utc IS NOT NULL
239+
AND c.lon_dec IS NOT NULL
240+
AND c.lat_dec IS NOT NULL
241+
AND c.datetime_utc >= TIMESTAMP '2018-01-01' - INTERVAL '72 hours'
242+
AND c.datetime_utc <= TIMESTAMP '2018-03-31' + INTERVAL '72 hours'
243+
),
244+
matched AS (
245+
-- temporal interval join: every env observation within ± max_time_hr
246+
SELECT
247+
bio.*,
248+
env.* EXCLUDE (env_lon, env_lat),
249+
abs(epoch(bio.bio_datetime) - epoch(env.env_datetime)) / 3600.0 AS time_diff_hr,
250+
ST_Distance_Sphere(
251+
ST_Point(bio.bio_lon, bio.bio_lat),
252+
ST_Point(env.env_lon, env.env_lat)) / 1000.0 AS dist_km
253+
FROM bio
254+
JOIN env
255+
ON env.env_datetime BETWEEN bio.bio_datetime - INTERVAL '72 hours'
256+
AND bio.bio_datetime + INTERVAL '72 hours'
257+
),
258+
within AS (
259+
-- spatial filter: keep pairs within max_dist_km
260+
SELECT * FROM matched
261+
WHERE dist_km <= 5
262+
),
263+
ranked AS (
264+
SELECT
265+
*,
266+
min(time_diff_hr) OVER (PARTITION BY bio_id) AS mn_time_diff_hr,
267+
min(dist_km) OVER (PARTITION BY bio_id) AS mn_dist_km
268+
FROM within
269+
)
270+
-- one row per bio observation (× measurement_type): env values aggregated
271+
SELECT
272+
* EXCLUDE (
273+
env_id, env_value, env_datetime, env_depth_m,
274+
time_diff_hr, dist_km, mn_time_diff_hr, mn_dist_km),
275+
count(*) AS n_env,
276+
avg(env_value) AS env_value,
277+
CASE WHEN count(*) = 1 THEN 0
278+
ELSE coalesce(stddev_samp(env_value), 0) END AS env_value_sd,
279+
avg(env_depth_m) AS env_depth_m,
280+
min(env_datetime) AS env_datetime_min,
281+
max(env_datetime) AS env_datetime_max,
282+
avg(dist_km) AS dist_km,
283+
avg(time_diff_hr) AS time_diff_hr
284+
FROM ranked
285+
WHERE time_diff_hr = mn_time_diff_hr
286+
GROUP BY ALL
287+
ORDER BY bio_id
288+
```
289+
290+
The next page, [Matching Helpers](helpers.qmd), shows this same query as a
291+
one-liner.

0 commit comments

Comments
 (0)