Merge pull request #324 from ropensci/data-release-2026

Ship 2026 data release (fb v26.07, slb v26.04) and script the import workflow

Author: cboettig

.Rbuildignore

@@ -29,3 +29,4 @@ circle.yml
 ^CRAN-SUBMISSION$
 ^\.claude$
 ^issues-triage\.md$
+^imports/$

.gitignore

@@ -11,3 +11,4 @@ parquet
 fbapp
 slbapp
 CRAN-SUBMISSION
+__pycache__

DESCRIPTION

@@ -6,7 +6,7 @@ Description: A programmatic interface to 'FishBase', re-written
     supports experimental access to 'SeaLifeBase' data, which contains
     nearly 200,000 species records for all types of aquatic life not covered by
     'FishBase.'
-Version: 5.0.3
+Version: 5.0.4
 Encoding: UTF-8
 License: CC0
 Authors@R: c(person("Carl", "Boettiger",

NEWS.md

@@ -23,6 +23,18 @@ And constructed with the following guidelines:
 
 For more information on SemVer, please visit http://semver.org/.
 
+v 5.0.4
+-------
+
+Data updated to the 2026 releases: FishBase `v26.07` and SeaLifeBase `v26.04`.
+These are now what `version = "latest"` resolves to. Earlier releases remain
+available via `available_releases()`. The new snapshots add 6 tables to
+FishBase (222 total) and 3 to SeaLifeBase (202 total); no previously published
+table was dropped.
+
+The workflow for importing a new database dump and publishing it to Source
+Cooperative is now scripted and documented in `data-raw/README.md`.
+
 v 5.0.3
 -------
 

data-raw/README.md

@@ -0,0 +1,56 @@
+# Preparing a new data release
+
+FishBase and SeaLifeBase send `fbapp.7z` / `slbapp.7z` (7z-compressed
+`mysqldump` output) roughly annually.  Publishing them is two commands:
+
+```bash
+# 1. drop the dumps in imports/ (git-ignored, excluded from the build), then
+bash data-raw/import_dumps.sh /tmp/fishbase-import
+# 2. read the verification summary, then publish (args are fb and slb versions)
+bash data-raw/upload.sh /tmp/fishbase-import 26.07 26.04
+```
+
+`import_dumps.sh` unpacks the dumps into a throwaway local MariaDB instance and
+calls `export_parquet.py`, which writes one zstd-compressed parquet file per
+table via DuckDB's `mysql` extension, then `verify_export.py`, which checks the
+result and exits non-zero if anything looks wrong.  `upload.sh` publishes to
+Source Cooperative under `cboettig/fishbase/{fb,slb}/v<version>/parquet/`,
+which is what `fb_urls()` lists.
+
+Nothing in the R package needs to change to ship a release:
+`available_releases()` reads the bucket and `version = "latest"` picks the
+highest version present.
+
+Version tags encode the *snapshot date* of the dump (`v26.07` = July 2026).
+The two servers can carry different tags when their dumps arrive at different
+times, since releases are listed per server.
+
+Requires `p7zip-full`, `mariadb-server` (10.11+), `python3` with `duckdb`, and
+an `rclone` remote named `source` holding Source Cooperative credentials.
+Budget roughly 10 GB of scratch disk and an hour of wall clock.
+
+## What the verification covers
+
+`verify_export.py` compares every table's parquet row count against the source
+database, diffs the table list against the most recent release already on
+Source Cooperative, and checks that the two servers did not get crossed.  Run
+it *before* uploading: it compares against whatever release is currently
+newest, so once you publish, the comparison baseline becomes the new release.
+
+## Notes and gotchas
+
+- The two dumps come from different MySQL versions (5.6 for `fbapp`, 8.0 for
+  `slbapp`).  MariaDB rejects MySQL 8's `utf8mb4_0900_ai_ci` collation, so the
+  import rewrites it; only the data matters here, not collation semantics.
+- DuckDB's `mysql` ATTACH exposes *every* schema on the server, not just the
+  one named in the connection string.  `export_parquet.py` therefore filters on
+  `schema_name` and fully qualifies each table — an unqualified `src."species"`
+  silently resolves across databases and will mix FishBase and SeaLifeBase.
+  `verify_export.py` guards against a regression here.
+- The publishing credentials can write the prefix but are not granted
+  `s3:CreateBucket`, which `rclone` probes for unless given
+  `--s3-no-check-bucket`.
+- Recent dumps have included working tables from the FishBase team (`*_copy`,
+  `mp_withphotos`, `country.orig`).  They are published as-is, since these are
+  raw snapshots of the backend, but they show up in the verification diff as
+  new tables — worth a glance before publishing.

data-raw/export_parquet.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+"""Export one MySQL schema of a fishbase snapshot to per-table parquet files.
+
+Usage: export_parquet.py <mysql_schema> <outdir>
+"""
+import json
+import sys
+
+import duckdb
+
+db, outdir = sys.argv[1], sys.argv[2]
+
+con = duckdb.connect()
+con.execute("INSTALL mysql; LOAD mysql;")
+con.execute(
+    f"ATTACH 'host=127.0.0.1 port=3307 user=root database={db}' AS src (TYPE mysql, READ_ONLY)"
+)
+
+# NB: the mysql attach exposes every schema on the server, so scope explicitly
+tables = [
+    r[0]
+    for r in con.execute(
+        "SELECT table_name FROM duckdb_tables() "
+        "WHERE database_name='src' AND schema_name=? ORDER BY table_name",
+        [db],
+    ).fetchall()
+]
+print(f"{db}: {len(tables)} tables", flush=True)
+
+ok, failed = [], {}
+for i, tbl in enumerate(tables, 1):
+    dest = f"{outdir}/{tbl}.parquet"
+    try:
+        con.execute(
+            f'COPY (SELECT * FROM src."{db}"."{tbl}") TO \'{dest}\' '
+            "(FORMAT parquet, COMPRESSION zstd)"
+        )
+        n = con.execute(f"SELECT count(*) FROM read_parquet('{dest}')").fetchone()[0]
+        ok.append((tbl, n))
+        print(f"[{i}/{len(tables)}] {tbl}: {n}", flush=True)
+    except Exception as e:
+        failed[tbl] = str(e).split("\n")[0]
+        print(f"[{i}/{len(tables)}] {tbl}: FAILED {failed[tbl]}", flush=True)
+
+with open(f"{outdir}/../{db}_export_report.json", "w") as f:
+    json.dump({"ok": ok, "failed": failed}, f, indent=2)
+
+print(f"done: {len(ok)} ok, {len(failed)} failed", flush=True)

data-raw/huggingface.R

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# Import FishBase / SeaLifeBase MySQL dumps and export them as parquet.
+#
+# The FishBase team sends `fbapp.7z` / `slbapp.7z` (7z-compressed mysqldump)
+# roughly annually.  Drop them in `imports/` (git- and build-ignored), run this
+# script, sanity-check the result, then publish with `upload.sh`.
+#
+# Requires: p7zip-full, mariadb-server (10.11+), python3 with duckdb.
+# Budget ~10 GB of scratch disk and about an hour of wall clock.
+#
+# Usage:  bash data-raw/import_dumps.sh [workdir]
+set -euo pipefail
+
+W="${1:-/tmp/fishbase-import}"
+REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+PORT=3307
+
+for f in fbapp slbapp; do
+  [ -f "$REPO/imports/$f.7z" ] || { echo "missing $REPO/imports/$f.7z" >&2; exit 1; }
+done
+
+mkdir -p "$W"/{sql,mysql-data,tmp,out/fb/parquet,out/slb/parquet}
+
+7z x -o"$W/sql" -y "$REPO/imports/fbapp.7z"
+7z x -o"$W/sql" -y "$REPO/imports/slbapp.7z"
+
+mariadb-install-db --datadir="$W/mysql-data" --user="$(whoami)" \
+  --auth-root-authentication-method=normal --skip-test-db
+
+cat > "$W/my.cnf" <<CNF
+[mysqld]
+user                = $(whoami)
+bind-address        = 127.0.0.1
+port                = $PORT
+socket              = $W/mysqld.sock
+pid-file            = $W/mysqld.pid
+datadir             = $W/mysql-data
+tmpdir              = $W/tmp
+key_buffer_size     = 8G
+myisam_sort_buffer_size = 4G
+innodb_buffer_pool_size = 16G
+innodb_flush_log_at_trx_commit = 0
+innodb_doublewrite  = 0
+max_allowed_packet  = 1G
+skip-log-bin
+sql_mode            = NO_ENGINE_SUBSTITUTION
+CNF
+
+mariadbd --defaults-file="$W/my.cnf" > "$W/mysqld.log" 2>&1 &
+for _ in $(seq 1 60); do
+  mariadb-admin --socket="$W/mysqld.sock" -u root ping >/dev/null 2>&1 && break
+  sleep 2
+done
+mariadb-admin --socket="$W/mysqld.sock" -u root ping >/dev/null \
+  || { echo "server failed to start, see $W/mysqld.log" >&2; exit 1; }
+
+# The fbapp dump comes from MySQL 5.6, slbapp from MySQL 8.0.  MariaDB does not
+# know MySQL 8's default collation, so rewrite it on the way in; only the data
+# matters here, not the collation semantics.
+mariadb --socket="$W/mysqld.sock" -u root < "$W/sql/fbapp.sql"
+sed 's/utf8mb4_0900_ai_ci/utf8mb4_general_ci/g' "$W/sql/slbapp.sql" \
+  | mariadb --socket="$W/mysqld.sock" -u root
+
+python3 "$REPO/data-raw/export_parquet.py" fbapp  "$W/out/fb/parquet"
+python3 "$REPO/data-raw/export_parquet.py" slbapp "$W/out/slb/parquet"
+
+python3 "$REPO/data-raw/verify_export.py" "$W" "$PORT"
+
+mariadb-admin --socket="$W/mysqld.sock" -u root shutdown
+echo "Export complete: $W/out.  Publish with data-raw/upload.sh"