Ship 2026 data release (fb v26.07, slb v26.04) and script the import workflow

[cboettig]

Data

The 2026 dumps are imported, verified, and already published to Source Cooperative:

• cboettig/fishbase/fb/v26.07/parquet/ — 222 tables, 140 MB
• cboettig/fishbase/slb/v26.04/parquet/ — 202 tables, 108 MB

available_releases() reads the bucket directly, so version = "latest" now resolves to these with no code change. Verified end to end: fb_tbl("species") returns 36,763 rows (fb) and 102,727 (slb), matching the source database exactly, and the full test suite passes (40 passed, 0 failed, 0 skipped).

Version tags follow the convention stated in the dataset README — vYY.MM is the snapshot date of the dump. The two dumps arrived separately (fb dated 2026-07-07, slb 2026-04-22), so they carry different tags; releases are listed per server, so this works cleanly. Row modification dates corroborate the dates. Happy to republish slb under a shared v26.07 tag instead if you'd prefer to match the v24.07/v25.04 precedent.

Switched the parquet from snappy to zstd, which cut the exports from 195 MB → 140 MB (fb) and 153 MB → 108 MB (slb).

Workflow

data-raw/ only described the pins-board and HuggingFace eras, and the dataset README pointed at a creation.py that doesn't exist in any repo. This adds a runnable pipeline reproducing exactly what was run:

# dumps go in imports/ (git-ignored, excluded from the build)
bash data-raw/import_dumps.sh /tmp/fishbase-import
bash data-raw/upload.sh /tmp/fishbase-import 26.07 26.04

• import_dumps.sh — unpack, load into a scratch MariaDB, export, verify
• export_parquet.py — one zstd parquet per table via DuckDB's mysql extension
• verify_export.py — row counts vs the source db, table diff vs the previous release, crossed-schema guard; exits non-zero on any problem
• upload.sh — publish to Source Cooperative and checksum-verify

Two traps worth knowing, both now documented and guarded:

• DuckDB's mysql ATTACH exposes every schema on the server, not just the one in the connection string. An unqualified src."species" resolves across databases and silently mixes FishBase and SeaLifeBase rows — this bit the first export attempt. export_parquet.py filters on schema_name and fully qualifies; verify_export.py checks for a regression.
• The two dumps come from different MySQL versions (5.6 and 8.0). MariaDB rejects MySQL 8's utf8mb4_0900_ai_ci, so the import rewrites the collation.

The old R scripts are kept as historical reference rather than deleted, labeled as such in data-raw/README.md — say the word if you'd rather drop them.

Worth a look before merging

The new dumps include what appear to be FishBase working tables: morphmettlratios_copy, picturesmain_copy, mp_withphotos, mp_withoutphotos, mp_bothphotos, picturesmain_withoutphotos (fb) and country.orig, species_famcode, species_hierarchy (slb). They're published as-is, consistent with these being raw backend snapshots, but the _copy/.orig ones may be accidental on the FishBase side.

Also updated the source.coop dataset README to drop the dead creation.py reference and point at data-raw/.