SeaweedFS - fix volume slot exhaustion and stop it recurring

[i5okie]

Overview

SeaweedFS in ca7f8f-tools had run out of volume slots, leaving dts-traces-dev and
dts-traces-prod pinned at a single writable volume each. This was never a disk-space
problem — the volume servers were sitting at 46–65% utilisation throughout. SeaweedFS
divides a disk into fixed slots, and the cluster was over-subscribed on slots while having
plenty of free bytes.

The root cause was the upstream default volume growth strategy. copy_1 = 7 / copy_2 = 6
means a single grow event allocates 7 volumes on 000 replication and 12 on 001 —
defaults written for multi-terabyte deployments, applied to a ~250GB one. This also
explained the poor packing: with a dozen volumes open at once, writes spread thin and none
of them ever filled (ha-volume-0 averaged 318MB against a 1000MB cap).

Quota was raised 270Gi → 360Gi to fund the remediation.

Completed

• Expand and equalise the HA volume servers — 50/100/50 → 100/100/100Gi
• Expand dev volume server 15 → 32Gi
• Reclaim the three HA master PVCs 5Gi → 1Gi (frees 12Gi; masters hold only raft
  log/snapshots, well under 1Gi)
• volumeGrowthCount 7/6 → 2 — the actual fix; a grow event now costs 4 volumes on
  HA and 2 on dev
• Raise minFreeSpacePercent 1 → 5 so vacuum always has room to rewrite a volume
• Purge the filer metadata log and reclaim ~12 GiB via an explicit 0.10 vacuum pass
• Weekly CronJob to purge the filer metadata log (no upstream retention setting exists)
• Mark the 7 stale filer-log volumes read-only so they drain instead of being merged
• Reconcile the values files with reality — they declared 45Gi/6Gi while live was
  50/100/50 and 15Gi

Result: HA free slots 17 → 68, dev 1 → 8. No data moved, no restarts beyond planned
rolling updates, ingestion uninterrupted.

Remaining

• 2026-08-30: manually run volume.deleteEmpty — now automatic, see comment below
• Add volume.deleteEmpty to the purge CronJob so the cycle is self-maintaining
  (also fixed: the job purged only one of the two filers)
• Re-assess whether volume.balance is still needed after the drain (it self-corrects
  much of the 78/92/78 skew)
• Production posture review — see Issues below

Issues

Gaps found during this work that are not fixed and need their own decisions:

• No shared filer store. Both HA filers run independent embedded LevelDB2 databases with
  no backup, kept consistent only by asynchronously replaying each other's metadata change
  log. Verified converging today, but the metadata is the name→needle mapping — lose it and
  the bytes are unaddressable. Options are a tiny single-pod Postgres with stateless filers,
  or dropping to one filer plus weed filer.meta.backup.
• Automatic vacuum is effectively dead on HA. Zero of 236 volumes exceeded the 0.3
  default garbage threshold, so the built-in loop walks the whole list and compacts nothing.
  It took a manual 0.10 pass to recover ~12 GiB.
• dts-logs-test-chunks holds 120 volumes for 28.5 GiB (466 MB/vol) after a past log
  flood. Retention will not reclaim these — the volumes stay writable so none ever empties.
• dev has no redundancy — single master, volume server and filer, replication: 000.
• SeaweedFSDiskUtilizationHigh thresholds never lead — set high enough (160 GiB HA /
  12 GiB dev) that the slot alert always trips first.
• Admin UI is on a public ingress (dts-seaweedfs-ha.apps.silver.devops.gov.bc.ca).

[i5okie]

Purge CronJob now runs the full reclaim cycle

ministry-gitops-ditp@00addee and @b33141a.

The job previously only purged the filer metadata log. Purging frees bytes inside volumes
but frees no slots — a volume is only reclaimed once completely empty. It now runs
purge → vacuum → deleteEmpty so reclamation is automatic, and the manual
volume.deleteEmpty step planned for ~2026-08-30 is no longer needed.

Verified by a manual run, not just deployed:

• Free slots 68 → 78
• dts-logs-test-chunks 120 → 104 volumes — 16 reclaimed. Notable because this is the
  collection left over from the past log flood, which was expected to need fs.mergeVolumes.
  A vacuum at a sane threshold emptied them and deleteEmpty collected them.
• All 7 collections intact; filers byte-identical (141395 blocks); the read-only filer-log
  volumes correctly left alone since they still hold data.

Bug found and fixed: the job purged only one filer

The job pointed at the filer service, which resolves to one filer arbitrarily. Deletions
inside /topics/.system/log do not replicate between filers — logging a log-deletion
would be circular — so each filer must purge its own view.

Purging only one drifted them apart and stranded the other's references to chunks the first
had already deleted. Observed directly: after a manual purge against filer-0 and a
scheduled run that happened to hit filer-1, the two disagreed on which date directories
existed (filer-0 had 08-12, filer-1 did not). Bucket metadata stayed byte-identical
throughout, so no tenant data was involved, and the two are back in agreement now.

This is a concrete instance of the filer-divergence risk described in the posture review as
theoretical. It is not theoretical; it just needed a trigger.

The job now iterates an explicit list of filer pod addresses. vacuum and deleteEmpty
remain single invocations — they are master-level, not per-filer.

Command traps worth not relearning

command	trap
volume.vacuum -garbageThreshold=0.10	= is silently ignored, falls back to the 0.3 default. Use a space.
volume.deleteEmpty	needs -apply, otherwise a dry run that reports success
volume.mark / any batch	multiple commands in one weed shell session silently no-op after the first
volume.fix.replication -n	-n is not a flag