learn-errors: cross-sample diversity in nbases accumulation (sample-level, --randomize doesn't spread reads)

[cjfields]

Summary

learn-errors (and R DADA2's learnErrors) accumulate the --nbases learning
budget at sample granularity: input files are taken whole — in supplied
order, or shuffled with --randomize — and each contributes all of its bases
until the running total reaches --nbases (default 100M). Reads are never
subsampled within a file.

The concern

With modern deep sequencing a single sample can supply the entire budget:

• 400k reads × 250 bp = 100M bases = the default --nbases on its own
• Even ~100k-read samples exhaust the budget in 3–4 files

So the unsupervised error model may be learned from the diversity of just one
or a few samples rather than the run as a whole. Crucially, --randomize
does not fix this — it only shuffles which sample dominates, not whether
one dominates, because accumulation is still whole-sample.

This is inherited from R DADA2 and is an under-appreciated footgun.

Open empirical question

Does it actually matter? Before changing behavior we want to measure how skewed
real runs are and whether spreading the budget across samples meaningfully
changes the learned error matrices / final ASVs. Any change to sample selection
alters the error matrices → ASVs and would break the R-concordance guardrail, so
it must be opt-in with default behavior unchanged.

Candidate directions (not yet decided)

1. Diagnostic first — report each sample's contribution (bases / % of
   budget, and which samples went unused) via --diag-dir, to quantify skew on
   real data before committing to any algorithm change.
2. Per-sample base cap — a --max-bases-per-sample flag so the budget must
   pull from more samples (need to decide what subset of a capped deep sample to
   keep: top-abundance vs multiplicity-weighted random).
3. Balanced round-robin draw — allocate ~nbases / n_samples to each sample
   so every sample is represented (strongest guarantee, largest departure from R).

Done for now

• Documented the caveat in the learn-errors command help and on the --nbases
  / --randomize flags.
• learn-errors now also accepts pre-dereplicated derep/sample JSON input,
  which makes per-sample pre-sample-ing a viable manual workaround.