Skip to content

Commit 3fa7d59

Browse files
CrispStrobeclaude
andcommitted
docs(learnings): a phoneme dialect mismatch is invisible to every guard built to catch a bad G2P
The Russian G2P passed the inventory check (zero dropped codepoints, with a control that fires on that exact path) and still lost the roundtrip to espeak, because the two spell the same sounds differently and BOTH spellings are inside zonos's inventory. The guard built to catch a bad G2P cannot see the one thing that was actually wrong. Generalisation: when a model was conditioned on some tool's output, correctness of your transcription cannot answer whether it is the same spelling. Also records the two instrument flaws the run exposed — a control anchored to a baseline that itself collapsed to 0.000, and an espeak baseline below the pre-registered floor making the roundtrip inconclusive rather than a pass — and the disjoint-files discovery in the upstream data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent c001ee2 commit 3fa7d59

2 files changed

Lines changed: 80 additions & 4 deletions

File tree

LEARNINGS.md

Lines changed: 76 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ If a lesson is still "live" (affects current work), it's linked from
1111
---
1212

1313
<!-- BEGIN INDEX POINTER — generated by tools/gen-learnings-index.py -->
14-
**Do not read this file top to bottom** — it is ~1 MB across 301 sections.
14+
**Do not read this file top to bottom** — it is ~1 MB across 302 sections.
1515
The generated topic + per-model index is [`docs/LEARNINGS-INDEX.md`](docs/LEARNINGS-INDEX.md).
1616
Find the lesson there, then jump with `grep -n "<heading text>" LEARNINGS.md`.
1717
Regenerate both with `python tools/gen-learnings-index.py` after adding a section.
@@ -18889,3 +18889,78 @@ before submission, and export the build cache immediately after a successful
1888918889
build as well as after final validation. The corrected P100 run was
1889018890
transcript-exact and made v0.23 0.53% faster on the 134 s clip; its 7,927-file
1889118891
cache became the next account-matched dataset version.
18892+
18893+
## A phoneme dialect mismatch is invisible to every guard built to catch a bad G2P (#435 follow-on, 2026-09-15)
18894+
18895+
Russian was the last language for which zonos still needed espeak-ng (GPL-3.0).
18896+
A built-in G2P now covers it: an 812,953-entry dictionary with lexical stress
18897+
already resolved, plus letter-to-sound rules. The inventory question that #435
18898+
exists to police was answered up front and answered well — the dictionary uses
18899+
37 distinct codepoints, 36 of them already in zonos's `conditioning.py` symbol
18900+
list, and the exception was the upstream stress marker, a BACKTICK (U+0060) that
18901+
is 7.57% of all symbols in the file. Rewriting it to U+02C8 when publishing the
18902+
dictionary took the fit to 100%. Had that been missed, the built-in path would
18903+
have produced audio, returned 0, and silently lost every stress mark.
18904+
18905+
The Kaggle run confirmed it: **zero dropped codepoints on the built-in path**,
18906+
and — the part that makes that number mean something — a control that fires on
18907+
that exact path. A one-line dictionary carrying the original backtick was
18908+
injected through `CRISPASR_RU_DICT_PATH`, and the counter reported
18909+
`dropped=1 hist={U+0060: 1}`. Without that arm, "the built-in dropped nothing"
18910+
is a negative the instrument has never been shown capable of contradicting.
18911+
18912+
And the ASR roundtrip still preferred espeak, 0.585 to 0.293.
18913+
18914+
Running espeak's own `ru` voice over 2,200 dictionary words locally said why,
18915+
and it is not a defect in the G2P. The two transcribe the same sounds in
18916+
different symbols:
18917+
18918+
ours məɫɐkˈo i xlʲep lʲɪʐˈat na stɐlʲˈe v bɐlʲʂˈoj kˈomnətʲe
18919+
espeak mʌɭʌkˈo ɪ xɭʲˈep ɭʲiʒˈɑt nə stʌɭʲˈe v bʌɭʃˈoj kˈomnʌtʲi
18920+
18921+
Raw symbol agreement 57.7%; not one word of the 2,200 identical. **Every symbol
18922+
on both sides is inside zonos's inventory**, so the drop counter — the guard
18923+
built specifically to catch a G2P that hands a model something it cannot use —
18924+
is blind to this by construction. So is the inventory unit test. So is the
18925+
refusal path. A narrower, phonetically better transcription can be strictly
18926+
worse for a model trained on a broader one, and none of the checks notice.
18927+
18928+
This is #316 (Kokoro trained on misaki's spelling, 58% agreement from CMUdict)
18929+
one language further on, which is the useful generalisation: **whenever a model
18930+
was conditioned on some tool's output, the question is not "is my transcription
18931+
correct" but "is it the same SPELLING", and correctness cannot answer it.** The
18932+
fix has the same shape too — the G2P keeps emitting its own transcription and
18933+
the consumer converts, because one dictionary serves consumers with different
18934+
conventions.
18935+
18936+
The conversion was DERIVED rather than assumed: each rule is the majority
18937+
alignment over those 2,200 pairs. It takes agreement to 88.0% and exact word
18938+
matches from 0.0% to 32.6%. One rule that felt obvious was measured and
18939+
REJECTED — moving `ˈ` from before the stressed vowel to the syllable onset
18940+
drops agreement to 75.1%, because espeak's ru voice marks the vowel too.
18941+
Another, `tɕ`→`tʃʲ`, measured better (89.6%) and was deliberately held out of
18942+
the shipped function so that the code matches the arm that was measured.
18943+
18944+
Two further things the run taught about instruments rather than about Russian.
18945+
First, a control leg compared the wrong-language arm's word-F1 against the
18946+
baseline for sentence 0 — whose espeak score was itself 0.000, so the leg could
18947+
not fire whatever the control produced. Its agreement leg fired decisively
18948+
(0.063 against 0.63–0.77), which is the only reason the flaw was visible.
18949+
**A control anchored to a baseline that can collapse is not a control.**
18950+
Second, the espeak baseline for Russian was 0.585, below the pre-registered 0.60
18951+
floor, and 0.000 on one of three sentences — zonos-v0.1 does not list Russian
18952+
among its languages. So the roundtrip measures zonos's Russian more than it
18953+
measures the G2P, and the honest verdict is INCONCLUSIVE for the default flip,
18954+
not a pass and not a failure. Spanish was once passed for exactly the mirror of
18955+
that reason.
18956+
18957+
The data itself carried one more lesson, unrelated to measurement. The upstream
18958+
project shipped `heteronyms.txt` alongside the vocabulary, and the two files are
18959+
DISJOINT: 0 of the 17,359 heteronyms appear among the 812,953 entries. They are
18960+
not entries with a chosen reading — they are the words upstream removed because
18961+
it could not choose. "On a heteronym, take the dictionary's reading" describes
18962+
an option that does not exist, and the set is not exotic (`все`, `уже`, `потом`,
18963+
`чем`, `небо`, `тест`). **Check the relationship between a source's files before
18964+
designing around it**; the shape that was assumed (a dictionary with one reading
18965+
per ambiguous word) and the shape that shipped (a dictionary with those words
18966+
deleted) lead to completely different fallback designs.

docs/LEARNINGS-INDEX.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ per-backend porting detail (Per-model port notes + the family cross-reference).
1717
Reach for the topic groups when you are stuck on a *technique*, and the
1818
cross-reference when you already know which model you are touching.
1919

20-
## Index by topic (301 lessons)
20+
## Index by topic (302 lessons)
2121

2222
**Security & untrusted input** (2)
2323

@@ -357,13 +357,14 @@ cross-reference when you already know which model you are touching.
357357
- L18205 — Four bugs that together caused empty/garbage transcripts; all patched in src/vibevoice.cpp +…
358358
- L18430 — "Verified byte-identical at 225 s" verified a clip, not a length — and a decoder that drops spans needs a repair pass, not a better cap
359359

360-
**Uncategorised** (5)
360+
**Uncategorised** (6)
361361

362362
- L214 — An arm that fails under every condition cannot discriminate a fix
363363
- L235 — Hardcoded token ids need a guard that DECODES them
364364
- L286 — Model-emitted sentinels are not transcript text
365365
- L18657 — An ISA gate that "safely refuses" the only CPU module ships a process that cannot run — and every safety layer downstream assumed the layer above had fired
366366
- L18770 — An amplifying decoder needs a bound against INPUT SIZE, not duration — and the regression corpus that would have caught it had never been committed
367+
- L18893 — A phoneme dialect mismatch is invisible to every guard built to catch a bad G2P
367368

368369
## Cross-reference by model / family
369370

@@ -407,6 +408,6 @@ lookup, not a partition.
407408
- **irodori** (2) — L1941, L15202
408409
- **f5-tts** (1) — L14544
409410
- **kokoro** (4) — L906, L7782, L15839, L16512
410-
- **zonos** (1) — L12209
411+
- **zonos** (2) — L12209, L18893
411412
- **omniasr / omnivoice** (6) — L733, L1636, L5103, L14341, L14369, L14755
412413
- **glm / mimo / mega** (3) — L11557, L14252, L14309

0 commit comments

Comments
 (0)