[Epic] Make MarkAccent a standalone pitch-accent service (drop OJAD dependency)

[torrid-fish]

Epic / tracking issue. Long-running theme, not a single PR. Individual work items get their own issues/PRs and link back here.

Vision

API-tools is converging on pitch-accent marking (/api/MarkAccent/) as its primary purpose; the other endpoints (DictQuery, SentenceQuery, UsageQuery) see little use. The goal of this epic is to make the accent service standalone, with no runtime calls to external services, so it is robust, reproducible, low-latency, and deployable offline.

Why

• Robustness: OJAD was an HTML scrape against a university server, unreachable from some networks, with no SLA and a ToS-grey status. Same fragility class that produced the scrape-drift bugs on the other endpoints (DictQuery returns 404 for valid words (先生): EDRDG/JMdictDB markup changed name="e" → name="g" #54, Invalid symbols block the whole result parsing #57, Furigana parsing for date with number tend to drift #58).
• Latency / scale: in-process inference vs a per-chunk HTTP round-trip plus a semaphore to avoid hammering OJAD.
• Reproducibility & offline deploy: pinned local model and dictionary, no network at request time.

The technical gap

OJAD did connected-speech contour: accent-phrase (アクセント句) segmentation, compound accent (複合語アクセント), particle attachment, and downstep. UniDic's aType is only lexical / dictionary-form pitch and cannot express phrasing, so the standalone replacement is a local prosody engine, not just aType.

Workstreams

• Tokeniser standalone: local fugashi + NINJAL UniDic CWJ (Evaluate local UniDic (fugashi/Sudachi) for in-process accent + lower latency #50)
• Local prosody engine: in-process OpenJTalk full-context accent (Evaluate local UniDic (fugashi/Sudachi) for in-process accent + lower latency #50, spike results in comments)
• Mapping layer: OpenJTalk full-context labels to per-mora accent_marking_type (0 / 1 / 2)
• Lexical fallback floor from lexical_kernel (Fallback to UniDic aType-derived lexical pitch when OJAD is unavailable #61)
• Remove the OJAD scrape

Still open

• Repo split: spin the accent service into its own repo, leaving Dict/Sentence/Usage behind? Affects packaging and CD.

Out of scope (parked)

• Standalone-ifying DictQuery / SentenceQuery / UsageQuery, and their known bugs (DictQuery returns 404 for valid words (先生): EDRDG/JMdictDB markup changed name="e" → name="g" #54, Invalid symbols block the whole result parsing #57, Furigana parsing for date with number tend to drift #58). These stay external for now; revisit only if the repo-split decision pulls them along.

References

• Tokeniser local: Evaluate local UniDic (fugashi/Sudachi) for in-process accent + lower latency #50
• Lexical fallback floor: Fallback to UniDic aType-derived lexical pitch when OJAD is unavailable #61
• Backend eval (OpenJTalk vs OJAD, MARINE): see the comments below

[torrid-fish]

Spike result: pyopenjtalk (OpenJTalk) as an offline OJAD replacement — promising

Ran OpenJTalk locally (ephemeral env, no changes to pyproject/lock) and reconstructed accent with the standard ESPnet-style full-context prosody parser.

Key evidence — the 端 / 箸 / 橋 minimal triple, disambiguated by context, fully offline

端を:  ^ h a [ sh i o #        → はし rise, no fall  = heiban    (端)  ✓
箸で:  # h a ] sh i d e #      → は]し fall@mora1    = atamadaka (箸)  ✓
橋を:  # h a [ sh i ] o #      → はし]を fall@mora2   = odaka     (橋)  ✓

Other hand-checked sentences also correct: 今日=きょ]う (頭高), 日本語=にほんご (平板), 勉強=べんきょう (平板), 人工知能=じんこうち]のう.

So OpenJTalk gives connected-speech accent (accent-phrase segmentation # + nucleus ]), not just lexical pitch — i.e. it genuinely covers what we use OJAD for, in-process.

Footprint / availability

• open_jtalk_dic ≈ 23 MB (vs UniDic 1.3 GB) — negligible image-size delta.
• Pure CPU, no network at request time.
• MARINE (neural accent refinement, run_marine=True): NOT available out of the box — needs marine + torch (heavy). Defer unless eval shows OpenJTalk-alone accuracy is insufficient.

Mapping to our schema

OpenJTalk per-accent-phrase nucleus → our per-mora accent_marking_type (0=LOW/1=HIGH/2=FALL) is the same Tokyo-dialect math as #61, just driven by OpenJTalk's phrasing instead of a single word kernel. The only careful part is mora grouping (small kana ゃゅょ, っ, ー) — already called out as the "mapping layer" workstream above.

What this spike did NOT establish

• Quantitative agreement vs OJAD: OJAD is unreachable from the dev/CI network (TCP connect timeout), so no gold baseline could be captured here. Need to run the OJAD half on an OJAD-reachable network and score nucleus-position agreement. The harness is ready for that (OpenJTalk side + an OJAD <TODO> slot).
• MARINE quality (not installed).

Recommendation

OpenJTalk is a credible drop-in for the get_ojad_result seam. Proposed next steps:

1. Capture an OJAD gold set (a few hundred sentences) on an OJAD-reachable network.
2. Score OpenJTalk (and OpenJTalk+MARINE) nucleus agreement vs OJAD, by word class.
3. Decide MARINE (accuracy vs +torch image cost) and whether OJAD stays as an optional high-accuracy backend.

[torrid-fish]

Update: accuracy vs human gold (not just OJAD-vs-OpenJTalk agreement)

The earlier spike measured mutual agreement (≈90% realized-pitch) between OJAD and OpenJTalk. That overstated OpenJTalk's quality. Using a free, human-annotated ground truth — the JSUT gold set (sarulab-speech/jsut-label, BASIC5000, CC-BY-SA) — we can now score accuracy.

Result (200 sentences, 5355 gold morae)

Backend	coverage	exact per-mora	realized-pitch
OJAD	71.6%	85.5%	89.9%
OpenJTalk (vanilla)	76.3%	56.1%	66.7%

• OJAD's 89.9% sits in the literature band (rule-based ≈87%, neural ≈94%) → validates the harness; OJAD is a strong reference.
• OpenJTalk-vanilla is ~23 pts behind. It's a viable offline fallback, not yet an OJAD-quality replacement.
• Coverage ~72–76%: accuracy is over kana-aligned morae; the unaligned ~25% is reading-notation divergence (は/わ, を/お, ー), reported transparently.

What's on spike/openjtalk-accent now

• scripts/download_jsut_gold.sh — fetch the gold set (labels only, no audio).
• scripts/eval_accent_backends.py --gold data/gold/jsut-label --limit N — accuracy-vs-gold scoring for both backends.
• data/corpora/handpicked_v1.txt (+README) — the probe corpus, now versioned data.
• Docs: docs/openjtalk-vs-ojad-eval.md (Part 2 = gold accuracy), docs/accent-benchmarks-survey.md, docs/accent-annotation.md (surface/furigana/accent convention).

Benchmark survey takeaways: tdmelodic would improve OpenJTalk readings but is non-commercial / eval-only; MARINE is the most promising shippable lever.

---

Next experiment — does MARINE close the gap?

Goal: measure whether OpenJTalk + MARINE (pyopenjtalk[marine], run_marine=True, DNN accent estimation) narrows the 66.7% → 89.9% gap enough to justify dropping/optionalising OJAD.

Hypothesis: MARINE lifts OpenJTalk realized-pitch accuracy from ~67% toward ~80%+ (its paper/JSUT-trained range), at the cost of a torch dependency (image-size hit).

Method (reuses the existing gold harness — minimal new code):

1. pip install pyopenjtalk[marine] (torch); add a third backend openjtalk-marine that calls extract_fullcontext(text, run_marine=True). The shared fullcontext.py parser already handles its labels unchanged.
2. Run the same gold eval over all three backends on an identical sample (start --limit 200, then scale the OpenJTalk/MARINE side — no network — to the full 5000).
3. Report realized-pitch + exact accuracy vs gold for OJAD / OpenJTalk / OpenJTalk+MARINE side by side.

Secondary (tightens the numbers): add a kana-normalisation pass (は↔わ, を↔お, へ↔え, ー) before the difflib alignment to raise gold coverage above ~72%, so accuracy isn't measured on a biased subset.

Decision criteria:

• MARINE within a few points of OJAD (say ≥85% realized-pitch) → pursue local-only with OJAD as optional high-accuracy backend.
• Still well below OJAD → keep OJAD primary, OpenJTalk(±MARINE) as the offline-degradation fallback (slots into the fix(accent): degrade gracefully when OJAD is unavailable #60 graceful-degradation path), and revisit readings (tdmelodic eval-only) / a learner-targeted accuracy bar.

Cost note: torch materially increases image size (already ~1.3 GB from UniDic) — factor into the keep-OJAD-or-not call.

[torrid-fish]

MARINE experiment result — does the DNN estimator close OpenJTalk's gap to OJAD?

Ran OpenJTalk + MARINE (pyopenjtalk[marine], run_marine=True) as a third backend on the same JSUT gold harness, in an isolated venv (torch; numpy pinned <2 — MARINE relies on pre-2.0 uint8 wrapping — so the main env was untouched).

Accuracy vs gold (realized-pitch)

Backend	@200 (3-way)	@1000 (offline)	raw, no pipeline (~94% cov)
OJAD	89.8%	—	—
OpenJTalk	66.6%	66.6%	67.5%
OpenJTalk + MARINE	69.9%	70.3%	72.1%

MARINE adds only ~+4 pts — it does not close the ~20-pt gap to OJAD.

A raw engine-vs-gold check (bypassing our align/override/postprocess, so morae line up ~1:1 with gold → 94% coverage) gives the same ~+4–5 pt gain, so our OJAD-tuned pipeline is not blunting MARINE — the gain is genuinely small. (The metric caps ~90% even for OJAD due to convention/alignment noise, so read these as relative: OJAD ≫ MARINE > vanilla OpenJTalk.)

Costs / problems

• torch ≈ 1.1 GB (default wheel pulled the full CUDA stack → 4.8 GB venv; CPU-only torch ≈200 MB, still ≫ the 23 MB OpenJTalk dict).
• Latency ~73 ms/sentence (vs ~few ms vanilla, ~1.8 s OJAD).
• Robustness bug: MARINE AssertionErrors (Invalid sequence sizes in njd_results, marine_results) on inputs with ・, full-width digits, or long katakana-name sentences — multiple BASIC5000 sentences failed outright.

Recommendation

Keep OJAD primary; vanilla OpenJTalk as the offline-degradation fallback (the #60 path). MARINE's ~+4 pts don't justify +1.1 GB, 7× latency, and an input-robustness regression.

If higher offline accuracy is ever needed, the remaining levers are: reconcile the systematic verb-ending / 尾高·平板 marking differences, and/or tdmelodic for readings (eval-only — non-commercial license).

On the branch (spike/openjtalk-accent, pushed)

• Gated ACCENT_BACKEND=openjtalk-marine (default behaviour unchanged) + --backends/MARINE-guard/kana-norm in scripts/eval_accent_backends.py. torch is NOT added to runtime deps — MARINE is an isolated eval-only install.
• Full write-up (3-way + offline + raw diagnostic) in docs/openjtalk-vs-ojad-eval.md Part 2.

[torrid-fish]

Status: the standalone goal is now realized by PR #63, which makes tokenisation (local fugashi + UniDic) and accent (in-process OpenJTalk) fully offline and removes the OJAD scrape. The backend decision is settled in the spike comments above: OpenJTalk is the sole backend; MARINE was rejected (about +4 pts of accuracy for +1.1 GB of torch, 7x latency, and an input-robustness regression); OJAD is not shippable (licence, latency). Keeping this epic open to track the remaining repo-split decision.