Plural/connector/long-scale support for scaled cardinals - #8
Merged
Conversation
Follow-up to the multi-scale composition. A round-trip sweep (num2words → words2num over values above the reverse table) surfaced three more bug classes in the reverse-table locales: - plural scale words: "deux millions" left "million" unmatched (only the singular was derived); - connectors inside a run: "sesenta y nueve mil ocho", "... mil e oito", "... de mii" broke at the joining word; - long-scale billions: es/pt "mil millones"/"mil milhões" (10^9) mis-mapped its last token to 10^9. Fixes: - scale_words now samples n/2n/5n of 10^3/10^6/10^9 to capture singular, plural and the Slavic genitive plural (tysięcy/тысяч), smallest-magnitude-first so the collision guard pins each token to its true scale; - includable connector words per locale let the sentence walker grow a run across y/e/et/i/und/en/și/de; - fragment edges are connector-trimmed before lookup. Round-trip now clean for es/pt/ca/pl/ru/uk/sr (were broken); fr/en unchanged except where blocked by orthography (fr cent-plural, en comma rendering). Agglutinative locales (de/nl/sv/…) still need sub-word segmentation — untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #7. A round-trip sweep (num2words → words2num over values above the reverse table) across ~30 locales surfaced three more bug classes:
deux millions→2 millions(only singular derived)sesenta **y** nueve mil ocho→60 y 9008dos millones→2000000000(mapped 10⁶→10⁹)Fixes
scale_wordssamples n/2n/5n of 10³/10⁶/10⁹ → singular + plural + Slavic genitive plural (tysięcy/тысяч), smallest-magnitude-first so the collision guard pins each token to its true scale (mil millones=10⁹ composes from parts).y e et i und en și de) let the walker grow a run across joining words.Result
Round-trip now clean for es/pt/ca/pl/ru/uk/sr (all were broken).
engrammar untouched (its remaining fails are comma-rendered output, pre-existing). Agglutinative locales (de/nl/sv/fi/hu/tr/ja/zh/…) still need sub-word segmentation — left untouched, no regression. 26/26 tests, new multi-lang assertions.🤖 Generated with Claude Code