chore: migrate L1 language validator to lingua detector#83
Merged
Conversation
…german-bleed, L2-structure, L3-state, L4-roads, L5-km-math)
…ches tests/README contract)
…to accept file args
…Eigennamen-only exception, detect German subtitles
…ANGUAGE_DETECTORS)
…ad files in validate_roads
…, update layer table
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the old regex-based German bleed detector with
lingua-language-detector, a span-based multilingual detection library.Why
The old approach used hardcoded German verb/phrase regexes - brittle, easy to evade, and unable to detect non-German violations. lingua detects language spans across the full prose, flags long non-English runs as violations, and ignores short spans (Eigennamen).
How
tests/validate_language.pyrewritten:detect_multiple_languages_ofreturns spans for both languages in mixed text. Spans ≥ 15 words in a non-allowed language = violation. Short spans = proper nouns, ignored.## Shown,## Offers,## Withheld).## Holdsskipped by design (carries km/exit lists dense with German place names).ALLOWED_LANGUAGESenv var (default:english). Override:ALLOWED_LANGUAGES=english,german python tests/validate_language.py.tests/requirements.txtadded (lingua-language-detector>=2.2.0).tests/language_exceptions.txtadded for genuine Eigenname false positives (currently onlyautobahn).pip install -r tests/requirements.txtadded to L1 job invalidate.ymlandvalidate-version-bump.yml.ARCHITECTURE.md: language policy section added (English + Eigennamen only; fix content if validator fires, do not adjust threshold).Content fixes (5 files)
Validator at threshold 15 found 5 files with real violations or architectural problems:
roads/a111/place_a111_tunnel_tegel.md-## Shownopened with tunnel identifiers and metre measurements (not driver experience). Rewritten to open with the cutting, ceiling, acoustics.roads/a20/place_a20_33_friedland.md-## Withheldhad dense Slavic/Germanic proper noun cluster triggering Polish false positive. Rewritten with better sentence structure; same history, proper nouns spread.roads/a24/place_a24_03_kreuz_hamburg_ost.md-## Showndescribed road network topology. Rewritten as driver approach.## WithheldBismarck/Sachsenwald paragraph rewritten to spread proper nouns.roads/a40/place_a40_23_essen_zentrum.md- Navigation bullet list in## Shown(architecture violation). Moved to## Holds.roads/a40/place_a40_24_essen_huttrop.md- Same navigation violation. Fixed.Validation
python tests/validate_language.py→OK: 343 file(s) passed language policy (english)