Merge pull request #831 from aallan/feat/645-explicit-utf8-encoding

feat(tooling): explicit encoding="utf-8" at every text-mode file call + gate (#645)

Author: aallan

.github/workflows/ci.yml

@@ -24,19 +24,12 @@ jobs:
 
     name: test (${{ matrix.os }}, ${{ matrix.python-version }})${{ matrix.python-version == '3.12' && matrix.os == 'ubuntu-latest' && ' + coverage' || '' }}
     runs-on: ${{ matrix.os }}
-    env:
-      # #641: force Python UTF-8 mode (PEP 540) so file I/O without an
-      # explicit `encoding=` kwarg defaults to UTF-8 on every platform
-      # rather than the locale's preferred encoding (cp1252 on Windows
-      # in en-US locales).  Without this, tests that read/write source
-      # files containing `→` (right arrow) or `—` (em-dash) tripped
-      # `'charmap' codec can't encode` errors on Windows.  Locally,
-      # users without `PYTHONUTF8=1` set in their shell environment
-      # will still hit the bug on the affected files — the durable
-      # fix is to audit `open()` / `read_text()` / `write_text()`
-      # call sites for explicit `encoding='utf-8'`.  Tracked as a
-      # follow-up.
-      PYTHONUTF8: 1
+    # No PYTHONUTF8 env: #645 made text I/O UTF-8 without the PEP-540 backstop
+    # (#641).  Every text-mode open()/read_text()/write_text() and every
+    # subprocess text=True capture passes an explicit encoding="utf-8" (enforced
+    # by scripts/check_explicit_encoding.py), and the `vera` CLI reconfigures its
+    # stdout/stderr to UTF-8 at startup — so a Vera program printing `→` / `—` is
+    # UTF-8 on every runner regardless of locale, cp1252 Windows included.
 
     steps:
       - uses: actions/checkout@v7
@@ -158,6 +151,9 @@ jobs:
       - name: Check every diagnostic carries rationale/fix/spec_ref (#682)
         run: python scripts/check_diagnostic_fields.py
 
+      - name: Check text-mode file I/O passes explicit encoding (#645)
+        run: python scripts/check_explicit_encoding.py
+
       - name: Check site assets are up-to-date
         run: python scripts/check_site_assets.py
 

.pre-commit-config.yaml

@@ -165,6 +165,16 @@ repos:
         # citation — include spec markdown so that edit re-runs the check too.
         files: '(vera/.*\.py$|spec/.*\.md$|scripts/check_diagnostic_fields\.py$)'
 
+      - id: explicit-encoding
+        name: explicit encoding
+        entry: .venv/bin/python scripts/check_explicit_encoding.py
+        language: system
+        pass_filenames: false
+        # Auto-discovers every text-mode open()/read_text()/write_text() under
+        # vera/, scripts/, tests/ (#645), so the trigger must be at least that
+        # broad (same reasoning as diagnostic-fields / walker-coverage).
+        files: '((vera|scripts|tests)/.*\.py$)'
+
       - id: limitations-sync
         name: limitations sync
         entry: .venv/bin/python scripts/check_limitations_sync.py

CHANGELOG.md

@@ -6,6 +6,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.0.190] - 2026-07-01
+
+### Changed
+
+- **Text I/O is UTF-8 regardless of the host locale, enforced by a gate** ([#645](https://github.com/aallan/vera/issues/645)).  Python's text-mode `open()` / `Path.read_text()` / `Path.write_text()` — and `subprocess.run/Popen/check_output(..., text=True)` captures — fall back to `locale.getpreferredencoding()` (cp1252 on en-US Windows) when no `encoding=` is given, so a Vera source / doc / fixture / program output containing `→`, `—`, or any non-ASCII byte failed on a locale-default Windows shell.  [#641](https://github.com/aallan/vera/issues/641) papered over CI with a `PYTHONUTF8=1` backstop; this is the durable fix.  A new `scripts/check_explicit_encoding.py` AST-audits every text-mode `open()` / `read_text()` / `write_text()` **and** every `subprocess(..., text=True)` capture under `vera/`, `scripts/`, `tests/`, requiring an explicit `encoding="utf-8"` literal (binary / bytes mode skipped; a deliberate exception opts out with `# encoding-exempt: <reason>`), wired into pre-commit and the CI `lint` job.  It found and fixed **160 bare file-I/O sites across 16 files** plus **97 subprocess text captures across 18 files** (far more than the ~30 the issue estimated — the suite has grown ~5× since #641).  The audit's scope also covers `tempfile.NamedTemporaryFile` / `TemporaryFile` / `SpooledTemporaryFile` opened in *text* mode (they default to binary, but a text one is a locale-encoded write just like `open(..., "w")` — the idiom test helpers use to stage `.vera` source containing `→` / `—`; **29 such sites across 9 files** fixed).  The `vera` CLI reconfigures **stdin** as well as stdout/stderr to UTF-8 at startup, so a Vera program's non-ASCII output *and* input (e.g. `IO.read_char` on piped UTF-8) round-trip on any locale.  Together these made text I/O locale-independent and the `PYTHONUTF8=1` CI backstop (#641) was **removed** — verified by a clean Windows matrix with the backstop gone (the tempfile and stdin gaps were surfaced by that matrix, not the audit, which is why removing the backstop is its own acceptance test).  Pinned by `tests/test_check_explicit_encoding.py` (checker logic, scope-discovery coverage, and a repo-clean assertion).  (Folds in the stdio / subprocess work that had briefly been split to [#832](https://github.com/aallan/vera/issues/832).)
+
 ## [0.0.189] - 2026-07-01
 
 ### Changed
@@ -2695,7 +2701,8 @@ Small docs sweep — closes six aging documentation issues in one PR.  No code c
 - Grammar: handler body simplified to avoid LALR reduce/reduce conflict
 - `pyproject.toml`: corrected build backend, package discovery, PEP 639 compliance
 
-[Unreleased]: https://github.com/aallan/vera/compare/v0.0.189...HEAD
+[Unreleased]: https://github.com/aallan/vera/compare/v0.0.190...HEAD
+[0.0.190]: https://github.com/aallan/vera/compare/v0.0.189...v0.0.190
 [0.0.189]: https://github.com/aallan/vera/compare/v0.0.188...v0.0.189
 [0.0.188]: https://github.com/aallan/vera/compare/v0.0.187...v0.0.188
 [0.0.187]: https://github.com/aallan/vera/compare/v0.0.186...v0.0.187

CLAUDE.md

@@ -64,6 +64,7 @@ python scripts/check_faq_examples.py  # Verify FAQ code blocks parse
 python scripts/check_html_examples.py # Verify HTML code blocks parse + check + verify
 python scripts/check_doc_builtin_shadowing.py # Verify no doc example redefines a built-in (E151; #819)
 python scripts/check_diagnostic_fields.py # Verify every diagnostic carries rationale + spec_ref (+ fix for errors; warnings exempt) or a # diag-fields-exempt reason (#682)
+python scripts/check_explicit_encoding.py # Verify every text-mode open()/read_text()/write_text() passes explicit encoding='utf-8' (#645)
 python scripts/build_site.py          # Regenerate AI-readable site assets (llms.txt, etc.)
 python scripts/check_site_assets.py   # Verify site assets are up-to-date
 python scripts/check_version_sync.py  # Verify version consistency
@@ -207,4 +208,4 @@ The CI matrix tests on `{ubuntu-latest, macos-15, macos-26, windows-latest} × {
 
 - `tempfile.NamedTemporaryFile` handed off to a subprocess MUST use `delete=False` + manual `Path.unlink()` (Windows can't reopen a held file).
 - Paths embedded into Vera string literals MUST be POSIX-form (`Path(tmp_path).as_posix()`); Windows backslashes trip Vera's `\U` escape grammar.
-- `open()` / `read_text()` / `write_text()` should pass `encoding="utf-8"` explicitly; CI sets `PYTHONUTF8=1` as a backstop, but the explicit form is portable to local Windows shells without that variable.
+- Text I/O MUST pass `encoding="utf-8"` explicitly, enforced by `scripts/check_explicit_encoding.py` (pre-commit + CI lint, #645): every text-mode `open()` / `read_text()` / `write_text()` **and** every `subprocess.run/Popen/check_output(..., text=True)` capture. A deliberate non-UTF-8 site opts out with `# encoding-exempt: <reason>`. The `vera` CLI also reconfigures its stdout/stderr to UTF-8 at startup, so a Vera program printing `→` / `—` is UTF-8 on any locale. Together these replaced the `PYTHONUTF8=1` CI backstop (#641), which has been removed — no reliance on the runner's or a local Windows shell's locale.

CONTRIBUTING.md

@@ -89,9 +89,9 @@ dependencies. CI enforces that `uv.lock` stays current.
 
 ### Pre-commit Hooks
 
-Every push is checked by 30 configured hooks across two stages: 28 are configured at the commit stage (run after `pre-commit install`), and 2 (`check-changelog-updated` and `uv-lock-check`, described below) are configured at the push stage (run after `pre-commit install --hook-type pre-push`). Most commit-stage hooks have per-hook `files:` / `types:` filters — the `python` type-check only runs when Python files are staged; `check_readme_examples.py` only runs when `README.md` or Vera sources change, etc. A plain-text commit touching only one markdown file triggers a small subset; a compiler-level commit triggers most of them.
+Every push is checked by 31 configured hooks across two stages: 29 are configured at the commit stage (run after `pre-commit install`), and 2 (`check-changelog-updated` and `uv-lock-check`, described below) are configured at the push stage (run after `pre-commit install --hook-type pre-push`). Most commit-stage hooks have per-hook `files:` / `types:` filters — the `python` type-check only runs when Python files are staged; `check_readme_examples.py` only runs when `README.md` or Vera sources change, etc. A plain-text commit touching only one markdown file triggers a small subset; a compiler-level commit triggers most of them.
 
-The **commit-stage** hooks (28, each gated to relevant files) include:
+The **commit-stage** hooks (29, each gated to relevant files) include:
 
 - Trailing whitespace and file endings
 - YAML/TOML validity

HISTORY.md

@@ -412,8 +412,9 @@ A sustained audit of every place where `vera verify` could prove a postcondition
 | v0.0.185 | 28 Jun | Redefining an opaque, verifier-modelled built-in is now a checker error, closing a `verify`/`run` soundness hole ([#815](https://github.com/aallan/vera/issues/815)). |
 | v0.0.186 | 29 Jun | `@Nat` → `@Int` widening is now sound: a value above i64.MAX no longer reinterprets to a negative `@Int` ([#813](https://github.com/aallan/vera/issues/813)). |
 | v0.0.187 | 30 Jun | Integer-overflow runtime traps now carry a precise `overflow` trap kind instead of the generic `unreachable` ([#808](https://github.com/aallan/vera/issues/808)). |
-| v0.0.188 | 30 Jun | The no-silent-failures discipline reaches the diagnostics layer: every diagnostic is gated for complete, spec-accurate metadata, with 54 checker sites backfilled and 30-plus stale spec references corrected ([#682](https://github.com/aallan/vera/issues/682)). |
-| v0.0.189 | 1 Jul | The UTF-8 "safe decode" invariant is centralised into one `safe_utf8_decode` helper, reached only through a shared `_slice_and_decode` helper that the three WASM-memory string readers (which the host imports delegate to) route through, replacing six brittle source-grep tests with a unit test plus three wire-real end-to-end tests ([#592](https://github.com/aallan/vera/issues/592)). |
+| v0.0.188 | 30 Jun | Every diagnostic is gated for spec-accurate `rationale` and `spec_ref` metadata, plus a `fix` for errors ([#682](https://github.com/aallan/vera/issues/682)). |
+| v0.0.189 | 1 Jul | The UTF-8 "safe decode" invariant is centralised behind one `safe_utf8_decode` helper, replacing six brittle source-grep tests with behavioural coverage ([#592](https://github.com/aallan/vera/issues/592)). |
+| v0.0.190 | 1 Jul | Text I/O (files and `subprocess` captures) is UTF-8 regardless of host locale, gated in pre-commit/CI, letting the `PYTHONUTF8` backstop be removed ([#645](https://github.com/aallan/vera/issues/645)). |
 
 ---
 

README.md

@@ -215,7 +215,7 @@ cp /path/to/vera/SKILL.md ~/.claude/skills/vera-language/SKILL.md
 
 ## Project status
 
-Vera is in **active development** at v0.0.189 — 1,700+ commits, 189 releases, 5,503 tests, 91% code coverage, 103 conformance programs, 35 examples, and a 13-chapter specification. See **[HISTORY.md](HISTORY.md)** for how the compiler was built.
+Vera is in **active development** at v0.0.190 — 1,700+ commits, 190 releases, 5,557 tests, 91% code coverage, 103 conformance programs, 35 examples, and a 13-chapter specification. See **[HISTORY.md](HISTORY.md)** for how the compiler was built.
 
 The reference compiler — parser, AST, type checker, contract verifier (Z3), WASM code generator, module system, browser runtime, and runtime contract insertion — is working. The language specification is in draft across [13 chapters](spec/).
 

ROADMAP.md

@@ -8,7 +8,7 @@ Priority lives in this file and nowhere else — issues carry kind and area labe
 
 ## Where we are
 
-5,503 tests, 103 conformance programs, 35 examples, 13 spec chapters.
+5,557 tests, 103 conformance programs, 35 examples, 13 spec chapters.
 
 ## The roadmap
 
@@ -20,7 +20,6 @@ The infrastructure that catches the next regression before a user does, plus the
 
 | Issue | What |
 |---|---|
-| [#645](https://github.com/aallan/vera/issues/645) | Explicit `encoding='utf-8'` at every text-mode file call, with a pre-commit check to hold the line. |
 | [#657](https://github.com/aallan/vera/issues/657) | Convert `INVARIANT_DEFENSIVE` sites and audit `PROPAGATE` cleanup (follow-up to the #626 error-handling taxonomy). |
 | [#419](https://github.com/aallan/vera/issues/419) | Split `tests/test_codegen.py` (21,093 lines — the largest file in the tree, and the codegen oracle for the mutation sweep) into feature-focused test files.  Promoted from Tier 3. |
 | [#420](https://github.com/aallan/vera/issues/420) | Split `tests/test_checker.py` (6,347 lines) into phase-focused test files.  Promoted from "Not doing now". |