v0.0.15 — VB-T5-002 verification fix for Vera v0.0.176+

Highlights

Forward-compat methodology release for Vera v0.0.176 and later. Fixes a latent verification gap in VB-T5-002's canonical solution that Vera v0.0.176 surfaced when it closed aallan/vera#730 ("Call-site precondition checking now covers calls in statement position").

Unblocks vera-bench CI on current Vera HEAD — any PR opened against main since 2026-06-22 was inheriting the regression. Surfaced by #85 (a routine actions/checkout v6→v7 dependabot bump that was innocent but inherited the CI failure).

---

Fixed

• VB-T5-002 greeter_io_boundary canonical Vera solution removed unused greet_all and greet_loop helper functions that were not required by the problem spec. The helpers contained a latent verification gap: greet_loop called greet(...) in statement position (discarded result) without propagating greet's precondition requires(string_length(@String.0) > 0). Pre-Vera v0.0.176 the verifier silently skipped statement-position calls during call-site precondition checking (aallan/vera#730), hiding the gap. v0.0.176 closed that hole, and vera verify on the pre-fix solution failed with E501 and a counterexample showing an empty-string array element would violate greet's contract. The fix simplifies the canonical solution to exactly what the problem asks for: the effect IO declaration, build_greeting (pure helper), and greet (IO entry point). No greet_all / greet_loop.

Compatibility note

This is a forward-compat methodology release for Vera v0.0.176 and later. Same shape as v0.0.11 (Aver 0.16 Console.print typing) and v0.0.14 (upcoming Aver Int.div migration): an upstream compiler tightened its semantics, vera-bench had a latent gap, and this release updates the canonical solution to be strictly correct.

For Vera ≤ v0.0.175 (when statement-position calls were silently skipped by the call-site precondition checker), the previous canonical solution worked but was technically writing incorrect Vera. For Vera ≥ v0.0.176, v0.0.14 and earlier vera-bench releases fail vera verify on VB-T5-002 — this is the unblock.

Bisection trail: v0.0.175 passed verification (9 Tier 1 + 1 Tier 3), v0.0.176 introduced the call-site check and failed with E501, v0.0.177 carried the same failure forward. Confirmed locally in a fresh venv across all three releases.

Vera, Aver, Python, TypeScript, and AILANG scoring for other problems are unaffected — the change is scoped to the single VB-T5-002 canonical solution. All other 59 canonical Vera solutions still verify cleanly against v0.0.177.

---

Full diff: v0.0.14...v0.0.15

v0.0.14 — Aver Int.div migration

Highlights

Forward-compat Aver baseline migration ahead of an upcoming Aver release that drops the partial integer / operator. The five Aver baselines that used integer division swap to Int.div(a, b) : Result<Int, String>, with two of them gaining a genuinely cleaner refactor along the way.

Contributed by @jasisz, Aver's upstream author — the authoritative source on Aver semantics.

---

Changed

• Aver baselines migrated from the integer / operator to Int.div (#82, by @jasisz, Aver's upstream author). Aver's upcoming release drops the partial integer / operator — integer division can divide by zero (and overflow on i64::MIN / -1), so it's now the Result-returning function Int.div(a, b) : Result<Int, String>, matching Int.mod and Aver's "partial operations are functions" rule. (Float / stays a total operator.) Five Aver baselines used integer /:

  • VB_T3_011_safe_divide and VB_T5_003_safe_division_exceptions consume Int.div's Result idiomatically via match Ok/Err. The redundant manual b == 0 checks are gone — Int.div already reports the failure — and try_div in T5-003 simplifies to Int.div(a, b), since the function was literally re-implementing what Int.div now does.

  • VB_T1_007_safe_modulo (a - (a / b) * b), VB_T4_002_greatest_common_divisor (a - (a / b) * b), and VB_T4_007_count_digits (n / 10) use Result.withDefault(Int.div(...), 0) — the divisor is precondition-guaranteed non-zero in each context, so the sentinel never fires in practice.

Compatibility note

Aver scoring on the upcoming Aver release (post-0.23) requires v0.0.14 — without this release, every aver run against an Int.div-aware compiler would fail. For Aver ≤ 0.23, the converse applies: these baselines will not compile against Int.div-less compilers. Result files are tagged with bench_version so cross-version comparisons can detect this boundary.

Same forward-compat shape as v0.0.11's Console.print("{x}") migration for Aver 0.16. Vera, Vera spec-from-NL, Python, TypeScript, and AILANG scoring are unaffected — 0.0.14 is purely an Aver baseline migration for those languages.

---

Full diff: v0.0.13...v0.0.14

v0.0.13 — Claude 4 deprecation migration

Highlights

Maintenance release. Pre-emptive migration off Anthropic's deprecated Claude 4 model IDs (claude-opus-4-20250514 and claude-sonnet-4-20250514), both of which retire 2026-06-15. No methodology change.

• Default Anthropic flagship: claude-opus-4-20250514 → claude-opus-4-8 (Claude Opus 4.8)
• Default Anthropic Sonnet-tier: claude-sonnet-4-20250514 → claude-sonnet-4-6 (Claude Sonnet 4.6)

Per the 4.6-generation naming convention, the new model IDs are dateless and are themselves pinned snapshots (not evergreen aliases).

---

Changed

• Default Anthropic flagship migrated from Claude Opus 4 to Claude Opus 4.8 (claude-opus-4-20250514 → claude-opus-4-8). Opus 4 is deprecated and retires 2026-06-15. Per the 4.6-generation naming convention, the new model ID is dateless and is itself a pinned snapshot (not an evergreen alias).

• Default Anthropic Sonnet-tier migrated from Claude Sonnet 4 to Claude Sonnet 4.6 (claude-sonnet-4-20250514 → claude-sonnet-4-6). Sonnet 4 is deprecated and retires the same day as Opus 4 (2026-06-15); without this migration the Sonnet benchmark slot would start returning 404 from the API.

• Affected files: scripts/run_full_benchmark.py (MODELS dict + docstring examples), scripts/plot_results.py (MODELS list of ModelSpecs), scripts/README.md (example commands + slug-convention prose), README.md (Quick start examples — 9 occurrences, all swapped via replace_all after verifying the historical v0.0.7 results table on lines 24/31 uses the marketing names "Claude Opus 4" / "Claude Sonnet 4" and is unaffected), and test fixtures in tests/test_models.py, tests/test_cli.py, tests/test_runner.py.

• Deliberately untouched: scripts/plot_slide.py (v0.0.7 talk-slide renderer, pinned to the v0.0.7 lineup) and the historical results table / narrative in README.md (locked to v0.0.7 data per the chart-pin policy documented in KNOWN_ISSUES.md).

Compatibility note

Vera, Vera spec-from-NL, Python, TypeScript, Aver, and AILANG scoring are unaffected — this release is purely a default-identity change for the Anthropic provider. Users who explicitly pass --model claude-opus-4-20250514 or claude-sonnet-4-20250514 will keep working until 2026-06-15 (when the API retires those IDs); after that they'll receive 404s regardless of vera-bench version and need to migrate downstream.

---

Full diff: v0.0.12...v0.0.13

v0.0.12 — AILANG comparison language + parallel sweeps

Highlights

• AILANG as fourth comparison language alongside Python, TypeScript, and Aver — another zero-training-data language (#70, #75)
• --parallel N flag for concurrent benchmark sweeps via ThreadPoolExecutor (#73) — useful for slow models (e.g. Kimi K2.5 goes from ~50s/problem sequentially to ~5s/problem at --parallel 10)
• Worker crash recording in run_benchmark — previously, parallel-path crashes vanished silently from the JSONL (#73)

---

Added

• AILANG comparison language (#70, #75). Fourth comparison language alongside Python, TypeScript, and Aver. AILANG is another zero-training-data language, providing an additional data point for the language-design-vs-training-data thesis. Includes: prompt builder (build_ailang_prompt / build_ailang_fix_prompt), code evaluator (_evaluate_ailang_code with per-test-case main synthesis), baseline runner (run_ailang_baseline), CLI plumbing (--language ailang for both run and baselines), and 60 canonical reference solutions (solutions/ailang/*.ail). The AILANG teaching prompt is loaded via ailang prompt --source embedded (matching the AILANG eval-harness pattern) — no URL fetching. The full-benchmark sweep script (scripts/run_full_benchmark.py) now includes AILANG LLM + AILANG baseline targets (#75), bringing the matrix from 8 to 10 targets per model.

• --parallel N flag for concurrent benchmark sweeps (#73). Dispatches problems to a ThreadPoolExecutor with N workers. Each worker is I/O-bound on its LLM call + subprocess check/run, so the GIL is not a bottleneck. Default parallel=1 preserves the existing sequential code path. Validated with click.IntRange(min=1) to reject 0/negative at parse time.

• OpenRouter client for accessing AILANG-capable models via the OpenAI-compatible OpenRouter API. Used by the AILANG LLM-eval mode to reach models not directly available from Anthropic/OpenAI/Moonshot. Explicit error handling for AuthenticationError, RateLimitError, BadRequestError, APIStatusError, empty choices, and empty content (with finish_reason surfaced).

• AILANG retry-on-error in run_single_problem. The existing build_ailang_fix_prompt was previously dispatched only as unreachable code; now --max-fix-attempts > 0 actually retries AILANG failures (was silently no-op, undercounting AILANG vs Aver/Vera by the entire attempt-2 contribution).

• Worker crash recording in run_benchmark. Both sequential and parallel paths now synthesise a ProblemResult with traceback.format_exc() in error_message when run_single_problem raises. Previously, parallel-path worker crashes vanished silently from the JSONL — a 60-problem sweep with 2 crashes wrote 58 rows and downstream vera-bench report showed "58/58 (100%)" with no record of the crashes.

Changed

• Per-test runtime error capture in AILANG evaluator. The _evaluate_ailang_code loop previously continued on timeout and non-zero exit, so a row where every test failed at runtime was indistinguishable from one that compiled but produced wrong output (both showed check_pass=True, run_correct=False, tests_passed=0, error_message=None). Now captures the first non-zero stderr/stdout/exit-N marker into error_message (truncated to 400 chars). Issue #72 tracks the broader per-test stderr aggregation shared with the Aver path.

• Compile-vs-runtime tag classification in baseline_runner.py is now regex-based (re.search(r"\bError ([A-Z]+)_", err)) with an explicit compile_tags = ("PAR", "TC", "MOD", "ELB", "LINK", "TY") allow-list. Previously used substring matching, which would have silently misclassified a future AILANG release adding Error PARSER_ (matches Error PAR) or any new tag.

• Sequential and parallel run_benchmark paths now share fault semantics. Pre-#73, --parallel 1 aborted on any worker exception while --parallel 2+ logged-and-continued. Now both paths wrap run_single_problem in identical try/except and route crashes through _crash_result + _record helpers.

Fixed

• _strip_ailang_main brace-counter bug: the previous heuristic mis-classified canonical AILANG main lines like export func main() -> () ! {IO} { because {IO} provides balanced braces, leaving the body as orphan code that broke the injected per-test-case main. Replaced with indentation + bare-} regex sentinel logic.
• _USER_AGENT constant in prompts.py was stuck at 0.0.9 since that release. Now matches the package version.

Compatibility note

Vera, Vera spec-from-NL, Python, TypeScript, and Aver scoring is unaffected — 0.0.12 is purely additive for those languages. The AILANG baseline + LLM-eval mode is a new fourth comparison target; result files from 0.0.12 onwards include AILANG rows that 0.0.11-and-earlier sweeps cannot have produced.

The --parallel N flag is opt-in; default parallel=1 exactly replicates the 0.0.11 sequential code path. JSONL line content is identical between sequential and parallel modes; only ordering may differ (parallel writes by completion order, not problem index).

---

Full diff: v0.0.11...v0.0.12

v0.0.11 — required for Aver 0.16+ Console=String

What's new

The Aver evaluation harness and all canonical Aver baselines migrated to string interpolation for compatibility with Aver 0.16's typed Console.print. Three previously-deleted baselines restored, nine coverage-gap fixes, and a real correctness improvement to Aver baseline scoring.

Required for Aver 0.16+ (#65, thanks @jasisz)

Aver 0.16 ("Anneal") tightens Console.print to require String — passing Int, Bool, List<T>, Option<T>, etc. is now a typecheck error. Without this release, every injected aver run from the bench's per-test wrapper would crash at typecheck on Aver 0.16+, dropping run_correct to 0% across the board.

The fix is the canonical Aver idiom: string interpolation. The harness now emits Console.print("{<call>}") instead of Console.print(<call>), and all 56 canonical baseline solutions migrated to the same form. The interpolation form predates the breaking change by many versions, so the same wrapper works on Aver 0.10 through 0.15 and on 0.16+.

Three Aver baselines restored

VB_T2_011_starts_with_prefix, VB_T2_012_ends_with_suffix, and VB_T2_013_get_char_code were originally added in v0.0.9 and then removed during PR #57 review because the Aver stdlib at the time didn't expose starts_with / ends_with / char_at. Aver 0.15+ now has String.startsWith, String.endsWith, String.charAt, and Char.toCode, so the three baselines are reinstated. VeraBench's Aver coverage now includes all 60 problems again.

Nine coverage-gap fixes

Nine baselines had main() printing only a subset of the test cases declared in the corresponding problem JSON. Until now this was masked: the baseline runner's run_correct check relies on stdout matching expected values, so if main() only printed 3 of 5 test cases, only 3 got compared. Migrating these to the new wrapper format surfaced the gap as a side effect, and main() was regenerated to print every test case.

This is a real correctness improvement, not just a stylistic change. It's the reason Aver baseline run_correct rises from 73% (v0.0.10) to 100% (v0.0.11) on Aver 0.15.2 — the previous denominator was inflated by partial-print masking.

Compatibility note

• On Aver 0.16+: required. Without this release, every aver run crashes at typecheck.
• On Aver 0.10–0.15: backward-compatible, but Aver scoring may differ slightly between v0.0.10 and v0.0.11 result files for the same model on the same problems, because (a) the 9 coverage-gap fixes change which test_cases get checked, and (b) the 3 restored T2 baselines now contribute to the run_correct denominator. Result files are tagged with bench_version so cross-version comparisons can detect the boundary.

Vera, Vera spec-from-NL, Python, and TypeScript scoring is unaffected.

Verification

Check	Result
Aver baselines on Aver 0.15.2	100% check@1, 100% run_correct, 60/60 across all 5 tiers
Python test suite	494 passed
Aver 0.16+ forward-compat	Verified locally on the released aver-lang 0.16.0 (2026-05-05): identical 100%/100% baselines as on 0.15.2
Backward-compat to Aver 0.10–0.15	Verified locally on 0.15.2

Headline chart

The chart in the top-level README (assets/results-graph.png) remains pinned to v0.0.7 content while the post-v0.0.7 writeup is being prepared. The v0.0.7 snapshot is also available as a release asset.

Contributors

Thanks again to @jasisz for tracking the Aver release cadence ahead of breaking changes — that's now twice in two weeks (v0.0.10 for Aver 0.13's effects boundary, v0.0.11 for Aver 0.16's Console=String) where the bench was forward-compat-ready before the upstream release shipped.

Full Changelog: v0.0.10...v0.0.11

v0.0.10 — Aver effects-strip for Aver 0.13 forward-compat

What's new

A targeted methodology change to the Aver evaluation harness, landing as forward-compat plumbing for the upcoming Aver 0.13 release.

Aver effects-strip (#62, thanks @jasisz)

Aver 0.13 will introduce a module-level effects [...] boundary that the type-checker enforces: every function's ! [Effect] must be covered, or compilation fails. The bench harness asks the LLM for a function only, then strips the LLM's main() and injects its own Console.print(fn(...)) main with ! [Console.print]. If the LLM declared a narrower module boundary in the original source — including the very common effects [] for a "pure" module — the injected main would violate that boundary and aver run would fail with an underdeclared-effects type error before any test case ran.

The new _strip_module_effects() pass in the harness removes the module-header effects [...] line so the module reverts to legacy / no-boundary mode and the injected main type-checks. The strip is:

• Window-scoped — only fires inside the module-header block, not on effects [...]-shaped lines elsewhere
• Whitespace-tolerant — effects[, effects [, effects [, and effects\t[ all strip cleanly
• Trailing-comment-tolerant — effects [...] // pure module strips correctly without falling into a runaway multi-line skip path

Compatibility note

This is a methodology change for Aver scoring: the same LLM output now goes through an extra strip pass before reaching the compiler.

• On Aver 0.12 and earlier the strip is a complete no-op — LLMs don't emit module-level effects [...] because the docs don't describe it yet — so today's Aver scores are byte-identical to v0.0.9.
• Once Aver 0.13 ships and the boundary becomes part of the doc nudge to models, Aver run_correct rates from v0.0.10 onwards will diverge from any v0.0.9-tagged Aver results run against Aver 0.13+ — the strip will activate on a measurable fraction of generations and prevent the underdeclared-effects type error.

Result files are tagged with bench_version so cross-version comparisons can detect the boundary.

Vera, Vera spec-from-NL, Python, and TypeScript scoring is unaffected.

Other changes

This release also incorporates the CI infrastructure work from #64, which adds a pip install --upgrade pip workaround for CVE-2026-3219 in pip 26.0.1 (the version bundled into actions/setup-python@v6's Python 3.12 toolchain image), and opens KNOWN_ISSUES.md as the catalogue location for active workarounds, dev-env gotchas, and analytical caveats. Tracked for cleanup in #63.

Headline chart

The chart in the top-level README (assets/results-graph.png) remains pinned to v0.0.7 content while the v0.0.9-and-later writeup is being prepared. The v0.0.7 snapshot is also available as a release asset.

Contributors

Thanks to @jasisz for the substantive Aver harness change, the regex-opener hardening that landed on the same PR, and the cross-language reviews that have shaped Aver scoring over the v0.0.8 / v0.0.9 / v0.0.10 series.

Full Changelog: v0.0.9...v0.0.10

v0.0.9 — 60 problems across 5 tiers

What's new

60 problems across 5 tiers

10 new problems (5 Tier 2, 5 Tier 3) expand the benchmark from 50 to 60 problems. The new problems all have testable signatures (primitive inputs/outputs), increasing the T1–T4 run_correct pool from 18 to 30 problems.

• New T2 problems (VB-T2-011 through VB-T2-015): string prefix/suffix checking, char codes, combined length, length comparison — testing string_starts_with, string_ends_with, string_char_code, and string_length discovery
• New T3 problems (VB-T3-011 through VB-T3-015): safe divide, pair sum, classify sign, color code, either select — all use Int-only signatures with internal ADT construction, testing pattern matching without requiring ADT CLI argument support
• Test cases added for VB-T2-004 (is_empty_string) and VB-T2-005 (contains_substring), which previously had none

Canonical solutions provided for all new problems in Vera, Python, TypeScript, and Aver.

Tier-split reporting (#50)

Reports now show separate "All Tiers (T1–T5)" and "Comparable (T1–T4)" summary sections for cross-language comparison. Tier 5 tests Vera's algebraic effect handlers, which other languages solve with fundamentally different native idioms — the T1–T4 aggregate provides a fairer cross-language headline rate.

• exclude_tiers parameter on compute_metrics() for tier-filtered aggregation
• Methodology note explaining why T5 is reported separately
• Comparable section suppressed when no T1–T4 problems are present

Contributors

Thanks to @jasisz for the tier-split methodology (#56) and Aver solution reviews.

Full Changelog: v0.0.8...v0.0.9

v0.0.8 — Aver support + description_neutral

What's new

• Aver language support — generation, checking, execution, fix-from-error, and baselines (PR #48 by @jasisz)
• description_neutral field on all 50 problem JSONs for fair cross-language prompting
• Python and TypeScript prompts now use language-neutral descriptions
• Full benchmark runner updated to 8 targets (added Aver LLM + Aver baselines)
• "Adding a New Comparison Language" guide in CONTRIBUTING.md
• description_neutral enforced by validator

See CHANGELOG.md for full details.

v0.0.7

What's new

• Moonshot (Kimi) provider support — OpenAI-compatible API via moonshot/* model prefix, with Kimi K2.5 (flagship) and Kimi K2 Turbo (fast) models
• Full benchmark runner script — scripts/run_full_benchmark.py runs all 6 targets with one command (interactive mode with provider/model/key menus, or autonomous via CLI args)
• Secure API key input via getpass in interactive mode

Improvements

• CI now installs LLM SDKs in test job for full model coverage
• Fixed model mock tests that were silently skipping (models.py coverage 48% to 87%)
• Subprocess timeout (30 min) on benchmark targets
• Fail-fast on unknown model provider prefix

Bug fixes

• Fixed Moonshot API endpoint (api.moonshot.cn to api.moonshot.ai)
• Fixed Moonshot model identifiers (kimi-k2 to kimi-k2.5 / kimi-k2-turbo-preview)
• Fixed FileNotFoundError for slash-prefixed model names (e.g. moonshot/kimi-k2.5)

Full Changelog: v0.0.6...v0.0.7

v0.0.6 — Test Coverage & Version Tracking

Test coverage increase and version tracking in results.

Added

• Bench and vera compiler versions in JSONL filenames and result records (#20)
• VeraRunner.version() method to query vera compiler version
• 4 new test files: test_cli.py, test_models.py, test_validate_integration.py, test_vera_runner_integration.py

Changed

• Test coverage: 66% → 83% (324 → 376 tests)
• CI coverage threshold raised from 35% to 80%

Filename format

model-bench-0-0-6-vera-0-0-105.jsonl         # vera full-spec
model-spec-from-nl-bench-0-0-6-vera-0-0-105.jsonl  # vera spec-from-nl
model-python-bench-0-0-6.jsonl                # python (no vera version)

Closes #20. Progress on #5.