feat(lanes): multi-language property lanes; add a Python/Hypothesis lane - #2
Merged
Merged
Conversation
Refactor the engine into pluggable per-language lanes behind the existing fail-closed core/report.mjs verdict + exit codes (marmorkrebs' multi-tool discipline, for property testing). The lane is auto-detected from the module extension or forced with --lane; core/report.mjs is unchanged (already language-agnostic). - Lane interface + registry (einsiedler/lanes/): fast-check (JS, in-process) becomes a lane implementing the interface; a generic SubprocessLane spawns a per-language runner that emits a normalized PropertyResult[] as JSON on stdout. - Python/Hypothesis lane (*.eks.py + einsiedler/runners/hypothesis_runner.py): a real, canary-validated proof of concept with shrunk counterexamples; a bool-returning property is wrapped so Hypothesis' bare-False-is-pass quirk can't hide a failure, and Unsatisfiable maps to VACUOUS. - Protocol generalized up front for the future Swift/Go/Rust/C++ lanes: seed is a string reproduction token, shrinkSteps is nullable, counterexample is a string, and the runner separates a graceful FALSIFIED from an unhandled ERROR. - validate-provider proves every installed lane's canary is live (SKIP when a runtime is absent, never a silent pass); per-lane canaries; --lane flag; README "Lanes" section; a worked Python example; tests (39 pass).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Multi-language property lanes. The engine is now a pluggable per-language lane registry
behind the existing fail-closed
core/report.mjsverdict + exit codes (marmorkrebs' multi-tooldiscipline, applied to property testing). Adds a real Python / Hypothesis lane as the
proof of concept.
core/report.mjs+decideVerdictare unchanged — they were alreadylanguage-agnostic.
Lanes
fast-check(JS/TS,*.eks.mjs, in-process) — moved into a lane implementing theinterface; no behaviour change (its canary + the 33 existing tests are unchanged).
hypothesis(Python,*.eks.py,python3subprocess) — a thin runner(
einsiedler/runners/hypothesis_runner.py) discovers the modules, drives Hypothesis, andprints a normalized
PropertyResult[]as JSON that a genericSubprocessLaneparses.Canary-validated; HELD / FALSIFIED / VACUOUS / ERROR mapped correctly; shrunk counterexamples
(the canary shrinks to
canary(1)). A bool-returning property is wrapped so Hypothesis' "bareFalseis a pass" quirk can't hide a failure — and any falsey return (0,"",numpy.False_)fails, not just literal
False; aNone(assertion-style void) is the pass; anasync def/generator property (whose body never runs) is anERROR, never a HELD. Modules areregistered in
sys.modulesbefore execution, so@dataclassand similar decorators work.The lane is auto-detected from the module extension or forced with
--lane <id>.Fail-closed hardening (found by dogfooding ClawSweeper on this PR)
Reviewing this PR with the local ClawSweeper caught several false-green paths — the exact defect
class this tool exists to prevent — and all are closed here:
validate-providervalidates explicitly-selected lanes; never exits 0 on a selected-but-absentruntime. By default it checks only bundled lanes (fast-check), so a JS-only consumer is never
forced to install Python — no upgrade break;
--all-lanes/--lane <id>opt a non-bundled lanein. Any selected lane whose runtime is absent is
QUARANTINED(exit 5), a distinct non-zeroverdict — no waiver launders it green. (This repo's CI runs
--all-lanesto prove every lane.)Argument parsing is strict (both the CLI and
validate-provider): an unknown lane id, anunrecognized option (
--all-lane), or a dangling--lanewith no value all fail loudly (exit 64)— a typo can never silently narrow the run and exit 0 having skipped what you asked for.
--validate --modules <dir>auto-detects the lane from the modules (so a Python dir validates the hypothesis canary, not fast-check).--author(fast-check-only) rejects a non-fast-check--lane(exit 64) rather than authoring forthe wrong engine.
validate-provider's lane list is derived from the single registry (einsiedler/lanes/ index.mjs), not duplicated — so a newly registered lane can't drift out of validation whilethe aggregate still reports PASS. "required" is the lane's own
bundledflag.--min-runsVACUOUSfloor. It used to report the requested max as
numRunsand ignore--min-runs, so aheavily-filtered run that executed only a handful of real examples was trusted as
HELD. Now afiltering-driven shortfall ⇒
VACUOUS(see the finite-domain refinement below).ERROR, not silently dropped (bothlanes). A
.eks.{mjs,py}that defines no properties tests nothing; without this, anothermodule's
HELDwould mask the coverage gap.bare
HELDwith no run count is downgraded toERROR— a pass must carry a finitenumRuns ≥ 1,so a buggy/dishonest runner can't forge an evidence-free green (the canary gate proves the runner
is honest; this proves it actually ran).
Each subprocess runs one module and must yield ≥ 1 result — an empty per-module result is an
ERROR, so a broken module can't vanish and let a sibling's HELD carry the run to green.
print()goes to stderr, never corrupting the payload), and module paths are passed after--so a dash-leading filename can't be mis-parsed as a flag.
The runner runs Python with
-Band the syntax check compiles in memory, so importing property/target modules never writes
__pycache__into the target checkout — the gate leaves a clean tree.forever — the subprocess is SIGKILLed after a timeout (120s default,
EINSIEDLERKREBS_LANE_TIMEOUT_MS);and a runner that floods stdout/stderr is killed once its output passes a 16 MiB cap, so a noisy
or runaway runner can't exhaust the parent's memory — bounded both per module (16 MiB) and across the whole invocation (64 MiB total). All fail closed to an ERROR.
property invocation, with no recorded args) is an
ERROR, never a mislabeledFALSIFIED; anda property module can import its sibling target source (
from clamp import clamp) — themodule's directory is placed on
sys.path.--min-runsstaysHELDonlywhen Hypothesis' own statistics PROVE it exhausted the whole reachable input space
(
stopped-because = "nothing left to do") — every value the strategy can produce was tested, thestrongest evidence there is. Any other shortfall (a heavy
.filter()/assume()the engine givesup on, or simply hitting the example budget) is
VACUOUS. This closes a false-green where a.filter()-limited run — whose rejections Hypothesis retries invisibly during generation, so arejection count can't see them — looked like a complete finite sweep. The floor lives in the
runner (which holds the authoritative stats); the subprocess adapter enforces only "a pass
executed ≥ 1 example".
per
.eks.py, so nothing an earlier module does can leak into a later one — not a same-namedtarget, not a rogue sibling
random.py, not an in-place monkeypatch ofrandom/builtinsormutated Hypothesis state. In-process reset can only restore object identity (not undo mutations),
so a fresh process per module is the only fail-closed isolation. A sibling that shadows a
stdlib/runtime name fails closed, never silently validating the wrong target.
parseInts--seed, and eachlane derives its engine seed from the WHOLE token: the hypothesis runner hashes it, and the
fast-check lane maps a pure integer to that exact seed but hashes any other token (so
12suffix≠
12and distinct non-numeric tokens map to distinct streams with overwhelming probability — via a portable inline hash — no new Node version requirement — bounded, like fast-check itself, by its 2³² seed space). Deterministic, so arun replays. Author mode applies the same conversion before driving fast-check.
--report-fileschema is versioned (schemaVersion: 2): the per-resultseedbecame astring replay token (was a fast-check integer) and a
lanefield was added, so a consumer canswitch on the version instead of breaking on the changed
seedtype. v2 is an intentional breaking change from the implicit v1 (numeric seed) — pre-1.0, with no released consumers to migrate; the version field is the signal.buildReportnormalizesevery seed to a string centrally, so even the
--authorpath (which feeds fast-check resultsstraight through) can't advertise v2 with a numeric seed.
check/lintscript is portable — the optional Pythonpy_compilestep runs through asmall Node helper (honoring
EINSIEDLERKREBS_PYTHON), not a POSIX shell group, sonpm run checkworks on Windows (cmd.exe) too, not just POSIX shells.
.github/workflows/ci.yml): actions are pinnedby commit SHA (not mutable
@vNtags), the Python runtime installs from a hash-lockedrequirements-ci.txt(--require-hashes, hypothesis + every transitive dep), and it runsvalidate:provider --all-lanes— so identical commits execute byte-identical third-party code andthis repo dogfoods every lane (fail closed if any is absent), while JS-only consumers keep the
bundled-only default.
Future-proofed for the next lanes
The subprocess protocol is generalized up front so Swift (SwiftCheck), Go (gopter /
testing/quick), Rust (proptest) and C++ (RapidCheck) drop in as "write a runner + register alane", no framework change:
seedis a string deterministic-replay token each lane feeds backto its own engine (a fast-check int seed, a hashed Hypothesis run seed, a proptest byte-array, …)
so the same token reproduces the same run,
shrinkStepsis nullable (Hypothesis and Gotesting/quickdon't expose it),counterexampleis a string, and the runner separates a gracefulFALSIFIEDfrom an unhandledERROR.Evidence
npm run check: JSnode --checkover the new lane files + Pythonpy_compile— exit 0.npm test: 74 pass (33 original + lane/registry + fail-closed regression tests; thehypothesis-lane tests run when the runtime is present, else self-skip). New regressions cover
the min-runs VACUOUS floor (num=5/min=100 ⇒ VACUOUS with the real executed count), the
validate-providerfail-closed aggregate (any SKIP ⇒ exit 5; FAIL ⇒ 1) and its registry-driftguard, and a present-but-empty property module ⇒ explicit ERROR.
npm run validate:provider(default) validates bundled lanes only → PASS even with noPython (no upgrade break);
-- --all-laneswith the runtime present → both lanes PASS, andwith it absent →
QUARANTINEDexit 5 (proven end-to-end), never a silent green.Proof (real runs,
hypothesis 6.156.6)Before / after (same command, base
cab3d96vs HEAD). On base the Python lane does not exist:--laneis ignored, the run falls back tolane=fast-check, resolves 0 modules, and failsNO_PROPERTY(exit 3). On HEAD the same command selectslane=hypothesisand drives a realHypothesis run — both
clampproperties HELD at 500 examples each,PASS(exit 0):Both lanes' canaries are live —
validate:provider --all-lanesdrives each lane's planted-falsecanary and confirms it is CAUGHT (
ENGINE_OK):A real Python property HOLDS (the worked
clampexample — 500 examples each), and the planted-falsecanary is FALSIFIED with a shrunk counterexample and the fail-closed exit code 2:
Review history
Review history (23 local ClawSweeper rounds): https://gist.github.com/f353c43ef23b0c6adf49e9c2e199a86f