Skip to content

Add the type-erased arg-parser runtime kernel with property tests - #566

Merged
Smaug123 merged 4 commits into
mainfrom
argparser-erased-core
Jul 15, 2026
Merged

Add the type-erased arg-parser runtime kernel with property tests#566
Smaug123 merged 4 commits into
mainfrom
argparser-erased-core

Conversation

@Smaug123

Copy link
Copy Markdown
Owner

Stacked on #565. Part of the arg-parser rewrite working towards discriminated-union (sum) argument schemas.

What this is

ArgParserRuntime is the future core of the generated argument parser: a pure, type-erased kernel that knows nothing about target types, only the shape of a schema (names, arity, repeatability, requiredness) and raw string tokens. It has three phases, all data-to-data:

  • scan — tokenizes argv into an ordered ScanEvent log (occurrences, positionals, errors, help, separator), preserving the historical grammar: greedy value consumption, boolean-like keys consuming only boolean literals, --no- negation, unknown-key laziness under [<PositionalArgs true>], and the Arg parser: a bare -- no longer discards pending parser state #563 rule that -- resolves pending state exactly like end-of-input.
  • select — chooses a case for every Sum node from which leaves were observed: exactly one touched case wins; two is a conflict; zero falls back to the unique case satisfiable with no arguments.
  • validate — missing-required / inactive-leaf / no-positional-sink structural errors, plus the list of leaves whose defaults must fire.

Because selection is shape-first (never depends on value conversion), the typed layer that will follow — converters, defaults, record assembly, message rendering — cannot change which argument a token was routed to nor which union case is selected. That is what will make it safe to embed this file verbatim into generated output (no runtime dependency) with only thin generated code per type.

Nothing consumes this yet: the generator cutover is the next PR.

The interesting part: the tests

The exhaustive reference semantics is implemented in the test suite: expand every Sum into its complete alternatives (P × (A + B) = P×A + P×B) and accept precisely those alternatives whose leaf set covers everything observed and whose required leaves were all observed; exactly one accepted alternative is a parse. FsCheck properties (with instrumented distribution assertions, so the tests fail if the generators stop exploring the interesting regimes):

  • select ≡ exhaustive expansion over random schema trees (fuzzed sum/product bias) and observed-sets biased from valid towards noisy/empty (fuzzed drop/optional/noise percentages), 2000 cases: unique-alternative ⟺ compositional accepts with identical choices and active leaves; zero alternatives ⟹ compositional errors; multiple ⟹ reported as ambiguity.
  • Scanning is lossless: unscan (scan args) = args for arbitrary token lists drawn from a mixed adversarial distribution (well/ill-formed keys, bool literals, separators, =-pathologies, unicode) — every input token is accounted for in exactly one event.
  • Trailing -- appends exactly a Separator event (the Arg parser: a bare -- no longer discards pending parser state #563 property, at the core level).
  • Round trip: rendering a random assignment through random syntax choices (= vs space, case flips, negation, bare flags with lookahead-aware fallback) scans to exactly the intended event list.

Plus unit tests pinning the token grammar and the motivating DU example (--foo=3 vs --bar=8 --baz=9; --foo=3 --bar=8 is a conflict; --bar=8 selects BarCase and reports missing --baz).

🤖 Generated with Claude Code

@Smaug123
Smaug123 force-pushed the argparser-erased-core branch 3 times, most recently from 268a9f8 to 332dc9b Compare July 14, 2026 07:59
@Smaug123
Smaug123 force-pushed the argparser-erased-core branch 3 times, most recently from 805ef3e to c844f1e Compare July 14, 2026 20:57
Base automatically changed from fix-opens-leak to main July 14, 2026 21:20
@Smaug123

Copy link
Copy Markdown
Owner Author

Codex reviewed clean, by the way. Still got to use my human eyes.

Smaug123 and others added 3 commits July 15, 2026 19:48
ArgParserRuntime is the future core of the generated argument parser: a
pure, type-erased kernel (scan argv into an ordered event log; select
discriminated-union cases; validate against the schema shape) which will
be embedded verbatim into generated parser output and drive a thin typed
layer of converters and record assembly. It supports Leaf/Product/Sum
schemas: sums are the upcoming DU-alternative feature, selected
shape-first ("exactly one interpretation consumes every argument") with
globally-unique argument names.

Nothing consumes it yet; the generator cutover comes separately.

Tests include an executable reference semantics (expand every sum into
its complete alternatives; accept exactly one) and FsCheck properties:
- compositional selection == exhaustive alternative expansion, over
  random schema trees and observed-sets fuzzed across valid/noisy/empty
  regimes, with instrumented distribution assertions;
- scanning is lossless (unscan . scan = id) for arbitrary token lists;
- a trailing "--" appends exactly a Separator event;
- round trip: rendered command lines scan to exactly the intended
  events, across =/space forms, case variation, negation, bare flags.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The selection semantics assume every addressable --token names at most
one claimant under the scanner's case-insensitive matching; without
that, matchLeaf silently routes colliding tokens to the first-declared
leaf. Generation-time checks cannot see forms supplied via [<Literal>]
constants, so generated code will re-check at runtime via
WellFormedSchema.checkOrFail before parsing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ToUpperInvariant keying is strictly coarser than OrdinalIgnoreCase (the
equality the scanner matches keys with): "s" and "ſ" uppercase to the
same string but are distinct keys, so the checked constructor falsely
rejected schemas the scanner routes unambiguously. Group claims with
StringComparer.OrdinalIgnoreCase instead.

Also reject forms no token can ever address: an empty form (its token
is the positional separator) and forms containing '=' (a --key=value
token splits at its first '='), either of which could otherwise leave a
required argument, or a whole union case, permanently unsatisfiable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Smaug123
Smaug123 force-pushed the argparser-erased-core branch from 3748b8e to f80f036 Compare July 15, 2026 19:08
Comment on lines +160 to +161
/// True if this error means "stop the parse immediately" (historically these were
/// exceptions thrown mid-scan rather than accumulated).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// True if this error means "stop the parse immediately" (historically these were
/// exceptions thrown mid-scan rather than accumulated).
/// True if this error means "stop the parse immediately".


/// Scan argv into an ordered event log. Pure: performs no conversion, throws no exceptions.
///
/// The grammar (deliberately preserving the historical parser's behaviour):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// The grammar (deliberately preserving the historical parser's behaviour):
/// The grammar:

go ScanState.AwaitingKey (ScanEvent.Help arg :: acc) rest
else
// (The string overload: char literals do not survive the parse-and-reprint
// round trip through which this file is embedded into generated code.)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a bug in Fantomas, by the way.

@Smaug123
Smaug123 merged commit 0df0aca into main Jul 15, 2026
20 checks passed
@Smaug123
Smaug123 deleted the argparser-erased-core branch July 15, 2026 19:33
Smaug123 added a commit that referenced this pull request Jul 15, 2026
Review edits which just missed the #566 squash: the isFatal and scan
grammar docstrings no longer describe the pre-rewrite parser's
behaviour as the reason for their shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant