feat(runtime-baseline): add frozen threshold module + verifier (#390) - #392
Conversation
Establish the threshold-freezing mechanism that serves as the #351 quantitative acceptance gate. This PR delivers the methodology plus placeholder constants, not fabricated numbers: - thresholds.rs: #351 acceptance threshold constants as placeholder sentinels (f64::NAN / usize::MAX / u64::MAX / i64::MAX, TODO(#351)) + all_frozen() - verify.rs: Outcome comparator returning Publishable / NonPublishable / Fail. While thresholds are still placeholders, any case is NonPublishable — this is the hard guarantee that no placeholder value is mistaken for a pass. (Design spec forbids freezing Windows / hosted-runner numbers.) - schema/outcome.schema.json: outcome structure expected by the verifier - docs/performance/storage-runtime-baseline.md: methodology + the Linux-only backfill plan Real values are filled on a fixed Linux/WSL host per a versioned cases.yaml (Baseline Results PR); only then does the verifier flip to Publishable. The observer wiring (Tasks 6-7) and the real-number matrix are separate Linux-host follow-ups. Refs #350 #378 #351.
|
Warning Review limit reached
Next review available in: 2 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Replace incomplete run-level publishability with a strict single-case checker. Validate frozen policy values and case-specific inputs before evaluating every declared threshold, while leaving manifest conservation, repetition, CV, and provenance to the later controller. Constraint: A single-case Pass is not a publishable baseline-run decision. Confidence: High; boundary tests cover each threshold and isolated mutations prove the resume-drain and PING checks are observable. Scope-risk: Limited to runtime-baseline threshold inputs, schema, fixture assembly, license configuration, and baseline documentation. Tested: WSL cargo test -p runtime-baseline --all-targets; WSL cargo clippy -p runtime-baseline --all-targets -- -D warnings -D clippy::unwrap_used; cargo fmt --all -- --check; SkyWalking Eyes header check; git diff --check. Not-tested: Fixed-host runtime baseline matrix remains deferred until real thresholds and the controller verifier exist. Co-authored-by: OmX <omx@oh-my-codex.dev>
1 participant
Summary
#390 / #350 follow-up: establish the threshold-freezing mechanism for the #351 quantitative acceptance gate without inventing benchmark numbers.
This PR now provides:
tools/runtime-baseline/src/thresholds.rs: placeholder constants plus a validatedThresholdPolicy; any placeholder or invalid policy returnsPolicyUnavailable.tools/runtime-baseline/src/verify.rs: a strict single-case threshold checker returningPass,PolicyUnavailable, orFail.tools/runtime-baseline/schema/outcome.schema.json: a strict JSON Schema 2020-12 contract with required common fields and tagged case-specific metrics.tools/runtime-baseline/tests/build_identity_test.rs: fixture coverage for the new source modules and embedded schema.docs/performance/storage-runtime-baseline.md: the fixed-host measurement and follow-up contract.A single-case
Passis not a publishable baseline-run decision. The later controller verifier must still prove manifest case-set conservation, required repetitions, CV, source identity, and provenance before publishing a complete run.Why placeholders, not real numbers
The design prohibits freezing thresholds from Windows or GitHub-hosted runner measurements. The complete matrix must run on the designated fixed Linux/WSL host with the versioned case definition.
Until real measurements replace every placeholder,
ThresholdPolicy::from_frozen_constants()fails closed and the checker returnsPolicyUnavailablefor otherwise valid input. Invalid input returnsFaileven while the policy is unavailable.Deferred follow-ups
Validation
cargo test -p runtime-baseline --all-targetsin WSL/Linux: 49 passed, 0 failed.cargo clippy -p runtime-baseline --all-targets -- -D warnings -D clippy::unwrap_used: passed.cargo fmt --all -- --check: passed.git diff --check: passed.Refs
Refs #390. Refs #350 #378 #351 #347.