fix: structural issue batch — bounded reads, oracle coherence, protocol comparison, vLLM percentiles - #293
Conversation
Replace unbounded stream.readline() in _excerpt() with a bounded chunk-based discard loop (_discard_rest_of_line). Peak memory is now bounded by _DISCARD_CHUNK_SIZE (4 KiB) instead of the full physical line length, which could be gigabytes for newline-free artifact lines. Closes #291
Add a cross-field model_validator to OracleReceiptV1 that rejects receipts where the categorical status contradicts the quantitative evidence. A status='pass' receipt with absolute_error exceeding tolerance.absolute is now rejected at validation time, and a status='fail' receipt with error within tolerance is rejected too. Receipts without quantitative evidence are accepted with any status. Closes #290
…iter-joined strings
Replace _normalize()'s unescaped delimiter encoding for dicts
('k1=v1,k2=v2') with canonical JSON serialization (sorted keys). The
old encoding could make distinct kv_transfer_config dicts produce
identical normalized strings when keys or values contained commas or
equals signs, causing incompatible server configurations to be
declared comparable.
Closes #288
…known/auto-detected as none Stop rewriting plan.quantization=None as 'none' when constructing ModelIdentity. An omitted quantization field means the provider will auto-detect (e.g. vLLM reads the model's quantization_config), which is not the same as explicitly unquantized. Preserving None lets the protocol comparator correctly distinguish unknown/auto-detected from explicitly-none. Closes #289
…utable Add twine>=6.1 to the dev dependency group so it is lockfile-bound instead of network-resolved at release time. Replace the unpinned 'uvx --from twine twine check' with 'uv run twine check' so the validation tool is installed from the committed uv.lock, not dynamically resolved. Also fix typing: add TextIO annotation to _discard_rest_of_line and use OracleStatus enum in oracle receipt tests. Closes #292
…ggregate statistics - Use exact percentile rank in metric name instead of int() truncation (p99.1 and p99.9 both produced 'p99' before) - Add std_* fields to the non-negative latency validator - Reject infinity for actual_duration and time_scale - Reject missing 'duration' field instead of defaulting to 0.0 Closes #287
The release.yml change requires workflow scope to push. Keep the pyproject.toml/uv.lock twine pinning and the typing fixes; apply the workflow change separately when authorized.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…lways running baseline first
…instead of Python equality Add scalar_identity(), scalar_equal(), scalar_contains(), scalar_subset() and scalar_identity_set() helpers that distinguish bool/int/float/str by exact JSON type. Replace all Python equality (==, in, set operations) in experiment factor validation with these type-safe helpers. Closes #258
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17e945a13c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 529aefb037
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d3ef1b709
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd57b18e80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6ebc9922ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4762de6777
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ee72e6f68
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| variant_parameter=variant_parameter, | ||
| variants=variants, | ||
| baseline_variant=( | ||
| self._factor_label(config.baseline_value) |
There was a problem hiding this comment.
When a treatment factor contains type-distinct values with the same display label, such as integer 1 and string "1", converting baseline_value through _factor_label() collapses both to "1". The plan then cannot distinguish the declared baseline: trials and run-set selections are grouped under the same label, causing automatic comparison to be skipped or outcome failures to be attributed to the wrong side. Store the typed scalar identity rather than only its display label.
AGENTS.md reference: AGENTS.md:L46-L49
Useful? React with 👍 / 👎.
| line, | ||
| metric, | ||
| unit, | ||
| _input_kind, |
There was a problem hiding this comment.
When a scaling experiment mixes integer and floating inputs, _input_kind is discarded here before computing Spearman correlations, so type-distinct workload points such as integer 1 and floating 1.0 are pooled onto the same numeric axis. Although this revision excludes mixed-kind fits, it can still report misleading hotspot coefficients and p-values from the same incompatible population; exclude or stratify mixed-kind hotspot correlations as well.
AGENTS.md reference: AGENTS.md:L46-L49
Useful? React with 👍 / 👎.
1 participant
Summary
This PR addresses multiple critical and high-severity open issues in the flameox codebase:
Fixed issues
fix(summaries): discard long excerpt lines with bounded streaming reads #291 — Bounded streaming reads for excerpt discarding: Replaced unbounded in with a bounded chunk-based discard loop (). Peak memory is now bounded by (4 KiB) instead of the full physical line length, which could be gigabytes for newline-free artifact lines.
fix(oracles): derive validation status from coherent receipt evidence, not a caller-authored enum #290 — Derive validation status from coherent receipt evidence: Added a cross-field to that rejects receipts where the categorical contradicts the quantitative evidence (e.g. with ).
fix(inference): compare protocol facets structurally instead of delimiter-joined strings #288 — Structural protocol facet comparison: Replaced 's unescaped delimiter encoding for dicts () with canonical JSON serialization (sorted keys). The old encoding could make distinct dicts produce identical normalized strings when keys or values contained commas or equals signs.
fix(inference): record effective quantization instead of rewriting unknown/auto-detected as none #289 — Record effective quantization: Stop rewriting as when constructing . An omitted quantization field means the provider will auto-detect, which is not the same as explicitly unquantized.
fix(vllm): preserve exact percentile identity and reject incoherent aggregate statistics #287 — Preserve exact percentile identity: Use exact percentile rank in metric name instead of truncation (p99.1 and p99.9 both produced 'p99'). Added std_* fields to non-negative latency validator. Reject infinity for and . Reject missing 'duration' field instead of defaulting to 0.0.
fix(release): pin all executed build tools and attest artifacts before publishing #292 — Pin all executed build tools: Added to the dev dependency group so it is lockfile-bound. Replaced the unpinned with . (Note: the change requires scope to push and will be applied separately.)
Commands run
All 924 tests pass. Ruff and mypy clean.
Continue this on Linzumi