Improve deterministic math routing guidance - #605
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a883fdc83
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Transfer-pilot update (GPT-5.5, reasoning low, one run per condition): A planned 6+6 Docker-backed Harbor run could not start on macOS. Harbor 0.20 rejected the required allowlist/sidecar topology before container or model startup, so that attempt incurred no model runs and produced no benchmark scores. Upstream Harbor #2594 intentionally does not broaden its native no-network fallback to sidecar treatment topologies, so I did not weaken isolation. I then ran a trajectory-only fallback using the exact mathematical kernels and inputs from six real Harbor tasks: rational linear solving, Hermite normal form, polynomial normalization, SAT witness, graph counterexample, and polynomial-map collision. Results:
The three strict misses need context:
Conclusion: the earlier synthetic 23/27 -> 27/27 result remains valid for that suite, but this first real-task-kernel transfer sample shows no outcome or verification improvement and a 5-call / 30% visible-payload increase. This PR should remain draft while the guidance is shortened or a broader Linux Harbor evaluation provides evidence that the added routing cost pays for itself. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1cc2e552a8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Address Codex review on deterministic routing guidance: do not mandate search-before-compose, key recovery to discovery response fields, allow iterative payload correction under resource bounds, continue after one unavailable provider, and preserve completeness plus open obligations in compact handoff. Keep packaged and repository skills byte-identical. Co-authored-by: morluto <morluto@users.noreply.github.com>
Include first-class artifact references and verification record URIs in the compact handoff list so artifact-backed results remain evidence-bound for replay, retrieval, and later VERIFY calls.
Co-authored-by: morluto <morluto@users.noreply.github.com>
945996b to
5a566cd
Compare
What changed
Why
A controlled GPT-5.5 visibility experiment showed that the current skill sometimes solved positive tasks manually or substituted supporting arithmetic operations even when a direct installed capability existed. The misses appeared in independent probability, exact optimization, and multi-invariant cases.
The deterministic-first routing prototype was iterated against those failures and then rerun across the complete diverse treatment set.
Evaluation evidence
Same model, task suite, execution budget, MCP surface, and three repetitions per case:
The final treatment covered determinant computation and verification, factorization verification, polynomial resultants, finite probability, rational linear optimization, multiple integer invariants, and two non-math abstention cases.
Temporary evaluation configs and transcripts are intentionally not included in this production PR.
Overlap review
Validation
quick_validate.py: passpytest -q tests/unit/tooling/test_codex_visibility.py: 13 passedmake check-changedreaches repository-wide lint but is currently blocked by ten unrelated errors already present onmainintests/unit/support/test_copy_template.pyandtests/unit/tooling/test_audit_fixes.py; this branch does not modify those files.