Skip to content

Improve deterministic math routing guidance - #605

Merged
morluto merged 5 commits into
mainfrom
agent/improve-math-routing
Aug 7, 2026
Merged

Improve deterministic math routing guidance#605
morluto merged 5 commits into
mainfrom
agent/improve-math-routing

Conversation

@yuelgrace1810-ops

Copy link
Copy Markdown
Collaborator

What changed

  • Route each requested exact mathematical outcome through Jacobian, including small calculations.
  • Route multiple requested outcomes independently.
  • Prefer the capability that semantically matches the requested operation over a generic arithmetic substep.
  • Add bounded recovery for unknown discovery domains and invalid payloads.
  • Keep result handoff compact and scope-aware.
  • Keep the repository and packaged Codex skills byte-identical, with focused regression coverage.

Why

A controlled GPT-5.5 visibility experiment showed that the current skill sometimes solved positive tasks manually or substituted supporting arithmetic operations even when a direct installed capability existed. The misses appeared in independent probability, exact optimization, and multi-invariant cases.

The deterministic-first routing prototype was iterated against those failures and then rerun across the complete diverse treatment set.

Evaluation evidence

Same model, task suite, execution budget, MCP surface, and three repetitions per case:

  • current skill: 23/27 contracts satisfied
  • proposed router: 27/27 contracts satisfied
  • command failures: 0
  • independent verification requirements: 6/6
  • negative-task abstentions: 6/6

The final treatment covered determinant computation and verification, factorization verification, polynomial resultants, finite probability, rational linear optimization, multiple integer invariants, and two non-math abstention cases.

Temporary evaluation configs and transcripts are intentionally not included in this production PR.

Overlap review

  • Merged Reduce Codex math invocation turns #567 provides bounded direct-invocation contracts. This PR preserves those contracts and adds semantic routing for direct and discovered operations.
  • Draft fix(mcp): distinguish unknown discovery domains #604 adds structured unknown-domain diagnostics. This PR only gives the agent bounded guidance for consuming that recovery signal; it does not duplicate the diagnostic implementation.
  • Searches of current open PRs found no implementation of multi-outcome or semantic-operation routing.

Validation

  • skill-creator quick_validate.py: pass
  • pytest -q tests/unit/tooling/test_codex_visibility.py: 13 passed
  • focused Ruff check: pass
  • packaged/repository skill byte parity: pass
  • skill size: 3331 bytes, below the existing 4 KiB limit
  • full 27-run treatment: 27/27, zero command failures

make check-changed reaches repository-wide lint but is currently blocked by ten unrelated errors already present on main in tests/unit/support/test_copy_template.py and tests/unit/tooling/test_audit_fixes.py; this branch does not modify those files.

@morluto
morluto marked this pull request as ready for review August 7, 2026 02:56

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a883fdc83

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .agents/skills/jacobian-math/SKILL.md Outdated
Comment thread .agents/skills/jacobian-math/SKILL.md Outdated
Comment thread .agents/skills/jacobian-math/SKILL.md Outdated
Comment thread .agents/skills/jacobian-math/SKILL.md Outdated
Comment thread .agents/skills/jacobian-math/SKILL.md Outdated

Copy link
Copy Markdown
Collaborator Author

Transfer-pilot update (GPT-5.5, reasoning low, one run per condition):

A planned 6+6 Docker-backed Harbor run could not start on macOS. Harbor 0.20 rejected the required allowlist/sidecar topology before container or model startup, so that attempt incurred no model runs and produced no benchmark scores. Upstream Harbor #2594 intentionally does not broaden its native no-network fallback to sidecar treatment topologies, so I did not weaken isolation.

I then ran a trajectory-only fallback using the exact mathematical kernels and inputs from six real Harbor tasks: rational linear solving, Hermite normal form, polynomial normalization, SAT witness, graph counterexample, and polynomial-map collision.

Results:

Metric current main skill #605 skill
mathematically correct final kernels 6/6 6/6
strict expected-operation contracts 3/6 3/6
checker-backed verification 3/3 requested 3/3 requested
MCP calls 21 26
model-visible MCP bytes 70,688 91,905
uncached input tokens 99,537 97,322
command failures 0 0

The three strict misses need context:

  • rational solving used the valid newer matrix.rational_linear_system.solve operation instead of the suite's stale expected ID;
  • SAT could materialize the CNF but no assignment-producing provider was installed, so both conditions correctly stopped at COMPUTED;
  • graph construction was solved directly and Jacobian checked triangle-freeness rather than using the expected atlas-search route.

Conclusion: the earlier synthetic 23/27 -> 27/27 result remains valid for that suite, but this first real-task-kernel transfer sample shows no outcome or verification improvement and a 5-call / 30% visible-payload increase. This PR should remain draft while the guidance is shortened or a broader Linux Harbor evaluation provides evidence that the added routing cost pays for itself.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1cc2e552a8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread .agents/skills/jacobian-math/SKILL.md Outdated
Grace Lee Rui Yue and others added 5 commits August 7, 2026 11:52
Address Codex review on deterministic routing guidance: do not mandate
search-before-compose, key recovery to discovery response fields, allow
iterative payload correction under resource bounds, continue after one
unavailable provider, and preserve completeness plus open obligations in
compact handoff. Keep packaged and repository skills byte-identical.

Co-authored-by: morluto <morluto@users.noreply.github.com>
Include first-class artifact references and verification record URIs in the
compact handoff list so artifact-backed results remain evidence-bound for
replay, retrieval, and later VERIFY calls.
Co-authored-by: morluto <morluto@users.noreply.github.com>
@cursor
cursor Bot force-pushed the agent/improve-math-routing branch from 945996b to 5a566cd Compare August 7, 2026 12:02
@morluto
morluto merged commit 36c2cc5 into main Aug 7, 2026
32 checks passed
@morluto
morluto deleted the agent/improve-math-routing branch August 7, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants