Skip to content

benchmarks: audit perfect-cuboid finite scope - #592

Merged
morluto merged 5 commits into
mainfrom
agent/conjecture-perfect-cuboid-scope
Aug 7, 2026
Merged

benchmarks: audit perfect-cuboid finite scope#592
morluto merged 5 commits into
mainfrom
agent/conjecture-perfect-cuboid-scope

Conversation

@yuelgrace1810-ops

Copy link
Copy Markdown
Collaborator

Summary

  • add an Assurance-family Harbor conjecture probe derived from retained spreadsheet row C-012
  • require exact classification of twelve integer cuboids as Euler bricks, near-perfect cuboids, perfect cuboids, or other cases
  • independently recompute all face and space diagonal square predicates
  • reject incomplete case sets, altered roots, false global scope, evidence tampering, and unsupported VERIFIED claims

Benchmark contract

  • Family: Assurance
  • Primary objective: semantic scope handling for finite conjecture evidence
  • Quality score: 86/100
  • Difficulty: Medium–Hard (provisional); complete exact classification and assurance calibration are required, with empirical calibration pending
  • Source: retained Unresolved Conjecture row C-012, with definitions/status checked against MathWorld's Perfect Cuboid and Euler Brick references
  • Assurance boundary: CHECKED only for the twelve frozen cuboids; no global existence or nonexistence conclusion

Shortcut and duplication audit

A class label or one public Euler-brick example is insufficient. Every submission must provide all four radicands and exact roots/null results for all twelve cases. Case order is free, but IDs must be complete and unique. The workflow is distinct from the merged bounded Vizing probe and existing Diophantine witness searches because it specifically tests semantic separation and assurance calibration.

Validation

Passed locally:

  • focused/adversarial verifier tests: 5 passed
  • selected Harbor contract check
  • selected-task preparation and verifier checksum synchronization
  • Ruff lint and format for all changed Python
  • git diff --check
  • Docker Client/Server 29.6.2
  • direct task and verifier image builds and execution under --network none
  • full applicable direct-container reward: 1.0

Native Harbor 0.20 on macOS reaches task selection and then rejects Docker Desktop's no-network capability detection with the known network_mode='no-network' is not supported error. The policy was not weakened and the Docker socket was not mounted. This draft awaits the GitHub Linux exact-task Oracle before it is considered complete.

Repository-wide static formatting currently reports unrelated pre-existing formatting drift in public-reproduction and symbolic-coordination files outside this branch; no unrelated files are changed.

@yuelgrace1810-ops
yuelgrace1810-ops marked this pull request as ready for review August 6, 2026 23:07
@yuelgrace1810-ops
yuelgrace1810-ops marked this pull request as draft August 6, 2026 23:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31cbc21c32

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

morluto commented Aug 7, 2026

Copy link
Copy Markdown
Owner

@codex

@morluto
morluto marked this pull request as ready for review August 7, 2026 09:05

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 195d7f1bae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Grace Lee Rui Yue and others added 3 commits August 7, 2026 17:34
…ormat

Fix root causes flagged across conjecture-probe verifier PRs:

1. Mathematics, evidence, scope, and assurance are now computed
   independently of contract validity. A correct result no longer
   reports mathematics:0.0 when an unrelated envelope field is invalid.

2. All advertised scoreable assurances (UNVERIFIED, COMPUTED, CHECKED)
   receive credit, not just CHECKED.

3. False-certification detection reads the raw submission JSON before
   strict schema validation can discard a malformed VERIFIED claim.

4. The evidence media type is corrected from text/plain to
   application/json, and the required payload shape is published in the
   public contract so agents can discover the JSON schema.

5. Reverted unrelated whitespace churn across 26 symbolic-coordination
   verifier_support copies and formatter-only rewrites of two
   reliability verifier files that changed Harbor task digests without
   changing behavior.
@morluto
morluto force-pushed the agent/conjecture-perfect-cuboid-scope branch from 195d7f1 to 86085bf Compare August 7, 2026 09:38

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86085bfdae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…cksum

- Bound _raw_submission() with is_regular_bounded_file to prevent
  reading oversized submission files before strict validation
- Refresh Dockerfile checksum label to match updated verifier code
Co-authored-by: morluto <morluto@users.noreply.github.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28fbbceb66

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@morluto
morluto merged commit f141962 into main Aug 7, 2026
40 checks passed
@morluto
morluto deleted the agent/conjecture-perfect-cuboid-scope branch August 7, 2026 11:34
@morluto

morluto commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Review comment fixes

1. Decouple mathematics from strict envelope validation

Problem: When a submission reported the correct result but had an unrelated envelope error (bad scope, evidence descriptor, etc.), load_submission() returned None, causing the mathematics diagnostic to be lost — even though the mathematical content was correct.

Fix: In verifier.py, the mathematics diagnostic now falls back to raw_submission (parsed without strict schema validation) when the strict submission is None. When the strict submission is valid, it is identical to the raw one, so this is a pure superset of the previous behaviour. The diagnostic is preserved for debugging while the aggregate reward still fails closed on the envelope error.

Added test test_bad_scope_does_not_mask_mathematics verifying that a bad scope preserves mathematics == 1.0 while scope == 0.0 and aggregate_reward == 0.0.

2. Wire the new verifier leaf into the benchmark planner

Problem: task_host_validation(root, "conjecture-probes-v1", "perfect-cuboid-scope-audit") fell through to the dataset-wide validation, which only runs test_vizing_bounded_cartesian_products.py — not the task-specific test.

Fix: Added a conjecture-probes-v1 case in task_host_validation() in benchmarks/tooling/validation_plan.py that checks for a dedicated test file at benchmarks/validation/conjecture_probes_v1/test_{task}.py, following the same pattern already used for mathematical-benchmarks-v1.

Added test test_conjecture_probes_task_change_selects_leaf verifying that task-scoped changes now select the correct test file.

Dockerfile checksum

Updated jacobian.checksum label in the Dockerfile to match the new verifier.py sha256.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants