[Benchmark task design] Do not encode unknown conclusions in the public answer language

[morluto]

Summary

A contract-first audit found multiple executable tasks whose agent-visible answer language directly supplies the formula, truth value, extremum, implication, or theorem conclusion that the task asks the agent to derive.

The concrete leak appears as JSON Schema const or singleton-enum fields under result, but the root defect is not “misuse of one schema keyword.” The task has modelled an unknown mathematical conclusion as a public protocol constant and often asks the agent to repeat it alongside the actual certificate.

A public task contract should define:

what mathematical object the agent may submit

not:

the unique successful mathematical answer the agent is meant to discover

This is a task-design and benchmark-validity issue. Removing const while retaining a redundant conclusion field may still leave the wrong answer object.

Confirmed examples

convergence-mode-separation

The task asks the agent to establish the typewriter-sequence behavior while the public answer language fixes values such as:

event_mass_formula = 1/2^k
limit = ZERO
relationship = IN_PROBABILITY_NOT_IMPLY_ALMOST_SURE

triplewise-empty-extremal-audit

The answer language supplies the corrected maximum formula and incidence-budget conclusions the agent is asked to derive.

positive-lower-density-separation

The public result fixes the closed form and theorem-level truth values for positivity and nonexistence of natural density.

squarefree-class-independence-audit

The result fixes classification formulas, the final consequence, and the maximum number of squares ruled out.

exponential-moment-rationality

The terminal rationality conclusion is supplied as one fixed accepted label.

bounded-variation-uniform-limit

The public answer language publishes the limiting function, variation behavior, convergence fact, and implication.

Repository-wide search finds many more conclusion-like constants. Some constants are legitimate public branch tags or fixed input echoes; the task contract currently does not distinguish those roles.

Required field classification

Every public result field must be classified as one of:

Frozen problem data

A value known before solving. It belongs in input.json or the problem statement, not in the answer merely to be echoed.

Public structural discriminator

A finite tag that selects a task-local result variant, such as proof_mode = LDL. A constant inside one branch may be legitimate because it identifies shape rather than revealing whether the theorem is true.

Authoritative certificate data

The finite object the agent must construct: points, coefficients, proof steps, table entries, factorization data, search coverage, or another bounded witness. The public contract describes its type and bounds without fixing the successful value.

Derived conclusion or summary

A truth value, theorem label, count, formula, optimum, or implication determined by the certificate/public input. Normally remove it from the submitted answer and derive it in evaluator output.

Explicit classification objective

A conclusion may remain agent-authored only when choosing among multiple meaningful public possibilities is itself the task. The schema may expose the finite type, but not the correct member.

Why removing const alone is insufficient

Changing:

{"const": true}

to:

{"type": "boolean"}

prevents direct leakage but may still create impossible combinations between a certificate and a separately authored truth label. Where the evaluator can derive the boolean, remove it. Where classification is the objective, structurally bind it to the result variant or certificate.

Likewise, moving the same answer literal into generated instruction prose does not repair the task.

Required correction

1. Inventory every const and singleton enum beneath the submitted result.
2. Classify it by the roles above.
3. Move frozen problem data to the public input/problem statement.
4. Keep only legitimate structural discriminators as constants.
5. Remove derived conclusions/summaries from the answer where the certificate determines them.
6. For explicit classification tasks, expose multiple meaningful choices and replay the evidence independently.
7. Update instruction, public declaration, generated schema, hidden solution, evaluator, and metamorphic tests together.

Authoring invariant

Repository validation should flag:

• fixed booleans in truth/existence/optimality/conclusion fields;
• fixed formulas or exact numeric answers;
• fixed theorem/consequence/relationship labels; and
• singleton enums whose only purpose is to tell the agent which answer to return.

A retained constant requires a narrow reviewed classification such as PUBLIC_VARIANT_TAG or FROZEN_INPUT_ECHO, with a reason.

Acceptance criteria

• An agent cannot recover the scored mathematical conclusion from the public answer schema alone.
• Confirmed tasks no longer publish their unknown formulas, extrema, truth values, or theorem consequences.
• Derived labels are removed and computed from authoritative certificate data where possible.
• Explicit classification tasks expose a real finite choice rather than one accepted singleton.
• Structural discriminators remain allowed without being confused with answer leakage.
• Positive tests prove the certificate is accepted; negative tests alter certificate semantics while keeping metadata/display choices fixed and require rejection.
• The correction does not replace leaked constants with equally leaked examples, filenames, or prose.

Related

• benchmark: public submission_schema.json leaks the answer (rank-one-spectral-limit-certificate) #867 — earlier local answer-constant leak.
• [Benchmark task design] Represent mathematical expressions as values, not privileged strings #1584 — mathematical formulas as structured/derived values.
• [Benchmark task design] Replace prose-as-answer with typed or derived conclusions #1588 — typed or derived conclusions instead of prose.
• [Benchmark task design] Model proof modes and success/failure cases as closed result variants #1618 — closed task-local result variants.
• [Benchmark task design] Bound the public answer language before mathematical replay #1619 — authoritative values versus derived summaries/counts.

[morluto]

Additional answer-leak form: an array can force the complete answer without using const or a singleton enum.

unit-fraction-classification-repair/environment/submission_schema.json requires:

"defects": {
  "items": {"enum": [/* exactly five named defects */]},
  "minItems": 5,
  "maxItems": 5,
  "uniqueItems": true
},
"nonmember_counterexamples": {
  "items": {"enum": [3, 5, 8]},
  "minItems": 3,
  "maxItems": 3,
  "uniqueItems": true
}

Those constraints force the array to contain every enumerated item, so the public schema supplies both the full defect diagnosis and all three requested counterexamples. The instruction asks the agent to diagnose each defect and recompute the classification.

The repository ratchet should therefore flag not only const and singleton enums, but also exhaustive enum + exact cardinality + uniqueness combinations under result, and similar contains/cardinality constructions that leave only one accepted mathematical set. Legitimate bounded vocabularies remain fine when the schema does not force which members are the answer.