ABT-1099: Keep unexploited pre-trial findings out of trial labels

AGENTS.md

@@ -23,6 +23,29 @@ Python `3.13` is required for `oddish` and `backend`. Node.js `20+` and `pnpm` a
   Modal app wiring, managed worker spawning, GitHub/webhook integrations, and
   cloud-only policy) belong in `backend/`.
 
+## Pull Request and Deployment Process
+
+- Keep each PR below 500 lines of application-code changes. If a coherent
+  change cannot fit, split it into independently reviewable PRs.
+- Write a human-readable PR title and description: explain the problem, the
+  behavior change, the important design choices, and how it was verified.
+  Remove filler, canned prose, and generated-sounding language before handoff.
+- Require a successful preview deployment before declaring a PR ready. Verify
+  the frontend, backend, and database/migrations when the change affects them.
+- Deploy to staging and closely test the affected workflows there before any
+  production promotion.
+- Never merge or promote a PR unless the user explicitly requests it. When
+  production promotion is requested, open a PR from `staging` to `main` and
+  comment `/promote` on that PR.
+- Own production regressions caused by your change: fix them and write a
+  postmortem. If the required process was skipped, obtain a reviewer before
+  further production changes.
+- Close abandoned or completed PRs when their lifecycle is actually finished;
+  do not close a fresh PR that is being handed off for review.
+- Treat this repository and its services as shared infrastructure. Keep scope
+  tight, preserve unrelated work, test carefully, and leave the codebase in a
+  state you are prepared to support.
+
 ## Repository Layout
 
 ```text

frontend/src/components/qa-report/tokens.ts

@@ -1,9 +1,4 @@
-import {
-  CircleCheck,
-  ShieldAlert,
-  TriangleAlert,
-  Unplug,
-} from "lucide-react";
+import { CircleCheck, ShieldAlert, TriangleAlert, Unplug } from "lucide-react";
 
 // Colour carries good/bad, not success/failure: green = valid signal,
 // amber = task needs fixing, red = false positive, orange = infrastructure.
@@ -61,7 +56,7 @@ export const TIER_META: Record<string, { label: string; labelEffect: string }> =
     must_fix: {
       label: "MUST FIX",
       labelEffect:
-        "Blocks GOOD FAILURE — a failed run with a must_fix item is BAD FAILURE.",
+        "Changes the trial label only when it affected the result. The task verdict still evaluates it.",
     },
     should_fix: {
       label: "SHOULD FIX",
@@ -74,8 +69,8 @@ export const TIER_META: Record<string, { label: string; labelEffect: string }> =
   };
 
 // Severity is deliberately not hue-coded: only must_fix gets colour, because
-// only must_fix can change the label, so a severity badge can never be
-// mistaken for a verdict.
+// only must_fix can change a label or task verdict, so a severity badge can
+// never be mistaken for a verdict.
 export const TIER_BADGE: Record<string, string> = {
   must_fix: "bg-destructive text-destructive-foreground",
   should_fix: "border-foreground/25 bg-foreground/10 text-foreground border",

oddish/src/oddish/analyze/classify_prompt.txt

@@ -8,12 +8,12 @@ You audit benchmark tasks. An AI agent attempted a coding task in a sandbox. A h
 
 You output one `classification` and one `subtype`.
 
-Four labels pair a judgment with the grade. The second word is the grade: SUCCESS or FAILURE. The first word is your judgment of the task. GOOD means that the task worked as intended. BAD means that a task problem marks this run: the problem changed the outcome, or a `must_fix` defect exists (rule 3). The fifth label, HARNESS_ERROR, gives no judgment of the task. It means that the infrastructure broke the run. The verifier result decides which labels are allowed. The exact mapping is the "Allowed labels for this result" list below.
+Four labels pair a judgment with the grade. The second word is the grade: SUCCESS or FAILURE. The first word tells whether this grade is valid. GOOD means that the trial produced a valid signal. BAD means that a task problem caused or invalidated this result. The fifth label, HARNESS_ERROR, gives no judgment of the task. It means that the infrastructure broke the run. The verifier result decides which labels are allowed. The exact mapping is the "Allowed labels for this result" list below.
 
 * GOOD_SUCCESS — passed by really solving it
 * BAD_SUCCESS — passed without solving it (reward hacking, or tests too weak to notice)
 * GOOD_FAILURE — failed a fair task; the agent's own work was not good enough
-* BAD_FAILURE — failed on a task with a `must_fix` defect: a correct solution would also have failed, the task leaked hidden files (used or not), or another `must_fix` defect exists (rule 3)
+* BAD_FAILURE — a task defect caused or invalidated the failure; a correct solution would also have failed
 * HARNESS_ERROR — infrastructure broke the run; the grade means nothing
 
 The subtype is the specific cause, from the SUBTYPES list below.
@@ -63,31 +63,31 @@ When the component map is present, use it to find the correct part of the trajec
 
 1. The grade is final. You explain it. You never change it.
 2. The agent is supposed to see only: instruction.md, the codebase in the container, standard tools, and what it learns when it runs things. It is not supposed to see tests/, solution/, or the PR the task was built from. Do not assume that the task obeys this rule. Check the Dockerfile, the container setup, and the trajectory to find what was really visible. If hidden files were reachable, the task is broken (rule 6 says how to handle it). Two rules follow from the intended setup. First, the instruction must state each behavior that the tests check, or the agent must be able to infer it from the code. Inference is fair work, because tasks leave some facts to infer by design. Second, the agent is never at fault when it does not use the reference solution.
-3. Blame the task only for a defect that you can cite (file and line). A `must_fix` action item blocks GOOD_FAILURE. A `must_fix` item says that the defect can decide trials. GOOD_FAILURE says that the task was fair. One report cannot say both. When the report of a failed run has a `must_fix` item, the label is BAD_FAILURE, and `root_cause` names that defect. A defect that cannot decide trials gets the tier `should_fix` or `optional`, and it does not change the label. This rule never changes HARNESS_ERROR: when the infrastructure broke the run, the label stays HARNESS_ERROR, and the override in the procedure sets its subtype.
+3. Blame the task only for a defect that you can cite (file and line). A `must_fix` item says that a defect can decide trials. It does not prove that the defect decided this trial. A pre-trial `must_fix` item does not change the trial label by itself. Use BAD_FAILURE only when evidence shows that the defect caused or invalidated this grade. When the agent failed for an independent reason, use GOOD_FAILURE. Record the defect in `action_items` or `exploitation`; the task verdict evaluates latent defects separately. A defect that cannot decide trials gets the tier `should_fix` or `optional`.
 4. Difficulty is not a defect. Exploration and investigation are part of the job. If a skilled engineer could solve the task from the instruction and the code, the task is fair at each level of difficulty.
 5. Do not start from a default answer. Most agents fail, and many tasks have defects. Both facts are true in general, but general facts are not evidence about this run. Use only the evidence from this run.
-6. When the agent reads tests/ or solution/, that is cheating by the agent AND a leak in the task. The same rule applies to all access that the task did not intend: root or sudo, the Docker socket, host mounts, verifier or log directories at runtime, secrets in environment variables, sight of the grader at runtime, or the original fix from the internet. Procedure step 3 gives the label. Record the hole as a `must_fix` item in `action_items` in every case. There are two exceptions. An oracle baseline reads solution/ by design. Honest reconstruction from allowed files is fair.
+6. When the agent reads tests/ or solution/, that is cheating by the agent AND a leak in the task. The same rule applies to all access that the task did not intend: root or sudo, the Docker socket, host mounts, verifier or log directories at runtime, secrets in environment variables, sight of the grader at runtime, or the original fix from the internet. Procedure step 3 gives the label. Record a new hole as a `must_fix` item in `action_items`. Assess an existing pre-trial item in `exploitation` instead. There are two exceptions. An oracle baseline reads solution/ by design. Honest reconstruction from allowed files is fair.
 7. Judge actions and outputs, never the agent's words. Agents often believe they succeeded. Self-reported success is worthless. And another model solving this task proves nothing — models reproduce known fixes from memory.
 
 ---
 
 ## PROCEDURE
 
-Read the inputs first. Then do the checks, in order. The first decisive finding sets the label. Finish the remaining checks anyway. Put each task problem you find in `action_items`, even when it did not cause this grade. There is one override. If a task hole exists (rule 6) and the label is HARNESS_ERROR, set the subtype to `hidden_file_leak`. This rule applies when the agent used the hole and when it did not. State the infrastructure failure in `root_cause`. This keeps the leak visible to the verdict. The BAD labels keep their own subtypes, because their cheat subtypes already show the task problem.
+Read the inputs first. Then do the checks, in order. The first decisive finding sets the label. Finish the remaining checks anyway. Put each task problem you find in `action_items`, even when it did not cause this grade. If hidden-file exposure invalidated the grade, use HARNESS_ERROR `hidden_file_leak`. A latent hole that did not affect the run does not change its label. The task verdict still reads the pre-trial finding.
 
 1. **Did the infrastructure work?** Agent never started, empty trajectory, edits lost before grading, killed in the middle of productive work, grading or parse error -> HARNESS_ERROR.
 2. **Was the task already broken before the agent started?** Reference solution fails its own tests, tests already failing before any edit, required files or fixtures missing from the container -> BAD_FAILURE `broken_baseline`.
 3. **Did the agent cheat?** Graded pass -> BAD_SUCCESS. Partial reward -> BAD_SUCCESS if the cheat produced a passing test. If it did not, treat the partial like a graded fail. Graded fail -> the label depends on what the cheat used:
    * The cheat used a task hole (rule 6: read tests/ or solution/, escalated privileges, watched the grader, fetched the fix) -> BAD_FAILURE `hidden_file_leak`. The task leaked AND the agent cheated. The BAD label keeps the leak visible in the verdict. Say in `root_cause` that the agent used the leak and still failed.
    * The cheat used no task hole (edited or weakened tests that are a visible part of the codebase, hardcoded expected values, special-cased test inputs) -> GOOD_FAILURE `reward_hacking`. This bullet never covers the hidden tests/. Access to those files is the leak, and then the first bullet applies.
-   * There is a different case: hidden files were readable, but the trajectory shows that the agent never read or used them. That is not cheating by the agent, but the hole is still a `must_fix` defect. On a graded pass, continue with step 7 — a pass never takes HARNESS_ERROR. On a graded fail, the label is BAD_FAILURE `hidden_file_leak`, because rule 3 blocks GOOD_FAILURE. There is one exception on a fail: when the exposure alone makes the grade meaningless, use HARNESS_ERROR `hidden_file_leak`. On a partial reward, record the same `must_fix` item, and let the partial rules in step 6 and LABELS pick the label — rule 3 still blocks GOOD_FAILURE. Do the remaining checks anyway, and name the agent's own wrong step in `root_cause` next to the leak.
-   When a task hole exists, used or not, record it as a `must_fix` item in `action_items`. A cheat that used no task hole produces no leak item.
+   * Hidden files can be readable even when the agent never reads or uses them. This is not cheating by the agent. Record `exploited=false` and `causal=false`, then continue. The latent hole does not change the trial label. On a fail caused by the agent's work, use GOOD_FAILURE. On a pass, continue with step 7. If the exposure alone makes the grade meaningless, use HARNESS_ERROR `hidden_file_leak`.
+   When a task hole exists, record it as a `must_fix` item in `action_items` if the pre-trial audit did not find it. A cheat that used no task hole produces no leak item.
 4. **Could any fair solution pass?** For everything the tests require — names, signatures, argument counts, error strings, output formats, behaviors — is it stated or inferable? If not, no honest attempt could pass -> BAD_FAILURE (`unstated_interface`, `exact_output_matching`, `out_of_scope_tests`, `underspecified_instruction`, `ambiguous_requirements`, or `contradictory_instruction`).
 5. **Did the environment or flaky tests decide it?** Image broken before the agent started, network failures, out of memory, disk full -> BAD_FAILURE `environment_defect`. Result depends on timing, ordering, randomness, locale, or live network -> BAD_FAILURE `nondeterministic_tests`. The agent broke the environment itself -> GOOD_FAILURE `self_inflicted_breakage`.
 6. **Would the agent's solution pass fair tests** — tests that check exactly what the instruction demanded, nothing more? Decide per failing test.
    * Yes — it fails only on things the instruction never mentioned -> BAD_FAILURE.
    * No — it does not do what the instruction asked -> GOOD_FAILURE. Go through the trajectory (explore -> locate -> diagnose -> implement -> verify -> stop) and name the first wrong step. Pick the subtype that matches that step. The wrong step is usually early, even when the visible failure is late.
-   * Both true — the agent failed on its own AND the task has a defect? If the agent's work would have passed fair tests -> BAD_FAILURE. If not -> put the defect in `action_items`, and rule 3 sets the label: a `must_fix` defect makes this BAD_FAILURE, a lower tier leaves it GOOD_FAILURE.
+   * Both true — the agent failed on its own AND the task has a defect? If the agent's work would have passed fair tests -> BAD_FAILURE. If not -> GOOD_FAILURE. Put the independent defect in `action_items` or assess the pre-trial item in `exploitation`.
    * Cannot tell from the artifacts? Do not guess. Name the missing artifact in `evidence`, label on the strongest evidence you have, and say in `root_cause` that confidence is low.
    * Partial reward: apply these questions to the failing tests and step 7 to the passing ones, then pick the label for the cause of the missing reward. If the agent did everything the instruction demanded and the maximum possible score is below 1.0 -> GOOD_SUCCESS (the LABELS rule for partials).
 7. **For passes, full or partial: was the pass earned?** Would a trivial or wrong solution also pass? Was the repo already fixed? -> BAD_SUCCESS (`permissive_tests`, `task_pre_solved`). Is the fix written in the instruction text? -> BAD_SUCCESS `solution_leaked_in_instruction`. A nop baseline that passes is always BAD_SUCCESS. An oracle baseline that passes is GOOD_SUCCESS.
@@ -143,7 +143,7 @@ The output-mismatch case is the most common one. Ask one question: did the instr
 
 ### BAD_FAILURE — task's fault, needs fixing
 
-(Rule 3 sets the label: a `must_fix` defect makes a failed run BAD_FAILURE, whether or not the defect decided this run. Pick the subtype that matches the `must_fix` defect named in `root_cause`.)
+(Every subtype requires the task defect to have caused or invalidated this failure.)
 
 * `unstated_interface` — tests require an exact name, signature, or argument count that exists only in the reference solution
 * `exact_output_matching` — tests compare text, formatting, or hardcoded values byte-for-byte where the instruction specified none of it

oddish/tests/analyze/test_classifier_inputs.py

@@ -23,6 +23,22 @@ def test_placeholders_render_none_when_absent():
     assert "(none)" in prompt
 
 
+def test_pre_trial_must_fix_requires_causality_to_change_trial_label():
+    prompt = build_classify_prompt(
+        result_str="fail",
+        task_dir="/task",
+        trial_dir="/trial",
+        trial_agent_context="",
+        pre_trial_context="one must_fix finding",
+    )
+
+    assert "does not change the trial label by itself" in prompt
+    assert (
+        "When the agent failed for an independent reason, use GOOD_FAILURE" in prompt
+    )
+    assert "the task verdict evaluates latent defects separately" in prompt
+
+
 def test_write_qa_context_writes_trajectory_components(tmp_path):
     components = [
         {"step_ids": [0, 1], "trajectory_component": "debugging", "summary": "s",