fix: synthesizer closes broad-scope subgoals too aggressively (terminates overnight runs early)

[bradtaylorsf]

Summary

The synthesizer is closing subgoals as confirmed too aggressively on broad-scope goals. Project 2025 overnight test (2026-05-07): synthesizer marked all 4 subgoals confirmed after 45 tasks, terminating what was supposed to be an all-night run after only 13 minutes.

This is the calibration follow-up to #119: subgoal-done tracking is wired correctly, but the prompt threshold for declaring confirmed is too generous when the goal is scope_class: broad or comprehensive.

Reproducer

uv run research start --skip-intake --local \
  --goal "Project 2025 implementation tracker: identify which specific policy proposals from the Heritage Foundation's Project 2025 document have been adopted, attempted, withdrawn, or remain pending under the current Trump administration. Organize by federal department (DOJ, DOI, EPA, DHS, State, etc.). For each tracked proposal, surface news coverage, public statements, and any pushback or legal challenges. Prioritize primary sources and date-stamp every finding." \
  --max-tasks 1000 --time-cap 10

Observed:

• Initial plan: scope_class=broad, 4 subgoals, 16 search tasks ✓
• 45 tasks executed, 1 drain_replan fired ✓
• Synthesis pass: closed=[1,2,3,4] — gemma confidently declared every subgoal done after seeing partial evidence on each.
• Loop terminated. Run ended in ~13 minutes.

The closed subgoals were:

1. "Identify core policy pillars and specific proposals from the Project 2025 document"
2. "Map identified policies to their respective federal departments and check for implementation status"
3. "Collect evidence of legal challenges, news coverage, and official public statements"
4. (a fourth one in the same vein)

For a 920-page policy document with proposals across 20+ federal departments, none of these should be closeable after 45 tasks of crawling.

Root cause

prompts/synthesizer.md (the v5 trailing-JSON contract from #119) instructs:

• confirmed — the findings affirmatively answer the subgoal. Closes it.
• inconclusive — findings are insufficient, contradictory, or absent.

The prompt does NOT differentiate by scope. Gemma — which favors decisiveness — defaults to confirmed whenever it can write any affirmative answer, regardless of completeness. For a narrow scope this is fine; for broad/comprehensive it terminates the run prematurely.

Acceptance Criteria

• prompts/synthesizer.md adds a scope-aware closure rule: when the plan's scope_class is broad or comprehensive, default subgoal status to inconclusive unless ALL of:
  • At least 5 distinct sources cited per subgoal in the corpus collected so far, AND
  • Synthesizer can articulate at least 2 specific examples per subgoal that resolve the question, AND
  • Findings span at least 3 distinct domains/entities the subgoal references (e.g., for "policies across federal departments", findings must touch ≥3 departments)
• The synthesis context includes the plan's scope_class so the synthesizer knows which threshold to apply.
• Add a unit test: feed a synthesized 45-task broad-scope corpus into the synthesizer and assert subgoal_status map values are predominantly inconclusive, not confirmed.
• After fix: re-run the Project 2025 goal locally; expect ≥1 subgoal to remain inconclusive after the first synthesis, driving drain-replan to fire ≥3 times instead of stopping after 1.

Files

• src/research_agent/prompts/synthesizer.md (the closure-status rules)
• src/research_agent/orchestrator/synth.py (pass scope_class into the synthesis context)
• tests/test_orchestrator_synth.py (new test for the scope-aware threshold)

Why this matters

Without this fix, every overnight run on a broad goal terminates early — exactly the problem #117/#118/#119 were supposed to solve. The architecture works; the prompt-level calibration is the last gate to "actually runs all night."

[bradtaylorsf]

AI Implementation Notes

Assumptions

• Assumed null scope_class should be treated like narrow (decisive behavior), since the issue did not specify how to handle plans missing scope_class — the prompt states "When absent, treat the goal as narrow."
• Assumed the three gates from the acceptance criteria should be evaluated by the LLM via prompt instructions rather than enforced programmatically post-hoc — i.e., the synthesizer is trusted to apply the rule, not gated by code that overrides its output.
• Assumed "5 distinct sources cited per subgoal" means 5 distinct source URLs in the corpus that bear on the subgoal, not 5 sources in the final report's source list.
• Assumed scope_class should be serialized as a string (str(plan.scope_class)) rather than relying on the enum's JSON encoding, since the prompt expects a literal string value to compare against.

Decisions

• Chose to expose scope_class as an explicit top-level field in the synthesis JSON context rather than embedding it in the goal text, so the prompt can reference it by name and the unit test can assert on it directly.
• Chose to bump the prompt version to v6 (rather than amending v5 in place) to make the calibration change traceable, consistent with the v5 contract introduced in Subgoal-done tracking via synthesis pass #119.
• Chose not to add a critic-side enforcement layer that would programmatically reopen broad subgoals failing the three gates — relying on prompt-level guidance plus the existing critic catch path mentioned in the prompt. A code-level gate would be more deterministic but adds coupling.
• Chose to write the test against a stubbed router with a hardcoded synthesizer response (3 of 4 inconclusive) rather than running gemma against a real 45-task corpus — the test verifies the wiring (context includes scope_class, inconclusive statuses persist), not the model's compliance with the prompt rule.
• Did not add a test that asserts gemma-the-actual-model honors the scope-aware rule on a real broad corpus, since that requires a live model call and is the manual re-run validation step from the issue.

Items to Validate

• Re-run the Project 2025 reproducer locally and confirm ≥1 subgoal stays inconclusive after the first synthesis and drain-replan fires ≥3 times — this is the only end-to-end validation that the prompt change actually changes gemma's behavior. The unit tests cover wiring but not model compliance.
• Verify the three thresholds (5 sources / 2 examples / 3 domains) are calibrated correctly for your typical broad goal — they are quoted verbatim from the issue but you may want to tune them after observing a few overnight runs.
• Confirm the prompt's behavior on scope_class: "comprehensive" is acceptable — it inherits the same gates as broad, which the issue implies but doesn't explicitly distinguish.
• Check that the v6 prompt's added length (~36 lines) does not push the synthesizer context past any token budget on long-running jobs with large finding corpora.
• The test asserts confirmed on subgoal 1 still closes it under broad scope — verify this is the intended semantics (the rule defaults to inconclusive, but explicit confirmed from the model still wins) rather than forcing all broad subgoals to inconclusive regardless of model output.

---

Posted by alpha-loop for user validation.

[bradtaylorsf]

Automated implementation complete.

PR: #168
Tests: PASSING
Review: Attached to PR body.

---

Processed by alpha-loop in 506s