synth prompt tuning PARKED: bench can't resolve prompt deltas while factual-lookup hedges nondeterministically (stock 1/3)

[askalf]

What happened

Attempted the roadmap's synthesize.ts prompt tuning (2026-06-12 evening) with full bench discipline. Three variants, four full boards, five spot-checks — and a control experiment that invalidated the whole comparison environment.

Variants tried (all on branch feat/synth-grounding, parked at 958be2c, NOT merged):

• v1: + verbatim-source-terms rule, + one-claim-per-cited-sentence rule, + "leave your own analytic sentences uncited", + Today's date line with "…not your training data" recency phrasing. Board: 4/6 — factual-lookup collapsed to a 97-word zero-citation answer; support down on 4 questions.
• v2: dropped the uncited-analysis rule and the training-data phrasing; kept date line + 2 grounding rules. Spot-checks passed; full board: 4/6 — factual-lookup zero-citation AGAIN, academic 0.25.
• v3: dropped the date line too (grounding rules only). Spots: 1 PASS, 1 zero-citation FAIL.

Control (the decisive bit): the UNMODIFIED prompt also produced the zero-citation hedge — 1 of 3 runs, same signature (≤97 words, no [N], hedged "current X" answer). Every variant's failures are within the noise of that environmental flake.

Conclusions

1. The zero-citation hedge on factual-lookup is environmental, not prompt-caused. It appeared only this evening (every morning/afternoon board cited at 0.86–1.00) and hits stock and tuned prompts alike at ~1/3 frequency. Most plausible class: upstream serving-side behavioral flux on claude-sonnet-4-6 (same family as the 2026-06-12 morning client-system-prompt drift, fixed in dario 4.8.66 — this one is subtler: structure obeyed, content hedged).
2. The bench cannot resolve prompt deltas while this flake is live. Any synth-prompt comparison needs a stock factual-lookup baseline of 3/3 first — treat that as the precondition for resuming this lane.
3. Two genuinely-rejected rules (real signal, not flake): "leave analytic sentences uncited" licenses fully uncited answers; date grounding IN THE SYNTH PROMPT correlates with hedge-mode even though the same line in the PLANNER (v0.25.0) measurably helps. Documented in src/synthesize.ts comments on the parked branch.
4. The two surviving rules (verbatim source terms, one claim per cited sentence) showed weak positive signal (technical-deep 0.89→1.00 twice, niche-ops 14/14 @ 1.00) but cannot be confirmed under the flake.

Resume protocol

1. Run --only=factual-lookup ×3 on stock master. All 3 must cite (support ≥ 0.5). If not, the environment is still polluted — do not tune.
2. Rebase feat/synth-grounding, re-run the full board, compare against bench/results/2026-06-12-v0.25.0-validation.md (the clean stock baseline: 5/6, support 0.83–1.00).
3. Boards on the parked branch: 2026-06-12-synth-grounding-v2-rejected.md shows what rejection looks like.

[askalf]

Re-ran the resume-protocol gate 2026-06-14 — root cause was misdiagnosed. It's the search layer, not an LLM flake.

Setup: deepdive v0.25.1 (synthesize.ts byte-identical to master — the only master commits past the 0.25.1 tag are doc-only), claude-sonnet-4-6 via the prod dario, default config. Ran resume-protocol step 1 (factual-lookup ×3 on stock).

Result: stock factual-lookup = 1/3 cite — but the 2 failures were not an LLM hedge

backend	run 1	run 2	run 3	verdict
DDG (default/stock)	PASS — 9 web sources, 22/23 (0.96)	FAIL — 0 citations, 70 words	FAIL — 0 citations, 129 words	1/3
searxng	PASS 1.00	PASS 0.81	PASS 1.00	3/3

The smoking gun

Both DDG failures logged the same line:

deepdive: primary search produced nothing — retrying the round via wikipedia

And their kept sources were 100% the wikipedia fallback — GPT-5.5, Large language model, FFmpeg, Anthropic, Mistral AI, Gemini, Easyship, … None of those articles contain Claude API rate-limit tables, so the synth model wrote "None of the 12 provided sources contain specific rate limit figures…" / "The provided sources do not answer this question."

That is correct grounded behavior: given sources that lack the answer, the model refused to fabricate citations. The "zero-citation hedge" is the right response to bad inputs — not a defect.

The PASS run (and all 3 searxng runs) pulled the real corpus — platform.claude.com/docs/.../rate-limits, tokencalculator.com/.../claude-api-rate-limits, etc. — and cited it at 0.81–1.00.

What this corrects in the original write-up

• Conclusion v0.2.0 — --deep loop, concurrent fetch, on-disk cache, --json output #1 ("environmental, not prompt-caused") — right direction, wrong class. Environmental ✅, but the mechanism is DuckDuckGo intermittently returning zero results for the box IP → wikipedia fallback → irrelevant sources, not "upstream serving-side behavioral flux on claude-sonnet-4-6." Same family as the v0.22.0 finding already in config.ts ("DDG rate-limiting the box's IP, 5/6 default-config questions died with zero sources"). The fallback was added so runs don't die at zero sources — but for a question Wikipedia can't answer, it converts a search failure into a confident-looking refusal that reads exactly like an LLM hedge. That's what fooled the 2026-06-12 diagnosis.
• The synth prompt was never hedging. It grounded correctly in every one of the 6 runs. The two surviving grounding rules and the two rejected rules can now be evaluated for real.
• Conclusion docs: rewrite README — sovereignty + cost-arbitrage angle #2 ("bench can't resolve prompt deltas while the flake is live") — still true, but the precondition was unachievable as written. "Stock factual-lookup 3/3 on DDG" depends on DDG not rate-limiting the box IP, which is outside our control. The right precondition is a reliable search backend.

Revised unblock path (this lane is no longer waiting on a phantom LLM flake)

1. Run the bench through searxng — DEEPDIVE_SEARCH=searxng DEEPDIVE_SEARXNG_URL=<box searxng>. Egresses via Proton, not the DDG-rate-limited box IP. Proven 3/3 above. This is the cleanest fix and it reopens the synth-tuning comparison immediately.
2. (Optional, separate) Stop the wikipedia fallback from masking search failures on non-recency factual questions — let a DDG empty surface as an honest 0 sources → completed FAIL instead of a misleading refusal. This alone prevents the misdiagnosis from recurring.
3. Then rebase feat/synth-grounding and compare against a searxng baseline (replacing the DDG-dependent 2026-06-12-v0.25.0-validation.md baseline).

Net: the synth-grounding lane can resume now under searxng — no need to wait out a claude-sonnet-4-6 flake that doesn't exist.

[askalf]

Resolved: ran the full board comparison on a stable (SearXNG) search layer → REJECT the grounding rules

With the bench now deterministic (#102), I rebased feat/synth-grounding onto master (2165c96 on top of the SearXNG bench c5160ab) and ran two full 6-question boards back-to-back through SearXNG, claude-sonnet-4-6, v0.25.1: the branch, then stock master.

Both scored 5/6 (factual-lookup wedged identically in both — fetch-stage infra issue, filed as #104, neutral here). The signal is the citation-support delta on the 5 measurable questions:

question	stock support	synth support	Δ	citations stock→synth
technical-deep	0.94 (34/36)	0.87 (39/45)	−0.07	36 → 45
academic	0.93 (14/15)	0.58 (14/24)	−0.35	15 → 24
comparison	0.92 (33/36)	0.83 (43/52)	−0.09	36 → 52
recent	1.00 (39/39)	0.92 (44/48)	−0.08	39 → 48
niche-ops	0.97 (32/33)	0.91 (21/23)	−0.06	33 → 23

Why reject

The two "surviving" rules (verbatim-source-terms + one-claim-per-cited-sentence) regress citation support on all 5 questions — consistent direction (a coin-flip lands 5/5 the same way ~3% of the time, so this isn't noise) and sometimes large.

The mechanism is in the citation counts: "one claim per cited sentence, don't stack cites" made the model cite more (technical 36→45, academic 15→24, comparison 36→52), but the marginal citations fail the lexical verifier. The academic case is the tell — synth added 9 citations, 0 of which verified (supported stayed at 14; total went 15→24), collapsing support to 0.58, a hair above the 0.50 floor.

That's the rules failing at their own stated purpose. The synthesize.ts comment justifies them as "reuse the source's key terms verbatim so a cited sentence is checkable against the cited text" — i.e., they exist to raise verifier support. Measured cleanly, they lower it. The earlier "weak positive signal" (technical-deep 0.89→1.00, niche-ops 14/14) was DuckDuckGo-flake noise; on a stable search layer it inverts.

Caveat (honest): support is a lexical-recall proxy, not prose quality, and this is n=1 board per side. It's possible the synth answers read better (more granular, more verbatim) and the verifier penalizes legitimate fine-grained citing. But by the project's own grounding metric the rules make citations less trustworthy — the opposite of intent — so they don't earn a merge as written.

Disposition

• feat/synth-grounding stays parked / rejected (rebased onto master so it's revivable, but not merged). The bench-rejected-variants comment in synthesize.ts is now joined by these two rules as measured-negative.
• The clean stock board is promoted to a committed SearXNG baseline on master (replaces the DDG-confounded 2026-06-12-v0.25.0-validation.md).
• factual-lookup's fetch wedge is factual-lookup intermittently fails: upstream synthesis-call stall (~20%, fast-failed by idle deadline) #104.

If the grounding idea is worth another pass, the next variant should target the verifier directly (raise supported-ratio, not citation count) and be judged on a 3×-per-side board, not 1×.

stock master board (full)

| question | run completed | enough sources | citation support | answer length | topical keywords | under cost ceiling | verdict | time |
|---|---|---|---|---|---|---|---|---|
| factual-lookup | ❌ exit 1 | ❌ 0/4 | ❌ no citations | ❌ 0 words | ❌ missing: tier, requests, tokens | ✅ $0.000 / $2.00 | FAIL | 418s |
| technical-deep | ✅ exit 0 | ✅ 12/5 | ✅ 34/36 (0.94) | ✅ 1023 words | ✅ all present | ✅ $0.302 / $2.00 | PASS | 205s |
| academic | ✅ exit 0 | ✅ 11/5 | ✅ 14/15 (0.93) | ✅ 598 words | ✅ all present | ✅ $0.110 / $2.00 | PASS | 127s |
| comparison | ✅ exit 0 | ✅ 11/4 | ✅ 33/36 (0.92) | ✅ 999 words | ✅ all present | ✅ $0.133 / $2.00 | PASS | 98s |
| recent | ✅ exit 0 | ✅ 9/3 | ✅ 39/39 (1.00) | ✅ 1044 words | ✅ all present | ✅ $0.141 / $2.00 | PASS | 98s |
| niche-ops | ✅ exit 0 | ✅ 10/3 | ✅ 32/33 (0.97) | ✅ 708 words | ✅ all present | ✅ $0.115 / $2.00 | PASS | 100s |
5/6 passed.

feat/synth-grounding board (full)

| question | run completed | enough sources | citation support | answer length | topical keywords | under cost ceiling | verdict | time |
|---|---|---|---|---|---|---|---|---|
| factual-lookup | ❌ exit 1 | ❌ 0/4 | ❌ no citations | ❌ 0 words | ❌ missing: tier, requests, tokens | ✅ $0.000 / $2.00 | FAIL | 405s |
| technical-deep | ✅ exit 0 | ✅ 12/5 | ✅ 39/45 (0.87) | ✅ 1292 words | ✅ all present | ✅ $0.292 / $2.00 | PASS | 223s |
| academic | ✅ exit 0 | ✅ 9/5 | ✅ 14/24 (0.58) | ✅ 719 words | ✅ all present | ✅ $0.085 / $2.00 | PASS | 120s |
| comparison | ✅ exit 0 | ✅ 11/4 | ✅ 43/52 (0.83) | ✅ 1080 words | ✅ all present | ✅ $0.149 / $2.00 | PASS | 121s |
| recent | ✅ exit 0 | ✅ 10/3 | ✅ 44/48 (0.92) | ✅ 937 words | ✅ all present | ✅ $0.214 / $2.00 | PASS | 168s |
| niche-ops | ✅ exit 0 | ✅ 7/3 | ✅ 21/23 (0.91) | ✅ 578 words | ✅ all present | ✅ $0.082 / $2.00 | PASS | 93s |
5/6 passed.

[askalf]

Resolved — the bench can now resolve prompt deltas (searxng, #102) and the grounding rules were measured and rejected (regress citation support on all 5 measurable questions). Full verdict + boards above (issue#97#issuecomment-4708848562). feat/synth-grounding stays parked/revivable; clean searxng baseline on master (#105). Closing.