feat(search): news adapter (keyless Bing News RSS) + recency-aware fallback default#92
Merged
Conversation
…allback default The v0.23.0 bench's one failure was the `recent` question: DDG throttled, and the wikipedia fallback's undated/stale pages were culled by --since=180d, leaving 1/3 sources. Recency was the gap. - New `news` adapter: Bing News RSS (no key). Unwraps the apiclick redirect to the real publisher URL (rejecting non-http(s) targets), decodes entity-encoded description markup before stripping it, and prefixes each snippet with the article's YYYY-MM-DD pubDate so the planner/synthesizer can see recency. 403/429 classify as SearchRateLimitError. Hand-rolled bounded-regex parsing, same stance as the arXiv/DDG adapters; no new dependencies. - Recency-aware fallback default: with --since set (and no explicit --search-fallback/env), the default becomes news,wikipedia so a throttled primary degrades into dated, fresh sources. Explicit flag/env still wins; =none still disables. - Bench: recent now passes twice (6/3 sources 1.00 support, then 5/3 at 0.80 in the full board). Full scoreboard 6/6 — first perfect run; committed in bench/results/2026-06-12-v0.24.0-news-fallback.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The v0.23.0 validation bench scored 5/6. The one failure was the
recentgolden question: with DDG throttled, the wikipedia fallback returned undated/stale encyclopedia pages that the--since=180dfreshness filter culled — the run completed with 1/3 sources. Recency-sensitive questions are exactly the lane hosted competitors are strongest in, and deepdive had no keyless path to dated, fresh sources.What
--search=news— new keyless adapter over the Bing News RSS endpoint. Result links arrive wrapped in abing.com/news/apiclick.aspxredirect; the adapter unwraps theurlparam to the real publisher URL (rejecting non-http(s) targets), so deepdive fetches and cites publishers directly. Each snippet is prefixed with the article'sYYYY-MM-DDpubDate. 403/429 →SearchRateLimitError(benching + degradation visibility work as for other adapters). Hand-rolled bounded-regex RSS parsing, same stance as the arXiv/DDG adapters; zero new dependencies.--sinceis set and no explicit--search-fallback/env is given, the default fallback becomesnews,wikipediainstead ofwikipedia. Explicit flag/env still wins;=nonestill disables.Bench evidence (with a disclosure)
recentpasses twice with the change: single-question run 6/3 sources at 1.00 citation support ($0.090, 67s), then 5/3 at 0.80 in the full board. Full scoreboard is 6/6 — first perfect board (trajectory: 1/6 → 4/6 → 5/6 → 6/6). Committed asbench/results/2026-06-12-v0.24.0-news-fallback.md.Disclosure: a concurrent session was editing the planner prompts in the same checkout (now
feat/date-grounded-planner), so the benchdist/contained both changes. They are mechanically disjoint (fallback adapter selection vs. prompt wording) and the news fallback is the direct mechanism behindrecent's dated sources, but the 6/6 cannot be attributed to either change alone — the scoreboard file says so, and the board should be re-run at the next release tag.