Skip to content

fix(importers): rank pre-filter candidates by relevance strength before the LLM-scoring cap - #149

Open
diegokolling wants to merge 1 commit into
NousResearch:mainfrom
diegokolling:fix/rank-prefilter-before-cap
Open

fix(importers): rank pre-filter candidates by relevance strength before the LLM-scoring cap#149
diegokolling wants to merge 1 commit into
NousResearch:mainfrom
diegokolling:fix/rank-prefilter-before-cap

Conversation

@diegokolling

Copy link
Copy Markdown

Bug

RelevanceFilter.filter_and_score caps heuristic candidates at max_examples * 3 without ranking them first. The pre-filter (_is_relevant_to_skill) returns candidates in the order the importers produced them (chronological), so the LLM relevance judge spends its entire scoring budget on the chronologically-first N messages — stronger matches beyond the cap are simply never scored.

Empirical impact (real session store, 2026-07-09)

Mining evals for a skill against a Hermes state.db with 911 user→assistant pairs:

  • heuristic pre-filter matched 321 candidates
  • cap truncated to the first 150 (chronological, not strongest)
  • only 5 survived LLM relevance scoring → 2 train / 1 val / 2 holdout

A 2-example holdout makes the final baseline-vs-evolved verdict pure noise (±0.01 deltas decide "improved or not"), which silently defeats the purpose of --eval-source sessiondb.

Fix

  • _is_relevant_to_skill becomes a thin boolean wrapper over a new _relevance_score(text, skill_name, skill_text) -> int (strength heuristic: +10 exact skill-name phrase, +3 per skill-name word >3 chars, +N keyword-overlap past the existing 2-keyword floor). Boolean semantics (score > 0) are byte-identical to the old rule, so no caller/test behavior changes.
  • filter_and_score sorts candidates by that score descending before applying the cap, so the LLM budget goes to the strongest matches.

Verification

  • pytest tests/core/test_external_importers.py → 116 passed (no test changes needed; boolean contract preserved).
  • Re-ran the same real-store probe after the fix: the top-150 sent to the LLM now span relevance scores 17..3 instead of arbitrary chronological picks.

No behavior change for stores with fewer candidates than the cap (sorting a short list is a no-op for selection).

Attributed to: Diego + Hermes

…re the LLM-scoring cap

filter_and_score capped heuristic candidates at max_examples*3 WITHOUT
ranking them, so the LLM relevance judge spent its whole budget on the
chronologically-first N messages while stronger matches beyond the cap
were never scored. Observed on a real session store: 321 heuristic hits
-> capped to the first 150 -> only 5 survived LLM scoring -> a 2-example
holdout whose verdicts are pure noise.

_is_relevant_to_skill is now a thin boolean wrapper over a new
_relevance_score (int strength: +10 exact skill-name phrase, +3 per
skill-name word >3 chars, +N keyword overlap past the existing 2-keyword
floor), and filter_and_score sorts candidates by that score descending
before applying the cap. Boolean semantics (score > 0) are unchanged, so
existing callers/tests are unaffected.

Attributed-to: Diego + Hermes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant