fix(importers): rank pre-filter candidates by relevance strength before the LLM-scoring cap - #149
Open
diegokolling wants to merge 1 commit into
Open
Conversation
…re the LLM-scoring cap filter_and_score capped heuristic candidates at max_examples*3 WITHOUT ranking them, so the LLM relevance judge spent its whole budget on the chronologically-first N messages while stronger matches beyond the cap were never scored. Observed on a real session store: 321 heuristic hits -> capped to the first 150 -> only 5 survived LLM scoring -> a 2-example holdout whose verdicts are pure noise. _is_relevant_to_skill is now a thin boolean wrapper over a new _relevance_score (int strength: +10 exact skill-name phrase, +3 per skill-name word >3 chars, +N keyword overlap past the existing 2-keyword floor), and filter_and_score sorts candidates by that score descending before applying the cap. Boolean semantics (score > 0) are unchanged, so existing callers/tests are unaffected. Attributed-to: Diego + Hermes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
RelevanceFilter.filter_and_scorecaps heuristic candidates atmax_examples * 3without ranking them first. The pre-filter (_is_relevant_to_skill) returns candidates in the order the importers produced them (chronological), so the LLM relevance judge spends its entire scoring budget on the chronologically-first N messages — stronger matches beyond the cap are simply never scored.Empirical impact (real session store, 2026-07-09)
Mining evals for a skill against a Hermes
state.dbwith 911 user→assistant pairs:A 2-example holdout makes the final baseline-vs-evolved verdict pure noise (±0.01 deltas decide "improved or not"), which silently defeats the purpose of
--eval-source sessiondb.Fix
_is_relevant_to_skillbecomes a thin boolean wrapper over a new_relevance_score(text, skill_name, skill_text) -> int(strength heuristic: +10 exact skill-name phrase, +3 per skill-name word >3 chars, +N keyword-overlap past the existing 2-keyword floor). Boolean semantics (score > 0) are byte-identical to the old rule, so no caller/test behavior changes.filter_and_scoresorts candidates by that score descending before applying the cap, so the LLM budget goes to the strongest matches.Verification
pytest tests/core/test_external_importers.py→ 116 passed (no test changes needed; boolean contract preserved).No behavior change for stores with fewer candidates than the cap (sorting a short list is a no-op for selection).
Attributed to: Diego + Hermes