night-shift: prompt-regression eval-harness skeleton - #2
Merged
Conversation
Tier 2 item 16 (PROJECT-GENESIS.md section 9): src/personal_llm/eval/ - a pure, generic pass/fail runner (harness.py) plus 3 builtin cases (cases.py) over rag.pipeline.ask and review.weekly.generate_review, so a reworded system prompt that silently changes observable behavior (grounding, citations, report shape) gets caught as a regression instead of only a subjective read of a diff. New `eval` CLI command runs the suite standalone (no engine bootstrap needed) and exits non-zero on failure - usable as a CI gate later. 7 new tests; full suite 107/107 green offline, no API key or network.
…ts TemporaryDirectory tears down
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
src/personal_llm/eval/:harness.py- a pure, generic runner:EvalCase(name +run()+Assertions),run_eval(cases) -> EvalReport. A case whoserun()raises is recorded as a failurewith the exception message instead of crashing the whole sweep.
cases.py- 3 builtin cases againstrag.pipeline.askandreview.weekly.generate_review, using a small self-contained_ScriptedRouter(mirrors
tests/conftest.py'sFakeRoutersurface, kept separate becausetests/isn't importable at runtime and these cases also run from the CLI).
evalCLI command that runs the suite standalone (no engine/API key needed)and exits non-zero if any case fails.
Why
PROJECT-GENESIS.md section 9, Tier 2, item 16: "personal-llm: eval-harness skeleton
for prompt regression". The existing test suite mocks the router's
complete()returnvalue directly, so it never actually exercises whether a reworded system prompt
(
_SYSTEMinrag/pipeline.pyorreview/weekly.py) still produces the rightobservable behavior - grounding falling back to "not in memory" correctly, citations
surviving, a review report keeping its shape. This harness gives that a name and a
place to grow: assertions check output shape/content, never literal prompt wording, so
prompts stay free to be improved.
How verified
7 new tests in
tests/test_eval.py: harness pass/fail aggregation, failure-descriptionreporting, exception isolation (one bad case doesn't take down the run), the empty-list
edge case, a baseline assertion that all 3 builtin cases currently pass, and the CLI
command's exit-code behavior (0 on pass, 1 on failure, monkeypatched to force a
failure).
pytest tests/ -q-> 107/107 passed, offline, no network or API key (installed thefull
requirements.txt, including chromadb/sentence-transformers, to run the realsuite rather than skip it). Also ran the new
evalcommand directly end-to-end:all 3 builtin cases print
[PASS]and the summary line.