|
1 | 1 | # hermes-blind |
2 | 2 |
|
| 3 | +**Context-compensation scaffold for LLM evaluation prompts.** A ~40-token language prefix that makes the model disclose prior exposure, score on quoted evidence only, and hedge on thin evidence — so the same prompt stops scoring 6.8 on one run and 8.4 on the next. |
| 4 | + |
3 | 5 | [](https://pypi.org/project/hermes-blind/) |
4 | 6 | [](https://pypi.org/project/hermes-blind/) |
5 | 7 | [](LICENSE) |
6 | 8 | [](#status) |
| 9 | +[](.hermes-seal.yaml) |
7 | 10 |
|
8 | | -**The same eval prompt should not score 6.8 on one run and 8.4 on the next.** Context, memory, and knowledge of authorship bias the scorer. `hermes-blind` is a ~40-token language scaffold you prepend to any scoring or evaluation prompt to force the model to disclose prior exposure, score on quoted evidence only, and hedge on thin evidence. |
9 | | - |
10 | | -It is a string. No model. No API. Backend-agnostic. |
| 11 | +If you're scoring with an LLM that has access to your CLAUDE.md, memory, or session transcript — and the eval is supposed to be neutral but you can feel the model flattering you — this is the string you prepend to the prompt. |
11 | 12 |
|
12 | 13 | ## Pain |
13 | 14 |
|
14 | | -- Same prompt, same target, different run → different score. No audit trail for which score is real. |
15 | | -- Claude scoring code it just wrote. The model knows. The score is inflated. |
16 | | -- CLAUDE.md, memory files, and session context leak into the rubric's judgment and the scorer optimizes to the owner's preferences instead of the target's evidence. |
17 | | -- You want a second opinion, but the second opinion is the same model that has the same context. |
18 | | -- `claude --bare` fixes this for the `claude-cli` backend only. Ollama, Sonnet-via-Bedrock, in-session scoring, and everyone else get nothing. |
| 15 | +- Same prompt, same target, two runs, two scores: 6.8 and 8.4. No way to tell which is real. |
| 16 | +- Claude scoring code Claude just wrote. The model knows it authored the target. The score is inflated and you can't measure by how much. |
| 17 | +- The scorer reads your `CLAUDE.md` and your memory files; it learns your preferences and grades to please you, not to surface evidence. |
| 18 | +- You wanted a second opinion. The second opinion is the same model with the same session context. It is not a second opinion. |
| 19 | +- `claude --bare` solves this for `claude-cli` only. Anything you score through Ollama, OpenAI, or any in-session call gets nothing. |
19 | 20 |
|
20 | 21 | ## Install |
21 | 22 |
|
|
0 commit comments