hermeneutic is an evidence-first drift gate that mines corrections from AI chat logs and gates the next response before drift ships.
Your AI overclaims. You correct it. Now your AI gets gated.
Mined 326 corrections across 1,423 chat sessions. 44% were post-completion overclaiming — the dominant drift mode. 6 regex rules now catch ~65% of that distribution before the next response ships. Three stages, fail-cheap to fail-expensive. Free, MIT, zero dependencies.
Every chat log you've ever produced contains a hidden, labeled dataset:
prior_assistant ← the drift
↓
user_correction ← you, doing the work
↓
next_assistant ← the repair
Most teams throw it away. hermeneutic mines it, classifies the drift, and runs it as a pre-flight gate on the next outgoing response.
draft → regex (~0ms) most ship here
↓ if hit
hermes-rubric many ship here
↓ if fail
PressureProbe ship | revise | hold
↓ if revise
repair pass ship
Cheap-to-expensive. Most drafts pass stage 1 untouched. The stage that costs you an API call only fires on the ~10–20% that look risky.
pip install hermeneutic
hermeneutic mine ~/.claude/projects/*/ --out triples.jsonl
hermeneutic bucket triples.jsonl
echo "Done — shipped 14 files, all tests pass." | hermeneutic gateThat last line returns:
RISK — high
completion_with_number: 'Done — shipped 14'
completion_with_all_quantifier: 'Done — shipped 14 files, all'
Your AI was about to overclaim. It can't anymore.
| Mine | Walk any chat-log directory (Claude Code, OpenAI), extract correction triples |
| Bucket | See your AI's actual drift modes — not what someone else thinks they are |
| Gate | Run the 3-stage pre-flight on any outgoing draft |
| Library | Full Python API. Plug into your pipeline in 4 lines. |
from hermeneutic import Router, PressureProbe
probe = PressureProbe(judge=your_llm) # any callable: prompt -> str
router = Router(
probe=probe,
repairer=lambda req, draft, why: your_llm(f"Revise: {why}\n\n{draft}"),
)
result = router.gate(request="Build me a thing", draft=your_draft)
print(result.summary()) # shipped@repair risk=high(2) twin=revise REPAIRED
print(result.final_output) # the safer, ship-ready versionThat's it. No subclassing, no config file, no provider lock-in.
A reviewer-twin has two layers:
- The role — a critic that forces structured output: verdict + flip-condition + evidence pointer
- The priors — whose judgment, what red flags, what severity calibration
We ship the role as PressureProbe. You bring your own calibration. The architecture does the work even when the calibration is generic — your users can't game "what would falsify this?" by being lazy.
| Audience | What you do | What you get |
|---|---|---|
| Solo dev | Use the default rigorous-skeptic prior | Instant gate on your AI's worst drafts |
| Team | Drop in your own calibration text | Codified house style, every commit |
| Enterprise | Swap in your domain priors (security, medical, legal) | Private calibration, public-tested architecture, ship to every team |
One library. Every audience.
After pip install hermeneutic and one mining pass on your existing logs:
- Every outgoing draft gets a free pre-flight check — most pass in microseconds, only the risky ones cost a downstream LLM call.
- Confident "Done — shipped 14 files, all green" claims get caught before they ship, with the specific drift pattern named (
completion_with_number). - Subagent-passthrough text ("the agents converged on…") gets flagged so you don't propagate unverified summaries.
- You build a labeled dataset of (my draft, gate verdict, your acceptance) every time the gate fires — that's your data flywheel for v0.2 patterns.
- Your team's house style gets codified as a
PressureProbecalibration string instead of living in tribal Slack messages.
The gate doesn't make your AI smarter. It stops the most common drift modes from reaching the user.
Both pluggable surfaces are typing.Protocol interfaces — no subclassing required, no framework lock-in:
from typing import Protocol
# Plug in any LLM as the critic — OpenAI, Anthropic, Ollama, your own twin.
class LLMJudge(Protocol):
def __call__(self, prompt: str) -> str: ...
# Plug in any chat-log format by subclassing LogReader (registered in READERS dict).
from hermeneutic.triples import LogReader, READERS
class MyFormatReader(LogReader):
name = "my-format"
def iter_turns(self, path):
# yield (role, text, timestamp) tuples
...
READERS["my-format"] = MyFormatReader()Default judge calibration is rigorous-skeptic; default readers cover Claude Code JSONL and OpenAI ChatCompletion JSON.
This isn't a thought experiment. The risk patterns ship with hermeneutic because we mined a real corpus first:
- 1,423 sessions of one heavy AI user
- 326 corrections extracted as
(drift, steer, repair)triples - 44% (143/326) were post-completion overclaiming — the dominant drift mode
- 6 regex rules cover ~65% of the corpus
Every pattern in gates/regex.py traces to corrections caught in the wild. Methodology, bucket distribution, and pattern derivation are documented in evals/triple-mining-receipts.md. The 326 triples themselves are not shipped (private session content). Your distribution will look different — that's the point. Run the miner on your own logs and your gate writes itself.
The gate should catch its own announcement language. If the demo below ships zero hits, the rules are broken:
git clone https://github.com/hermes-labs-ai/hermeneutic && cd hermeneutic
pip install -e .
bash evals/self_test.sh
# PASS — gate correctly flagged the deliberately drift-shaped draft.Then run hermeneutic mine against your own chat logs to bucket your distribution and see which rules apply most.
hermeneutic is free and will stay free. No tier. No telemetry. No auth wall. No "open core" bait.
It's part of the Hermes Labs audit stack — small, sharp, free OSS for teams shipping AI in production:
| Tool | Catches |
|---|---|
scaffold-lint |
Bad prompt structure (static) |
hermes-rubric |
Vibes-based scoring (evidence-first judge) |
agent-convergence-scorer |
Multi-agent disagreement |
hermes-seal |
Tampered or unverified artifacts |
hermeneutic |
Drifted responses, before they ship |
Static linters catch the prompt. hermeneutic catches the response. You get both. Free.
- You ship an AI feature and your users say "wait, are you sure?" too often.
- You run agents and want a cheap pre-flight gate before responses go out.
- You build LLM apps and don't have time for another evaluation framework.
- You audit AI systems and want a tool that grounds in evidence, not vibes.
- Your AI never drifts. If users don't push back on outputs, there's no signal to mine and the gate has nothing to enforce.
- You need foresight, not memory. The gate catches drift modes already corrected in your logs. Genuinely novel drift modes pass through silently until you re-mine.
- You want aggregate model evaluation. This scores individual outgoing drafts. For benchmarks, use a benchmark suite.
- You expect it to replace human review. It raises the floor on the worst drafts; the top of your distribution is unaffected.
- You can't afford one extra LLM call on the ~10–20% of drafts that look risky. Stage 1 (regex) is free; stage 3 (PressureProbe) costs one API call per gated draft.
docs/THEORY.md— why this works, and where it doesn'texamples/before_after.md— one drift caught and repaired, end-to-endAGENTS.md— how a coding agent should use this toolllms.txt— machine-readable summary
MIT. Use it, fork it, ship it. If you build something interesting on top, tell us.
Hermes Labs is an independent AI-reliability lab building open-source tools that catch silent failure modes in production AI. More at hermes-labs.ai.