docs(eval-harness): note single-shot retrieval scope vs tool-based agents (#579)

The harness performs one fixed batch of graph searches per test case and
passes the resulting context block to the response model in a single turn.
That mirrors deterministic/programmatic retrieval, but production agents are
often built by exposing Zep to the model as tools, where the LLM decides when
and what to search.

Add a "Scope: Single-Shot Retrieval" section to the README and the eval
harness skill so results aren't read as a prediction of agent performance,
plus a reminder in the skill's results-analysis guidance to state the caveat
when presenting conclusions.

Co-authored-by: Claude <noreply@anthropic.com>

Author: jackaldenryan

zep-eval-harness/.claude/commands/zep-eval-harness/SKILL.md

@@ -13,6 +13,12 @@ The pipeline has four steps:
 3. **Ingest documents** — send pre-chunked documents to a standalone Zep document graph
 4. **Evaluate** — for each test case: search graphs → assess context completeness → generate LLM response → grade answer accuracy
 
+### Scope: Single-Shot Retrieval
+
+**The harness evaluates single-shot retrieval only.** Every test case issues one fixed batch of scoped searches from the raw test question (nodes + edges, optionally episodes, across the user graph and any document graph), then hands the resulting context block to the response model in a single turn — no second retrieval round, no query reformulation. This mirrors deterministic/programmatic retrieval, not the tool-based pattern where an agent is handed Zep search tools (e.g. `search_graph` from the Zep MCP server) and decides when and what to search.
+
+That makes the harness a clean instrument for the ingestion and search configuration, but it says nothing about agent tool-use behavior. A tool-based agent may do better (several targeted searches, reformulating after a weak result) or worse (never searching, poorly phrased queries, running out of turns). When reporting results, scope conclusions to the config under test and never present them as a prediction of production agent performance.
+
 ### Evaluation Metrics
 
 **Context Completeness (PRIMARY)** — Did Zep retrieve sufficient information to answer the question?
@@ -203,6 +209,8 @@ Evaluation results live at `runs/evaluations/{N}_{ts}/results.json`. Each result
 
 When comparing runs, focus on completeness differences. Accuracy is still worth tracking (it catches cases where context is technically present but hard for the LLM to use), but completeness is what tells you whether Zep's graph and search are doing their job.
 
+Both metrics describe a single-shot retrieval path (see **Scope** above). State that caveat when presenting conclusions, and if the user's production design exposes Zep through tools, note that the agent path needs its own end-to-end evaluation.
+
 ## Typical Full Pipeline
 
 ```bash

zep-eval-harness/README.md

@@ -179,6 +179,19 @@ Rate limits are handled automatically — if you hit limits, the retry backoff w
 3. **Generate Response**: Use LLM with retrieved context to answer questions
 4. **Grade Answer**: Evaluate answers against golden answers using LLM judge (SECONDARY METRIC)
 
+### Scope: Single-Shot Retrieval
+
+**This harness evaluates single-shot retrieval only.** Each test case issues one fixed batch of searches — the raw test question against nodes and edges (plus episodes if enabled), across the user graph and any document graph, all in parallel — assembles the results into a context block, and hands it to the response model in a single turn. There is no second retrieval round and no query reformulation. That mirrors *deterministic/programmatic* retrieval, where your application searches on every turn and injects the context itself.
+
+Production agents are frequently built the other way: Zep is exposed to the model as **tools** — for example `search_graph` from the [Zep MCP server](../mcp/zep-mcp-server/), or your own tool definitions — and the LLM decides when to search, how to phrase each query, and whether to search again after seeing results.
+
+Read the scores accordingly:
+
+- **What they measure**: whether your ingestion and search configuration (ontology, custom instructions, chunking, search limits, reranker) puts the right facts within reach of one well-formed query. Pinning retrieval to a single deterministic search is what keeps runs comparable — the config stays the only variable.
+- **What they do not measure**: agent behavior. A tool-based agent can beat these numbers by issuing several targeted searches and reformulating after a weak result, or fall short of them by not searching at all, phrasing a query poorly, or running out of turns. Tool choice, query formulation, and multi-turn dynamics are untested here.
+
+If your production path exposes Zep through tools, treat a strong result here as a prerequisite rather than a verdict, and evaluate the agent end-to-end as well. See [Evaluate Zep for your use case](https://help.getzep.com/evaluate-zep-for-your-use-case).
+
 ## Configuration
 
 All use-case-specific configuration lives in `config/`, organized by pipeline step: