Skip to content

Commit e9f27de

Browse files
lightningRalflightningRalf
committed
docs(evalset): add stronger example datasets and usage notes
Add v2 and v3 fixed-task-set examples and document their purpose in README to make compare results more meaningful. Update changelog to include new datasets.
1 parent 5d62583 commit e9f27de

4 files changed

Lines changed: 261 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ All notable changes to this project should be documented here.
2323
- `compare` to evaluate baseline vs candidate system prompts
2424
- Added example files in `examples/`:
2525
- `fixed-task-set.json`
26+
- `fixed-task-set-v2.json`
27+
- `fixed-task-set-v3.json`
2628
- `system-baseline.txt`
2729
- `system-candidate.txt`
2830
- Added report output support to `.evalset/reports/*.json` with per-case and aggregate metrics.

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,12 @@ pi -p "/evalset compare examples/fixed-task-set.json examples/system-baseline.tx
7373
/evalset compare examples/fixed-task-set.json examples/system-baseline.txt examples/system-candidate.txt
7474
```
7575

76+
### Included datasets
77+
78+
- `examples/fixed-task-set.json` — tiny smoke set (3 cases)
79+
- `examples/fixed-task-set-v2.json` — larger first pass set
80+
- `examples/fixed-task-set-v3.json` — less brittle checks (recommended)
81+
7682
The command writes JSON reports to:
7783
- explicit `--out <path>` when provided
7884
- otherwise `.evalset/reports/*.json` under your current project directory

‎examples/fixed-task-set-v2.json‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
{
2+
"name": "maintainer-clarity-v2",
3+
"systemPrompt": "Answer in concise, plain English. Prefer direct wording over jargon.",
4+
"cases": [
5+
{
6+
"id": "fixed-task-set-definition",
7+
"input": "In plain English, what does a fixed task set mean in eval workflows?",
8+
"expectContains": ["same", "tasks"]
9+
},
10+
{
11+
"id": "fixed-task-set-benefit",
12+
"input": "Why does using a fixed task set improve comparison quality across runs?",
13+
"expectContains": ["compare", "runs"]
14+
},
15+
{
16+
"id": "reproducibility-two-factors",
17+
"input": "Name two things you should lock or record for reproducible eval runs.",
18+
"expectContains": ["model", "dataset"]
19+
},
20+
{
21+
"id": "pass-rate-calculation",
22+
"input": "If 18 of 24 scored cases pass, what pass rate should be reported?",
23+
"expectContains": ["75"]
24+
},
25+
{
26+
"id": "scored-case-definition",
27+
"input": "What makes a case \"scored\" in this evalset format?",
28+
"expectContains": ["checks", "scored"]
29+
},
30+
{
31+
"id": "variant-hash-purpose",
32+
"input": "What does variantHash help you verify?",
33+
"expectContains": ["variant", "hash"]
34+
},
35+
{
36+
"id": "dataset-hash-purpose",
37+
"input": "Why is datasetHash useful when comparing two reports?",
38+
"expectContains": ["dataset", "same"]
39+
},
40+
{
41+
"id": "delta-interpretation-speed-cost",
42+
"input": "Candidate delta avg latency is -800ms and delta total cost is +0.0003. Summarize the tradeoff.",
43+
"expectContains": ["faster", "cost"]
44+
},
45+
{
46+
"id": "command-non-interactive-pattern",
47+
"input": "Show the correct non-interactive pattern to run compare with a local extension file.",
48+
"expectContains": ["pi -e", "-p", "/evalset compare"]
49+
},
50+
{
51+
"id": "slash-command-shell",
52+
"input": "Can you run /evalset directly in bash without pi? Answer yes or no and one reason.",
53+
"expectContains": ["no", "pi"]
54+
},
55+
{
56+
"id": "report-default-location",
57+
"input": "If --out is omitted, where are evalset reports written by default?",
58+
"expectContains": [".evalset/reports"]
59+
},
60+
{
61+
"id": "model-prerequisite",
62+
"input": "What should you do before /evalset run if no active model is selected?",
63+
"expectContains": ["/model"]
64+
},
65+
{
66+
"id": "max-cases-behavior",
67+
"input": "What does --max-cases 5 do during run/compare?",
68+
"expectContains": ["first", "5"]
69+
},
70+
{
71+
"id": "temperature-range",
72+
"input": "What is the accepted numeric range for --temperature in this extension?",
73+
"expectContains": ["0", "2"]
74+
},
75+
{
76+
"id": "system-merge-behavior",
77+
"input": "If dataset.systemPrompt exists and --system-file is provided, are prompts merged or replaced?",
78+
"expectContains": ["merge"]
79+
},
80+
{
81+
"id": "mutually-exclusive-system-options",
82+
"input": "Can --system-file and --system-text be used together?",
83+
"expectContains": ["no"]
84+
},
85+
{
86+
"id": "run-identity-fields",
87+
"input": "Name any three run identity fields in a run report.",
88+
"expectContains": ["runid", "datasethash", "caseshash"]
89+
},
90+
{
91+
"id": "compare-identity-fields",
92+
"input": "Name the two run-id fields that link baseline and candidate inside compare.run.",
93+
"expectContains": ["baselinerunid", "candidaterunid"]
94+
},
95+
{
96+
"id": "delta-passrate-zero",
97+
"input": "If delta pass rate is 0, what does that imply?",
98+
"expectContains": ["same", "pass rate"]
99+
},
100+
{
101+
"id": "keyword-check-limitation",
102+
"input": "Why can simple keyword checks be misleading for quality?",
103+
"expectContains": ["wording", "false"]
104+
},
105+
{
106+
"id": "improve-weak-evalset",
107+
"input": "Give two concrete ways to improve a weak 3-case evalset.",
108+
"expectContains": ["more", "cases", "criteria"]
109+
},
110+
{
111+
"id": "no-overclaim-rollout",
112+
"input": "Should this be pitched as a huge replacement right away? Answer yes or no and one short reason.",
113+
"expectContains": ["no"],
114+
"expectNotContains": ["huge replacement"]
115+
},
116+
{
117+
"id": "stakeholder-brief",
118+
"input": "Write a one-line stakeholder brief that includes pilot scope and measurement.",
119+
"expectContains": ["pilot", "measure"]
120+
},
121+
{
122+
"id": "tie-communication",
123+
"input": "Baseline and candidate both scored 33.3% pass rate. How should that be communicated?",
124+
"expectContains": ["same", "pass rate"]
125+
}
126+
]
127+
}

‎examples/fixed-task-set-v3.json‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
{
2+
"name": "maintainer-clarity-v3",
3+
"systemPrompt": "Answer in concise, plain English. Prefer direct wording over jargon.",
4+
"cases": [
5+
{
6+
"id": "fixed-task-set-definition",
7+
"input": "In plain English, what does a fixed task set mean in eval workflows?",
8+
"expectContains": ["same", "tasks"]
9+
},
10+
{
11+
"id": "fixed-task-set-benefit",
12+
"input": "Why does a fixed task set improve comparison quality across runs?",
13+
"expectContains": ["compar"]
14+
},
15+
{
16+
"id": "reproducibility-two-factors",
17+
"input": "Name two things you should lock or record for reproducible eval runs.",
18+
"expectContains": ["model", "dataset"]
19+
},
20+
{
21+
"id": "pass-rate-calculation",
22+
"input": "If 18 of 24 scored cases pass, what pass rate should be reported?",
23+
"expectContains": ["75"]
24+
},
25+
{
26+
"id": "scored-case-definition",
27+
"input": "What makes a case scored in this evalset format?",
28+
"expectRegex": "([Ee]xpect|[Cc]heck|[Cc]riteria|[Rr]ule)"
29+
},
30+
{
31+
"id": "variant-hash-purpose",
32+
"input": "What does variantHash help you verify?",
33+
"expectContains": ["variant", "hash"]
34+
},
35+
{
36+
"id": "dataset-hash-purpose",
37+
"input": "Why is datasetHash useful when comparing two reports?",
38+
"expectContains": ["dataset", "hash"]
39+
},
40+
{
41+
"id": "delta-interpretation-speed-cost",
42+
"input": "Candidate delta avg latency is -800ms and delta total cost is +0.0003. Summarize the tradeoff.",
43+
"expectContains": ["faster"]
44+
},
45+
{
46+
"id": "command-non-interactive-pattern",
47+
"input": "Show the correct non-interactive pattern to run compare from shell.",
48+
"expectContains": ["pi", "-p", "evalset"]
49+
},
50+
{
51+
"id": "slash-command-shell",
52+
"input": "Can you run /evalset directly in bash without pi? Answer yes or no and one reason.",
53+
"expectContains": ["no", "slash"]
54+
},
55+
{
56+
"id": "report-default-location",
57+
"input": "If --out is omitted, where are evalset reports written by default?",
58+
"expectContains": ["evalset", "report"]
59+
},
60+
{
61+
"id": "model-prerequisite",
62+
"input": "What should you do before /evalset run if no active model is selected?",
63+
"expectContains": ["/model"]
64+
},
65+
{
66+
"id": "max-cases-behavior",
67+
"input": "What does --max-cases 5 do during run/compare?",
68+
"expectRegex": "([Ff]irst|[Ll]imit|5.*case|case.*5)"
69+
},
70+
{
71+
"id": "temperature-range",
72+
"input": "What is the accepted numeric range for --temperature in this extension?",
73+
"expectContains": ["0", "2"]
74+
},
75+
{
76+
"id": "system-merge-behavior",
77+
"input": "If dataset.systemPrompt exists and --system-file is provided, are prompts merged or replaced?",
78+
"expectContains": ["merge"]
79+
},
80+
{
81+
"id": "mutually-exclusive-system-options",
82+
"input": "Can --system-file and --system-text be used together?",
83+
"expectRegex": "([Nn]o|[Ee]ither|[Oo]ne|not together|[Bb]oth)"
84+
},
85+
{
86+
"id": "run-identity-fields",
87+
"input": "Name any three run identity fields in a run report.",
88+
"expectContains": ["run", "dataset", "case"]
89+
},
90+
{
91+
"id": "compare-identity-fields",
92+
"input": "Name the two run-id fields that link baseline and candidate inside compare.run.",
93+
"expectContains": ["baseline", "candidate", "run"]
94+
},
95+
{
96+
"id": "delta-passrate-zero",
97+
"input": "If delta pass rate is 0, what does that imply?",
98+
"expectRegex": "([Nn]o change|same pass rate|[Uu]nchanged)"
99+
},
100+
{
101+
"id": "keyword-check-limitation",
102+
"input": "Why can simple keyword checks be misleading for quality?",
103+
"expectRegex": "([Kk]eyword).*(mislead|false|[Ee]rror|[Bb]rittle)|([Mm]islead|false|[Ee]rror|[Bb]rittle).*[Kk]eyword"
104+
},
105+
{
106+
"id": "improve-weak-evalset",
107+
"input": "Give two concrete ways to improve a weak 3-case evalset.",
108+
"expectContains": ["more", "cases"]
109+
},
110+
{
111+
"id": "no-overclaim-rollout",
112+
"input": "Should this be pitched as a huge replacement right away? Answer yes or no and one short reason.",
113+
"expectContains": ["no", "phase"]
114+
},
115+
{
116+
"id": "stakeholder-brief",
117+
"input": "Write a one-line stakeholder brief that includes pilot scope and measurement.",
118+
"expectContains": ["pilot", "measure"]
119+
},
120+
{
121+
"id": "tie-communication",
122+
"input": "Baseline and candidate both scored 33.3% pass rate. How should that be communicated?",
123+
"expectRegex": "([Ss]ame|[Nn]o difference|[Uu]nchanged)"
124+
}
125+
]
126+
}

0 commit comments

Comments
 (0)