Skip to content

Commit 9f21845

Browse files
aborrusoclaude
aborruso
and
claude
authored
test(smoke): cover the strict pass, and fix a case that passed for free (#545)
* test(smoke): cover the strict pass, and fix a case that passed for free The mm=100% pass from #544 was verified by hand only. Two cases now cover it: a boolean query must skip it (all_terms_results: null), and a query no dataset satisfies in full must still be answered by the fill pass (all_terms_results: 0). Checking that they fail on the defects they name found that the second did not. It asserted total_results and all_terms_results but never that any result came back, so a build where the strict pass emptied the answer passed it. Added `returned_min`: a tool can report thousands of matches and return an empty list, and those are different claims. Both cases now fail on the broken build: FAIL a boolean query skips the strict pass all_terms_results is 10, expected null FAIL no dataset carries every term, so the default pass fills in returned 0 results, expected at least 3 Also added `field_equals`, an exact check on any payload field including null — the strict count needed pinning at 1, at 0 and at null. Assertions documented in scripts/README.md. 16 cases, 16 passing. 550 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEpSWpAwuMaGkfnMpQdqK2 * test(smoke): require field presence, use floors for catalog counts Two review findings, both fair: - `field_equals` treated a missing field as an explicit null, so a response that stopped emitting `all_terms_results` would have passed the boolean-query case. Presence is checked now: removing the field from the payload fails with "all_terms_results is missing from the response, expected null". - pinning the Milano case at exactly 327 matching datasets broke this file's own rule that thresholds survive catalog drift — my own line from yesterday. Counts that track the catalog are floors now (`field_min`), while `null` and `0` stay equalities: they are behaviours, not counts. Both defects still caught: removing the boolean skip fails with "all_terms_results is 10, expected null". 16/16. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEpSWpAwuMaGkfnMpQdqK2 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 0035fe1 commit 9f21845

4 files changed

Lines changed: 95 additions & 2 deletions

File tree

‎LOG.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,29 @@
22

33
## 2026-09-06
44

5+
### Two gate cases for the strict pass, and one that was passing for free
6+
7+
The `mm=100%` pass added in #544 was verified by hand, so nothing stopped it from
8+
regressing. Two cases now cover it: a boolean query must skip the strict pass
9+
(`all_terms_results: null`), and a query no dataset satisfies in full must still be
10+
answered by the fill pass (`all_terms_results: 0`, results returned).
11+
12+
Checking that they fail on the defects they name — which is the point of this file —
13+
found that the second one did not. It asserted `total_results` and `all_terms_results`
14+
but never that any result came back, so a build where the strict pass emptied the answer
15+
passed it. `returned_min` was added: a tool can report thousands of matches and hand back
16+
an empty list, and those are different claims. Both cases now fail on the broken build and
17+
pass on the fixed one.
18+
19+
Also added `field_equals`, an exact check on any payload field, and `field_min`, a floor.
20+
The split came from review: pinning the Milano case at exactly 327 matching datasets broke
21+
the file's own rule that thresholds survive catalog drift, so counts that track the catalog
22+
are floors now, while `null` and `0` stay equalities — they are behaviours, not counts.
23+
`field_equals` also requires the field to be present, or a response that stopped emitting
24+
`all_terms_results` would read as an explicit null and pass.
25+
26+
16 cases, 16 passing.
27+
528
### #539: the window is the lever, not the score
629

730
The issue proposed IDF for the tie at 9.7 on `defibrillatori Comune di Lecce`. Prototyped

‎scripts/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,14 @@ wrapping rule fails 4 of the 12 cases; removing the parser probe from
111111
`ckan_find_relevant_datasets`, the v0.4.122 regression, fails the case that requires the
112112
two search tools to agree.
113113

114+
Assertions available in a case's `expect`: `total_min` / `total_max` (catalog matches),
115+
`returned_min` (rows actually handed back — not the same claim), `count_min` (rows in a
116+
listing tool), `first_matches` (regex on the first title), `margin_min` (how far the first
117+
result leads the second), `terms_equal` (the extracted query terms), `field_equals` (an
118+
exact value for any payload field, `null` included — the field must be present),
119+
`field_min` (a numeric floor, for values that track the catalog and drift), `wrapped` / `not_wrapped` /
120+
`effective_contains` (what reached Solr), `error_matches` (the case expects an error).
121+
114122
Cases hit real portals, so the thresholds are loose enough to survive catalog drift and
115123
a failure can also mean a portal is down — check the message before assuming a code bug.
116124

‎scripts/smoke.mjs‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,10 @@ function check(expect, payload) {
137137
fail.push(`expected at most ${expect.total_max} results, got ${total}`);
138138
if (expect.count_min !== undefined && !(listLength(payload) >= expect.count_min))
139139
fail.push(`expected at least ${expect.count_min} rows, got ${listLength(payload)}`);
140+
// `returned_min` is about the answer, not the catalog: a tool can report thousands of
141+
// matches and hand back an empty list. The fill case passed a broken build without it.
142+
if (expect.returned_min !== undefined && !((payload.results ?? []).length >= expect.returned_min))
143+
fail.push(`returned ${(payload.results ?? []).length} results, expected at least ${expect.returned_min}`);
140144
if (expect.first_matches && !new RegExp(expect.first_matches).test(firstTitle(payload)))
141145
fail.push(`first result "${firstTitle(payload).slice(0, 60)}" does not match /${expect.first_matches}/`);
142146
if (expect.wrapped && !effective.startsWith("text:("))
@@ -145,8 +149,24 @@ function check(expect, payload) {
145149
fail.push(`expected no wrapper, effective query was "${effective}"`);
146150
if (expect.effective_contains && !effective.includes(expect.effective_contains))
147151
fail.push(`effective query "${effective}" does not contain "${expect.effective_contains}"`);
152+
// `field_equals: { name: value }` — an exact check on any payload field, `null`
153+
// included. Written generic rather than per-field: the cases that needed it wanted
154+
// `all_terms_results` at 1, at 0 and at null, which is three meanings of one field.
155+
for (const [field, want] of Object.entries(expect.field_equals ?? {})) {
156+
// Presence is part of the check: a field that stopped being emitted must not read
157+
// as an explicit null, or the boolean-query case would pass on a response that no
158+
// longer carries `all_terms_results` at all.
159+
if (!Object.hasOwn(payload, field))
160+
fail.push(`${field} is missing from the response, expected ${JSON.stringify(want)}`);
161+
else if (JSON.stringify(payload[field]) !== JSON.stringify(want))
162+
fail.push(`${field} is ${JSON.stringify(payload[field])}, expected ${JSON.stringify(want)}`);
163+
}
148164
if (expect.terms_equal && JSON.stringify(payload.terms ?? null) !== JSON.stringify(expect.terms_equal))
149165
fail.push(`terms ${JSON.stringify(payload.terms)} differ from ${JSON.stringify(expect.terms_equal)}`);
166+
for (const [field, min] of Object.entries(expect.field_min ?? {})) {
167+
if (!(typeof payload[field] === "number" && payload[field] >= min))
168+
fail.push(`${field} is ${JSON.stringify(payload[field])}, expected a number of at least ${min}`);
169+
}
150170
if (expect.margin_min !== undefined) {
151171
const [a, b] = (payload.results ?? []).map((r) => r.score ?? 0);
152172
if (a === undefined || b === undefined || a - b < expect.margin_min)

‎tests/smoke/cases.json‎

Lines changed: 44 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,10 @@
135135
"defibrillatori",
136136
"comune",
137137
"lecce"
138-
]
138+
],
139+
"field_min": {
140+
"all_terms_results": 1
141+
}
139142
}
140143
},
141144
{
@@ -204,7 +207,46 @@
204207
"qualità",
205208
"aria",
206209
"milano"
207-
]
210+
],
211+
"margin_min": 0,
212+
"field_min": {
213+
"all_terms_results": 163
214+
}
215+
}
216+
},
217+
{
218+
"name": "a boolean query skips the strict pass",
219+
"regression": "mm=100% would require the OR token itself; caught in review on #544",
220+
"tool": "ckan_find_relevant_datasets",
221+
"server": "https://data.stadt-zuerich.ch",
222+
"args": {
223+
"query": "Bevölkerung OR Einwohner",
224+
"limit": 2
225+
},
226+
"expect": {
227+
"total_min": 5,
228+
"field_equals": {
229+
"all_terms_results": null
230+
},
231+
"returned_min": 2
232+
}
233+
},
234+
{
235+
"name": "no dataset carries every term, so the default pass fills in",
236+
"regression": "a strict pass returning nothing must not empty the answer (#544)",
237+
"tool": "ckan_find_relevant_datasets",
238+
"server": "https://dati.comune.messina.it",
239+
"args": {
240+
"query": "bilancio spese comune",
241+
"limit": 3
242+
},
243+
"expect": {
244+
"total_min": 1,
245+
"field_equals": {
246+
"all_terms_results": 0
247+
},
248+
"returned_min": 3,
249+
"first_matches": "[Ss]pese|[Bb]ilancio|[Cc]omune"
208250
}
209251
}
210252
]

0 commit comments

Comments
 (0)