Skip to content

Commit d5c2c4c

Browse files
aborrusoclaude
andcommitted
test(smoke): require field presence, use floors for catalog counts
Two review findings, both fair: - `field_equals` treated a missing field as an explicit null, so a response that stopped emitting `all_terms_results` would have passed the boolean-query case. Presence is checked now: removing the field from the payload fails with "all_terms_results is missing from the response, expected null". - pinning the Milano case at exactly 327 matching datasets broke this file's own rule that thresholds survive catalog drift — my own line from yesterday. Counts that track the catalog are floors now (`field_min`), while `null` and `0` stay equalities: they are behaviours, not counts. Both defects still caught: removing the boolean skip fails with "all_terms_results is 10, expected null". 16/16. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEpSWpAwuMaGkfnMpQdqK2
1 parent 7d07f8a commit d5c2c4c

4 files changed

Lines changed: 24 additions & 10 deletions

File tree

LOG.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,12 @@ passed it. `returned_min` was added: a tool can report thousands of matches and
1616
an empty list, and those are different claims. Both cases now fail on the broken build and
1717
pass on the fixed one.
1818

19-
Also added `field_equals`, an exact check on any payload field: the strict count had to be
20-
pinned at 1, at 0 and at null, which is three meanings of one field.
19+
Also added `field_equals`, an exact check on any payload field, and `field_min`, a floor.
20+
The split came from review: pinning the Milano case at exactly 327 matching datasets broke
21+
the file's own rule that thresholds survive catalog drift, so counts that track the catalog
22+
are floors now, while `null` and `0` stay equalities — they are behaviours, not counts.
23+
`field_equals` also requires the field to be present, or a response that stopped emitting
24+
`all_terms_results` would read as an explicit null and pass.
2125

2226
16 cases, 16 passing.
2327

scripts/README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,8 @@ Assertions available in a case's `expect`: `total_min` / `total_max` (catalog ma
115115
`returned_min` (rows actually handed back — not the same claim), `count_min` (rows in a
116116
listing tool), `first_matches` (regex on the first title), `margin_min` (how far the first
117117
result leads the second), `terms_equal` (the extracted query terms), `field_equals` (an
118-
exact value for any payload field, `null` included), `wrapped` / `not_wrapped` /
118+
exact value for any payload field, `null` included — the field must be present),
119+
`field_min` (a numeric floor, for values that track the catalog and drift), `wrapped` / `not_wrapped` /
119120
`effective_contains` (what reached Solr), `error_matches` (the case expects an error).
120121

121122
Cases hit real portals, so the thresholds are loose enough to survive catalog drift and

scripts/smoke.mjs

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -153,11 +153,20 @@ function check(expect, payload) {
153153
// included. Written generic rather than per-field: the cases that needed it wanted
154154
// `all_terms_results` at 1, at 0 and at null, which is three meanings of one field.
155155
for (const [field, want] of Object.entries(expect.field_equals ?? {})) {
156-
if (JSON.stringify(payload[field] ?? null) !== JSON.stringify(want))
157-
fail.push(`${field} is ${JSON.stringify(payload[field] ?? null)}, expected ${JSON.stringify(want)}`);
156+
// Presence is part of the check: a field that stopped being emitted must not read
157+
// as an explicit null, or the boolean-query case would pass on a response that no
158+
// longer carries `all_terms_results` at all.
159+
if (!Object.hasOwn(payload, field))
160+
fail.push(`${field} is missing from the response, expected ${JSON.stringify(want)}`);
161+
else if (JSON.stringify(payload[field]) !== JSON.stringify(want))
162+
fail.push(`${field} is ${JSON.stringify(payload[field])}, expected ${JSON.stringify(want)}`);
158163
}
159164
if (expect.terms_equal && JSON.stringify(payload.terms ?? null) !== JSON.stringify(expect.terms_equal))
160165
fail.push(`terms ${JSON.stringify(payload.terms)} differ from ${JSON.stringify(expect.terms_equal)}`);
166+
for (const [field, min] of Object.entries(expect.field_min ?? {})) {
167+
if (!(typeof payload[field] === "number" && payload[field] >= min))
168+
fail.push(`${field} is ${JSON.stringify(payload[field])}, expected a number of at least ${min}`);
169+
}
161170
if (expect.margin_min !== undefined) {
162171
const [a, b] = (payload.results ?? []).map((r) => r.score ?? 0);
163172
if (a === undefined || b === undefined || a - b < expect.margin_min)

tests/smoke/cases.json

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,7 @@
136136
"comune",
137137
"lecce"
138138
],
139-
"field_equals": {
139+
"field_min": {
140140
"all_terms_results": 1
141141
}
142142
}
@@ -208,10 +208,10 @@
208208
"aria",
209209
"milano"
210210
],
211-
"field_equals": {
212-
"all_terms_results": 327
213-
},
214-
"margin_min": 0
211+
"margin_min": 0,
212+
"field_min": {
213+
"all_terms_results": 163
214+
}
215215
}
216216
},
217217
{

0 commit comments

Comments
 (0)