fix(search): wrap only boolean queries, probe every portal for the verdict

[aborruso]

What the telemetry showed

On 29 July an LLM client spent three hours reformulating the same request against www.dati.gov.it/opendata:

11:47  bonifica siti contaminati Piemonte
12:04  bonifica siti contaminati Piemonte
13:05  siti contaminati bonifica Piemonte
14:25  siti contaminati bonifica aree industriali dismesse brownfield amianto Piemonte
14:38  siti contaminati bonifica Gargallo Novara

The data was there from the first attempt. That query returns 22 datasets on the portal, the Piedmont contaminated-sites registry (ASCO) first. We were returning 5.

Why

package_search hands a colon-free query to Solr's dismax parser with q.op=AND, mm='2<-1 5<80%' and qf='name^4 title^4 tags^2 groups^2 text' (ckan/lib/search/query.py). dismax has no boolean syntax, so A OR B collapses into A AND B — on dati.gov.it aria OR Milano returns 59, exactly what aria AND Milano returns. A colon takes the query off dismax, which is what wrapping it as text:(...) exploits.

The same switch is why the wrapper hurts everything else: it drops the qf boosts that rank titles and tags first, searches the catch-all text field alone, and ANDs every term instead of applying mm.

This is CKAN's own default, not a defect of particular portals — which is why the same pattern shows up everywhere:

portal	multi-word: plain → text	boolean: plain → text
dati.gov.it	650 → 51	59 → 3421
dati.comune.milano.it	52 → 45	0 → 87
dati.toscana.it	14 → 1	2 → 339
dati.regione.sicilia.it	6 → 4	4 → 11
data.gov.ua	4215 → 353	2310 → 20571
data.stadt-zuerich.ch	172 → 0	10 → 0

On dati.gov.it: defibrillatori Comune di Lecce 678 → 1, musei roma arte opere catalogo 9 → 0 — the second most repeated query of the semester, answered with an empty page while nine datasets existed.

On 40 real queries from telemetry the separation is clean:

	wrapper helps	wrapper hurts	no change
with OR/AND/NOT	8	0	1
without	0	10	21

Six of the ten go to zero results.

Changes

The rule (src/utils/search.ts) — the wrapper is reserved for the queries dismax cannot serve:

query shape	wrapped	why
bonifica siti contaminati Piemonte	no	dismax ranks it across qf with mm; wrapping drops both
aria OR Milano	yes	dismax has no boolean syntax and q.op=AND swallows the OR
ambiente -rifiuti	no	dismax honours unary operators natively, so there is nothing to repair
e-government, COVID-19	yes	no operator to the caller, but dismax reads the hyphen as NOT: 58919 of ~65000 datasets against 278 for the escaped literal
title:(aria OR acqua)	no	the colon has already taken it off dismax

An explicit query_parser: "text" from the caller stays a literal override.

The escaping (escapeForTextWrapping) — wrapping only helps if it preserves what the caller wrote. escapeSolrQuery escapes every special character, which inverts a unary operator: text:(ambiente \-rifiuti) returns 398 on dati.gov.it, exactly the set the caller excluded, against 7649 for the unescaped form. So a +/-/! in operator position and balanced grouping parentheses survive; everything else is escaped as before, including a parenthesis the caller escaped, which is a literal rather than a group.

This is what lets a mixed query keep both halves: text:(aria OR acqua -rifiuti) returns 1349 against 1389 for the disjunction alone, so the OR is honoured and the exclusion applied. dismax returns 21, having swallowed the OR. And on a real grouped query from the telemetry, (aria OR "qualità dell'aria") AND Milano: dismax 0, escaped parentheses 51, preserved 59.

The probe (src/tools/package.ts) — rewritten, and now run for every portal, configured ones included. force_text_field is gone from portals.json (8 entries) so there is one source of truth; the stored values had gone stale, with two portals marked "no wrapping" for a reason the rule now handles.

The old probe asked data OR dati, two words common enough to saturate: on Milano data alone and data OR dati both return 2564, so it read the portal as healthy while aria OR acqua returned 0 against 54 and 33 for the single terms. It now picks two terms from the catalog itself — single-word tag facets between 0.5% and 30%, falling back to frequent title words, restricted to letters, digits and underscore so no Solr syntax leaks into the probe — and compares A, B, A OR B, text:(A OR B). An OR returning fewer hits than either operand is not being honoured, and the wrapper is the answer only if the wrapped form returns more. data.stadt-zuerich.ch, whose text field returns 0 for every query, is correctly left alone.

A verdict is cached only when it was actually measured, so a portal that times out or 500s is retried rather than written off for the session. The title fallback prefers a second term appearing where the first does not, since two words that always travel together make A OR B equal A even on a healthy portal.

Cost — a plain query pays nothing, since nothing but a boolean query can be wrapped. Measured from the audit log: 1 call for a plain search, 6 for the first boolean search on a portal (5 probe + 1 real), 1 afterwards.

Verification

• 527 tests, 29 added
• e2e against live portals:

query	before	after
bonifica siti contaminati Piemonte	5	22, ASCO registry first
musei roma arte opere catalogo	0	9
aria OR acqua (Milano)	0	87
ambiente -rifiuti	398, inverted	7649, exclusion intact
aria OR acqua -rifiuti	471	1349, both halves
(aria OR acqua) AND Milano	39	98, grouping preserved
e-government	278	278, unchanged
Zurich, boolean and plain	10 / 172	unchanged

Out of scope, found on the way

catalog.data.gov answers 404 on package_search, and dati.arpae.it is intermittent — 200 on one call, 500 on the next, wrapped or not. Both are configured portals and deserve their own issue. data.gov.uk is fine: it answers through its configured /api/action path.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MEpSWpAwuMaGkfnMpQdqK2

[greptile-apps]

Greptile Summary

The PR narrows automatic text:(...) wrapping to boolean queries and replaces stored portal parser settings with runtime measurements. The latest changes correctly distinguish escaped literal parentheses from actual balanced groups.

• Plain keyword searches retain each portal’s default relevance behavior.
• Boolean searches probe portal behavior and cache only successfully measured verdicts.
• Explicit parser overrides and accent-removal retries preserve the selected parser.
• Solr escaping now preserves real grouping and unary operators while keeping escaped parentheses literal.
• Portal configuration and documentation were updated to reflect runtime probing.

Important Files Changed

Filename	Overview
src/utils/search.ts	Adds boolean-only wrapping and syntax-aware escaping; the latest escaped-parenthesis fix addresses the previous finding.
src/tools/package.ts	Replaces static parser decisions with measured per-portal probing and preserves the resulting override during retries.
src/portals.json	Removes stale per-portal search-parser verdicts in favor of runtime detection.
tests/unit/search.test.ts	Adds coverage for boolean detection, text wrapping, unary operators, grouping, escaped parentheses, and parser overrides.
src/README.md	Documents boolean-only wrapping, runtime portal probing, and its request cost.
LOG.md	Records the search regression, parser behavior, implementation strategy, and verification results.

_{Reviews (5): Last reviewed commit: "fix(search): a parenthesis the caller es..." | Re-trigger Greptile}