fix(search): wrap only boolean queries, probe every portal for the verdict

LOG.md

@@ -2,6 +2,52 @@
 
 ## 2026-09-05
 
+### Solr: wrap only boolean queries, and measure every portal instead of storing a verdict
+
+The usage review surfaced an LLM client reformulating the same request against
+`www.dati.gov.it/opendata` for three hours on 29 July. The data was there from the first
+attempt: `bonifica siti contaminati Piemonte` returns 22 datasets on that portal, the
+Piedmont contaminated-sites registry first. Our `text:(...)` rewrite was returning 5.
+
+`package_search` hands a colon-free query to Solr's dismax parser with `q.op=AND`,
+`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'` (`ckan/lib/search/query.py`).
+dismax has no boolean syntax, so `A OR B` collapses into `A AND B`: on dati.gov.it
+`aria OR Milano` returns 59, exactly what `aria AND Milano` returns. A colon takes the query
+off dismax, which is what the wrapper exploits — the wrapped form returns 3421. The same
+switch is why it hurts everything else: it drops the `qf` boosts that rank titles and tags
+first, searches the catch-all `text` field alone, and ANDs every term instead of applying
+`mm`. This is CKAN's own default, not a per-portal defect, which is why the same pattern
+showed up on Milano, Toscana, Sicilia, Ucraina and open.canada.ca.
+
+Measured on dati.gov.it: `defibrillatori Comune di Lecce` 678 -> 1, `qualità aria Milano`
+650 -> 51, `musei roma arte opere catalogo` 9 -> 0 — the second most repeated query of the
+semester, answered with an empty page while nine datasets existed. On 40 real queries from
+telemetry the separation is clean: with a boolean operator the wrapper helped 8 times and
+hurt none; without one it helped none and hurt 10, six of them down to zero.
+
+So the wrapper now applies only to queries carrying a boolean operator. The shape that an
+LLM client generates from a user's request — a run of keywords, no operators — is left to
+the portal's own parser, boosts and `mm` included.
+
+`probePortalParser` was rewritten and now runs for every portal, configured ones included;
+`force_text_field` is gone from `portals.json` (8 entries) so there is one source of truth.
+The old probe asked `data OR dati`, two words common enough to saturate: on Milano `data`
+alone and `data OR dati` both return 2564, so it read the portal as healthy while
+`aria OR acqua` returned 0 against 54 and 33 for the single terms. It now picks two terms
+from the catalog itself — single-word tag facets between 0.5% and 30%, falling back to
+frequent title words — and compares `A`, `B`, `A OR B`, `text:(A OR B)`. An OR returning
+fewer hits than either operand is not being honoured, and the wrapper is the answer only if
+the wrapped form returns more. `data.stadt-zuerich.ch`, where the `text` field returns 0 for
+every query, is correctly left alone.
+
+Cost: a plain query pays nothing, since nothing but a boolean query can be wrapped. The
+probe costs 5 extra `rows=0` calls the first time a boolean query reaches a portal in a
+session, then nothing.
+
+Verified e2e against live portals: dati.gov.it 5 -> 22 and 0 -> 9 on the two queries from
+the telemetry, Milano `aria OR acqua` 0 -> 87, Zurich unchanged at 10 and 172. 516 tests
+pass, 18 added.
+
 ### v0.4.121 - clearer 404 on datastore_search_sql
 
 Patch release for the fix in #532: a 404 on `datastore_search_sql` now says the portal does not expose the SQL endpoint, instead of sending the caller to `ckan_package_show` for a resource_id that was already valid. Also ships the telemetry pipeline fixes and the DEPLOYMENT.md/CLAUDE.md realignment from earlier today.

src/README.md

@@ -53,10 +53,10 @@ Each entry in `portals` array supports:
   - Example: data.europa.eu requires this due to its DCAT-AP based field structure
 
 - **`search`** (object): Search behavior configuration
-  - **`force_text_field`** (boolean): Force wrapping non-fielded queries in `text:(...)`
-    - Default: `false`
-    - Set to `true` for portals with restrictive query parsers that break on long OR queries
-    - Example: dati.gov.it requires this to handle queries like `"hotel OR alberghi"`
+  - **`force_text_field`** (boolean): forces `text:(...)` wrapping on non-fielded queries.
+    Still honoured if present, but **no portal sets it any more** and new ones should not:
+    the decision is measured at runtime by `probePortalParser()` in `src/tools/package.ts`.
+    The stored values went stale and were removed on 2026-09-05.
 
 ### Defaults
 
@@ -72,14 +72,46 @@ The `defaults` object provides fallback values when a portal is not found in the
 }
 ```
 
+### Query parser: when `text:(...)` wrapping is applied
+
+`package_search` sends a colon-free query to Solr's **dismax** parser with `q.op=AND`,
+`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'` (`ckan/lib/search/query.py`).
+dismax has no boolean syntax, so `A OR B` collapses into `A AND B`. A colon in the query
+takes it off dismax, which is what wrapping it as `text:(A OR B)` exploits.
+
+The same switch is why wrapping hurts everything else: it drops the `qf` boosts that rank
+titles and tags first, searches the catch-all `text` field alone, and ANDs every term
+instead of applying `mm`. Measured on dati.gov.it, 2026-09-05:
+
+| query | plain | `text:(...)` |
+|---|---|---|
+| `ambiente` | 8047 | 8047 |
+| `qualità aria Milano` | 650 | 51 |
+| `defibrillatori Comune di Lecce` | 678 | 1 |
+| `musei roma arte opere catalogo` | 9 | 0 |
+| `aria OR Milano` | 59 | 3421 |
+
+So the wrapper is applied **only to queries carrying a boolean operator**
+(`mayNeedTextWrapping` in `src/utils/search.ts`), and only when the portal needs it.
+
+`probePortalParser()` decides that per portal: it picks two terms that occur in the
+catalog — single-word tag facets between 0.5% and 30% of the catalog, falling back to
+frequent title words — and compares `A`, `B`, `A OR B` and `text:(A OR B)`. An `A OR B`
+returning fewer hits than `A` or `B` alone is not being honoured; the wrapper is the answer
+only if the wrapped form returns more. On `data.stadt-zuerich.ch` the `text` field returns 0
+for every query, so the probe correctly declines to wrap there.
+
+Cost: nothing for a plain query, 5 extra `rows=0` calls the first time a boolean query hits
+a portal in a session, nothing afterwards (cached, negative verdicts included).
+
 ### Adding a New Portal
 
 1. Add entry to `portals` array
 2. Set `id`, `name`, and `api_url` (required)
 3. Add `api_url_aliases` if the portal has multiple URL variants
 4. Set `api_path` if the portal uses non-standard API path (e.g., `/api/action/` instead of `/api/3/action/`)
 5. Customize `dataset_view_url` and/or `organization_view_url` only if non-standard
-6. Set `search.force_text_field: true` if the portal has query parser issues
+6. Leave `search.force_text_field` unset: the runtime probe decides
 7. Set `normalize: "multilingual"` if the portal uses multilingual/DCAT-AP field structures
 
 **Note**: To determine the correct `api_path`, test the portal's API endpoints:
@@ -158,7 +190,7 @@ The following portals have been tested and verified (as of v0.4.37):
 
 | Portal | Country | CKAN Version | Notes |
 |--------|---------|--------------|-------|
-| dati.gov.it/opendata | 🇮🇹 Italy | 2.10.3 | `force_text_field: true`; custom `dataset_view_url` and `organization_view_url` |
+| dati.gov.it/opendata | 🇮🇹 Italy | 2.10.3 | Custom `dataset_view_url` and `organization_view_url` |
 | dati.anticorruzione.it/opendata | 🇮🇹 Italy | — | Standard configuration |
 | catalog.data.gov | 🇺🇸 USA | 2.11.4 | Standard configuration |
 | open.canada.ca/data | 🇨🇦 Canada | 2.10.8 | Standard configuration |

src/portals.json

@@ -11,9 +11,6 @@
         "https://dati.gov.it",
         "https://www.dati.gov.it"
       ],
-      "search": {
-        "force_text_field": true
-      },
       "hvd": {
         "category_field": "hvd_category"
       },
@@ -30,10 +27,7 @@
       "api_url": "https://dati.arpae.it",
       "api_url_aliases": [
         "http://dati.arpae.it"
-      ],
-      "search": {
-        "force_text_field": true
-      }
+      ]
     },
     {
       "id": "anac-opendata",
@@ -55,9 +49,6 @@
         "http://www.data.gov.uk"
       ],
       "api_path": "/api/action",
-      "search": {
-        "force_text_field": true
-      },
       "dataset_view_url": "https://www.data.gov.uk/dataset/{id}/{name}",
       "organization_view_url": "https://www.data.gov.uk/organization/{name}"
     },
@@ -67,10 +58,7 @@
       "api_url": "https://catalog.data.gov",
       "api_url_aliases": [
         "http://catalog.data.gov"
-      ],
-      "search": {
-        "force_text_field": true
-      }
+      ]
     },
     {
       "id": "open-canada",
@@ -80,10 +68,7 @@
         "http://open.canada.ca/data",
         "https://open.canada.ca",
         "http://open.canada.ca"
-      ],
-      "search": {
-        "force_text_field": true
-      }
+      ]
     },
     {
       "id": "data-gov-au",
@@ -126,21 +111,15 @@
       "api_url": "https://data.gov.ua",
       "api_url_aliases": [
         "http://data.gov.ua"
-      ],
-      "search": {
-        "force_text_field": false
-      }
+      ]
     },
     {
       "id": "dati-regione-sicilia",
       "name": "dati.regione.sicilia.it",
       "api_url": "https://dati.regione.sicilia.it",
       "api_url_aliases": [
         "http://dati.regione.sicilia.it"
-      ],
-      "search": {
-        "force_text_field": false
-      }
+      ]
     },
     {
       "id": "bdap-rgs-mef",
@@ -169,9 +148,6 @@
   ],
   "defaults": {
     "dataset_view_url": "{server_url}/dataset/{name}",
-    "organization_view_url": "{server_url}/organization/{name}",
-    "search": {
-      "force_text_field": false
-    }
+    "organization_view_url": "{server_url}/organization/{name}"
   }
 }

src/tools/package.ts

@@ -7,8 +7,8 @@ import { ResponseFormat, ResponseFormatSchema, CkanTag, CkanResource, CkanPackag
 import { makeCkanRequest, formatCkanError } from "../utils/http.js";
 import { truncateText, truncateJson, formatDate, formatBytes, addDemoFooter, wrapUntrusted, safeUrlText, formatError, jsonToolResult, sanitizeInline } from "../utils/formatting.js";
 import { getDatasetViewUrl, extractSourcePortal } from "../utils/url-generator.js";
-import { resolveSearchQuery, stripAccents, hasAccents, isPlainMultiTermQuery, buildOrQuery } from "../utils/search.js";
-import { getPortalHvdConfig, getPortalApiPath, requiresMultilingualNormalization, isPortalSearchExplicitlyConfigured } from "../utils/portal-config.js";
+import { resolveSearchQuery, stripAccents, hasAccents, isPlainMultiTermQuery, buildOrQuery, mayNeedTextWrapping } from "../utils/search.js";
+import { getPortalHvdConfig, getPortalApiPath, requiresMultilingualNormalization } from "../utils/portal-config.js";
 import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 
 /**
@@ -19,25 +19,93 @@ import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 const _portalParserCache = new Map<string, boolean>();
 
 /**
- * Probe a portal to detect whether it needs force_text_field.
- * Runs two parallel rows=0 queries (default vs text parser) using "data OR dati".
- * If text count > default count * 2, the portal has the Solr df bug and needs wrapping.
- * Result is cached for the session lifetime.
+ * Pick two terms that actually occur in this catalog, for the parser probe below.
+ *
+ * They must be single words (a multi-word term would need quoting and change the
+ * parse) and neither rare nor saturating: a term matching most of the catalog makes
+ * `A OR B` indistinguishable from `A`, which is how the previous probe — hardcoded to
+ * "data OR dati" — read dati.comune.milano.it as healthy while `aria OR acqua` there
+ * returned 0 against 54 and 33 for the single terms.
+ *
+ * Tag facets first, since tags are in the catalog's own language; frequent title words
+ * as a fallback for portals that expose no tag facets (open.canada.ca) or too few
+ * (dati.regione.sicilia.it).
+ */
+async function pickProbeTerms(serverUrl: string): Promise<[string, string] | null> {
+  const facetRes = await makeCkanRequest<any>(serverUrl, 'package_search', {
+    q: '*:*',
+    rows: 0,
+    'facet.field': '["tags"]',
+    'facet.limit': 100
+  }).catch(() => null);
+
+  const total: number = facetRes?.count ?? 0;
+  if (!total) return null;
+
+  const items: Array<{ name?: string; count?: number }> =
+    facetRes?.search_facets?.tags?.items ?? [];
+  const usable = items
+    .filter(i => typeof i.name === 'string' && !/\s/.test(i.name))
+    .filter(i => (i.count ?? 0) >= total * 0.005 && (i.count ?? 0) <= total * 0.3)
+    .sort((a, b) => (b.count ?? 0) - (a.count ?? 0));
+  if (usable.length >= 2) return [usable[0].name!, usable[1].name!];
+
+  const sampleRes = await makeCkanRequest<any>(serverUrl, 'package_search', {
+    q: '*:*',
+    rows: 25
+  }).catch(() => null);
+  const titles: string[] = (sampleRes?.results ?? []).map((r: any) => r?.title ?? '');
+  const freq = new Map<string, number>();
+  for (const word of titles.join(' ').toLowerCase().split(/[^\p{L}]+/u)) {
+    if (word.length > 4) freq.set(word, (freq.get(word) ?? 0) + 1);
+  }
+  const top = [...freq.entries()].sort((a, b) => b[1] - a[1]).slice(0, 2);
+  return top.length === 2 ? [top[0][0], top[1][0]] : null;
+}
+
+/**
+ * Does this portal need `text:(...)` wrapping to honour a boolean query?
+ *
+ * `package_search` hands a colon-free query to Solr's dismax parser with `q.op=AND`
+ * (ckan/lib/search/query.py), and dismax has no boolean syntax: `A OR B` collapses to
+ * `A AND B`. A colon takes the query off dismax, which is what the wrapper exploits.
+ * That is CKAN's own default, so most portals need it — but not all: on
+ * data.stadt-zuerich.ch the catch-all `text` field returns 0 for every query, and
+ * wrapping there loses everything.
+ *
+ * So: an `A OR B` that returns fewer hits than `A` or `B` alone is not being honoured,
+ * and the wrapper is the fix only if the wrapped form actually returns more.
+ * Four rows=0 counts, cached per portal for the session, negative verdicts included.
+ * Callers must only reach here for queries that carry a boolean operator — nothing
+ * else is ever wrapped, so nothing else needs to pay for this.
  */
 async function probePortalParser(serverUrl: string): Promise<boolean> {
   const key = serverUrl.replace(/\/$/, '').toLowerCase();
   if (_portalParserCache.has(key)) return _portalParserCache.get(key)!;
 
-  const probe = 'data OR dati';
-  const [defaultRes, textRes] = await Promise.allSettled([
-    makeCkanRequest<any>(serverUrl, 'package_search', { q: probe, rows: 0 }),
-    makeCkanRequest<any>(serverUrl, 'package_search', { q: `text:(${probe})`, rows: 0 })
-  ]);
-
-  const defaultCount = defaultRes.status === 'fulfilled' ? (defaultRes.value.count ?? 0) : 0;
-  const textCount = textRes.status === 'fulfilled' ? (textRes.value.count ?? 0) : 0;
+  let needsText = false;
+  const terms = await pickProbeTerms(serverUrl).catch(() => null);
+
+  if (terms) {
+    const [a, b] = terms;
+    const count = async (q: string): Promise<number | null> => {
+      const res = await makeCkanRequest<any>(serverUrl, 'package_search', { q, rows: 0 })
+        .catch(() => null);
+      return typeof res?.count === 'number' ? res.count : null;
+    };
+    const [ca, cb, cOr, cText] = await Promise.all([
+      count(a),
+      count(b),
+      count(`${a} OR ${b}`),
+      count(`text:(${a} OR ${b})`)
+    ]);
+
+    if (ca !== null && cb !== null && cOr !== null && cText !== null) {
+      const booleanIgnored = cOr < Math.max(ca, cb);
+      needsText = booleanIgnored && cText > cOr;
+    }
+  }
 
-  const needsText = textCount > 0 && (defaultCount === 0 || textCount > defaultCount * 2);
   _portalParserCache.set(key, needsText);
   return needsText;
 }
@@ -773,10 +841,11 @@ Typical workflow: ckan_status_show (check locale) → ckan_package_search (query
           if (!effectiveSort) effectiveSort = "issued desc, metadata_created desc";
         }
 
-        // For portals not explicitly configured in portals.json, auto-detect
-        // whether they need text:(...) wrapping by probing with a two-term OR query.
+        // Only a boolean query can benefit from text:(...) wrapping, so only a boolean
+        // query pays for the probe. Every portal is probed, configured ones included:
+        // the values that used to live in portals.json went stale.
         let parserOverride = params.query_parser;
-        if (!parserOverride && !isPortalSearchExplicitlyConfigured(params.server_url)) {
+        if (!parserOverride && mayNeedTextWrapping(query)) {
           const needsText = await probePortalParser(params.server_url);
           if (needsText) parserOverride = "text";
         }
@@ -922,7 +991,10 @@ ${hvdNote}`;
           markdown += `No datasets found matching your query.\n`;
           markdown += `\n> **Note**: No data was found on this portal. Do not use information from other sources to supplement this result.\n`;
           if (isPlainMultiTermQuery(params.q)) {
-            markdown += `\n> **Tip**: Multi-term queries use AND by default (all terms must match). Try OR to broaden the search:\n`;
+            // CKAN's dismax applies mm='2<-1 5<80%', so a plain multi-term query is
+            // already a partial match: spelling out OR relaxes it the rest of the way
+            // and, on portals that ignore boolean operators, switches parser too.
+            markdown += `\n> **Tip**: With several terms the portal requires most of them to match. Spelling out OR broadens the search:\n`;
             markdown += `> \`q: "${buildOrQuery(params.q)}"\`\n`;
           }
         }