fix(search): wrap only boolean queries, probe every portal for the verdict

LOG.md

@@ -2,6 +2,52 @@
 
 ## 2026-09-05
 
+### Solr: wrap only boolean queries, and measure every portal instead of storing a verdict
+
+The usage review surfaced an LLM client reformulating the same request against
+`www.dati.gov.it/opendata` for three hours on 29 July. The data was there from the first
+attempt: `bonifica siti contaminati Piemonte` returns 22 datasets on that portal, the
+Piedmont contaminated-sites registry first. Our `text:(...)` rewrite was returning 5.
+
+`package_search` hands a colon-free query to Solr's dismax parser with `q.op=AND`,
+`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'` (`ckan/lib/search/query.py`).
+dismax has no boolean syntax, so `A OR B` collapses into `A AND B`: on dati.gov.it
+`aria OR Milano` returns 59, exactly what `aria AND Milano` returns. A colon takes the query
+off dismax, which is what the wrapper exploits — the wrapped form returns 3421. The same
+switch is why it hurts everything else: it drops the `qf` boosts that rank titles and tags
+first, searches the catch-all `text` field alone, and ANDs every term instead of applying
+`mm`. This is CKAN's own default, not a per-portal defect, which is why the same pattern
+showed up on Milano, Toscana, Sicilia, Ucraina and open.canada.ca.
+
+Measured on dati.gov.it: `defibrillatori Comune di Lecce` 678 -> 1, `qualità aria Milano`
+650 -> 51, `musei roma arte opere catalogo` 9 -> 0 — the second most repeated query of the
+semester, answered with an empty page while nine datasets existed. On 40 real queries from
+telemetry the separation is clean: with a boolean operator the wrapper helped 8 times and
+hurt none; without one it helped none and hurt 10, six of them down to zero.
+
+So the wrapper now applies only to queries carrying a boolean operator. The shape that an
+LLM client generates from a user's request — a run of keywords, no operators — is left to
+the portal's own parser, boosts and `mm` included.
+
+`probePortalParser` was rewritten and now runs for every portal, configured ones included;
+`force_text_field` is gone from `portals.json` (8 entries) so there is one source of truth.
+The old probe asked `data OR dati`, two words common enough to saturate: on Milano `data`
+alone and `data OR dati` both return 2564, so it read the portal as healthy while
+`aria OR acqua` returned 0 against 54 and 33 for the single terms. It now picks two terms
+from the catalog itself — single-word tag facets between 0.5% and 30%, falling back to
+frequent title words — and compares `A`, `B`, `A OR B`, `text:(A OR B)`. An OR returning
+fewer hits than either operand is not being honoured, and the wrapper is the answer only if
+the wrapped form returns more. `data.stadt-zuerich.ch`, where the `text` field returns 0 for
+every query, is correctly left alone.
+
+Cost: a plain query pays nothing, since nothing but a boolean query can be wrapped. The
+probe costs 5 extra `rows=0` calls the first time a boolean query reaches a portal in a
+session, then nothing.
+
+Verified e2e against live portals: dati.gov.it 5 -> 22 and 0 -> 9 on the two queries from
+the telemetry, Milano `aria OR acqua` 0 -> 87, Zurich unchanged at 10 and 172. 516 tests
+pass, 18 added.
+
 ### v0.4.121 - clearer 404 on datastore_search_sql
 
 Patch release for the fix in #532: a 404 on `datastore_search_sql` now says the portal does not expose the SQL endpoint, instead of sending the caller to `ckan_package_show` for a resource_id that was already valid. Also ships the telemetry pipeline fixes and the DEPLOYMENT.md/CLAUDE.md realignment from earlier today.

src/README.md

@@ -53,10 +53,10 @@ Each entry in `portals` array supports:
   - Example: data.europa.eu requires this due to its DCAT-AP based field structure
 
 - **`search`** (object): Search behavior configuration
-  - **`force_text_field`** (boolean): Force wrapping non-fielded queries in `text:(...)`
-    - Default: `false`
-    - Set to `true` for portals with restrictive query parsers that break on long OR queries
-    - Example: dati.gov.it requires this to handle queries like `"hotel OR alberghi"`
+  - **`force_text_field`** (boolean): forces `text:(...)` wrapping on non-fielded queries.
+    Still honoured if present, but **no portal sets it any more** and new ones should not:
+    the decision is measured at runtime by `probePortalParser()` in `src/tools/package.ts`.
+    The stored values went stale and were removed on 2026-09-05.
 
 ### Defaults
 
@@ -72,14 +72,46 @@ The `defaults` object provides fallback values when a portal is not found in the
 }
 ```
 
+### Query parser: when `text:(...)` wrapping is applied
+
+`package_search` sends a colon-free query to Solr's **dismax** parser with `q.op=AND`,
+`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'` (`ckan/lib/search/query.py`).
+dismax has no boolean syntax, so `A OR B` collapses into `A AND B`. A colon in the query
+takes it off dismax, which is what wrapping it as `text:(A OR B)` exploits.
+
+The same switch is why wrapping hurts everything else: it drops the `qf` boosts that rank
+titles and tags first, searches the catch-all `text` field alone, and ANDs every term
+instead of applying `mm`. Measured on dati.gov.it, 2026-09-05:
+
+| query | plain | `text:(...)` |
+|---|---|---|
+| `ambiente` | 8047 | 8047 |
+| `qualità aria Milano` | 650 | 51 |
+| `defibrillatori Comune di Lecce` | 678 | 1 |
+| `musei roma arte opere catalogo` | 9 | 0 |
+| `aria OR Milano` | 59 | 3421 |
+
+So the wrapper is applied **only to queries carrying a boolean operator**
+(`mayNeedTextWrapping` in `src/utils/search.ts`), and only when the portal needs it.
+
+`probePortalParser()` decides that per portal: it picks two terms that occur in the
+catalog — single-word tag facets between 0.5% and 30% of the catalog, falling back to
+frequent title words — and compares `A`, `B`, `A OR B` and `text:(A OR B)`. An `A OR B`
+returning fewer hits than `A` or `B` alone is not being honoured; the wrapper is the answer
+only if the wrapped form returns more. On `data.stadt-zuerich.ch` the `text` field returns 0
+for every query, so the probe correctly declines to wrap there.
+
+Cost: nothing for a plain query, 5 extra `rows=0` calls the first time a boolean query hits
+a portal in a session, nothing afterwards (cached, negative verdicts included).
+
 ### Adding a New Portal
 
 1. Add entry to `portals` array
 2. Set `id`, `name`, and `api_url` (required)
 3. Add `api_url_aliases` if the portal has multiple URL variants
 4. Set `api_path` if the portal uses non-standard API path (e.g., `/api/action/` instead of `/api/3/action/`)
 5. Customize `dataset_view_url` and/or `organization_view_url` only if non-standard
-6. Set `search.force_text_field: true` if the portal has query parser issues
+6. Leave `search.force_text_field` unset: the runtime probe decides
 7. Set `normalize: "multilingual"` if the portal uses multilingual/DCAT-AP field structures
 
 **Note**: To determine the correct `api_path`, test the portal's API endpoints:
@@ -158,7 +190,7 @@ The following portals have been tested and verified (as of v0.4.37):
 
 | Portal | Country | CKAN Version | Notes |
 |--------|---------|--------------|-------|
-| dati.gov.it/opendata | 🇮🇹 Italy | 2.10.3 | `force_text_field: true`; custom `dataset_view_url` and `organization_view_url` |
+| dati.gov.it/opendata | 🇮🇹 Italy | 2.10.3 | Custom `dataset_view_url` and `organization_view_url` |
 | dati.anticorruzione.it/opendata | 🇮🇹 Italy | — | Standard configuration |
 | catalog.data.gov | 🇺🇸 USA | 2.11.4 | Standard configuration |
 | open.canada.ca/data | 🇨🇦 Canada | 2.10.8 | Standard configuration |

src/portals.json

@@ -11,9 +11,6 @@
         "https://dati.gov.it",
         "https://www.dati.gov.it"
       ],
-      "search": {
-        "force_text_field": true
-      },
       "hvd": {
         "category_field": "hvd_category"
       },
@@ -30,10 +27,7 @@
       "api_url": "https://dati.arpae.it",
       "api_url_aliases": [
         "http://dati.arpae.it"
-      ],
-      "search": {
-        "force_text_field": true
-      }
+      ]
     },
     {
       "id": "anac-opendata",
@@ -55,9 +49,6 @@
         "http://www.data.gov.uk"
       ],
       "api_path": "/api/action",
-      "search": {
-        "force_text_field": true
-      },
       "dataset_view_url": "https://www.data.gov.uk/dataset/{id}/{name}",
       "organization_view_url": "https://www.data.gov.uk/organization/{name}"
     },
@@ -67,10 +58,7 @@
       "api_url": "https://catalog.data.gov",
       "api_url_aliases": [
         "http://catalog.data.gov"
-      ],
-      "search": {
-        "force_text_field": true
-      }
+      ]
     },
     {
       "id": "open-canada",
@@ -80,10 +68,7 @@
         "http://open.canada.ca/data",
         "https://open.canada.ca",
         "http://open.canada.ca"
-      ],
-      "search": {
-        "force_text_field": true
-      }
+      ]
     },
     {
       "id": "data-gov-au",
@@ -126,21 +111,15 @@
       "api_url": "https://data.gov.ua",
       "api_url_aliases": [
         "http://data.gov.ua"
-      ],
-      "search": {
-        "force_text_field": false
-      }
+      ]
     },
     {
       "id": "dati-regione-sicilia",
       "name": "dati.regione.sicilia.it",
       "api_url": "https://dati.regione.sicilia.it",
       "api_url_aliases": [
         "http://dati.regione.sicilia.it"
-      ],
-      "search": {
-        "force_text_field": false
-      }
+      ]
     },
     {
       "id": "bdap-rgs-mef",
@@ -169,9 +148,6 @@
   ],
   "defaults": {
     "dataset_view_url": "{server_url}/dataset/{name}",
-    "organization_view_url": "{server_url}/organization/{name}",
-    "search": {
-      "force_text_field": false
-    }
+    "organization_view_url": "{server_url}/organization/{name}"
   }
 }