fix(search): relevance scoring, shared parser probe and accent-safe filters

LOG.md

@@ -2,6 +2,59 @@
 
 ## 2026-09-05
 
+### Relevance scoring: three defects that only became visible once search worked
+
+Testing v0.4.122 through an MCP client, not curl, showed `ckan_find_relevant_datasets`
+answering its own documented example badly. `defibrillatori Comune di Lecce` on
+dati.gov.it used to return the catalog's only Lecce dataset — because the search
+returned exactly one result. With recall restored it returns 679, and the top three
+became Trento, Martina Franca and Desio while the Lecce dataset fell out of the window.
+
+The wrapping fix did not cause this; it removed the cover. Three defects, all older:
+
+- `scoreTextField` awarded the **whole** field weight when **any** query term matched.
+  "Comune di Martina Franca" and "Comune di Lecce" both scored a full holder match, so
+  the right dataset could not outrank the wrong ones. It now scores the share of terms
+  the field carries.
+- the stopword list was English-only, so `di` counted as a term: "Provincia Autonoma
+  **di** Trento" earned a full holder match on a query asking for Lecce. Italian
+  stopwords added.
+- `\b` is ASCII-only in JavaScript, so a term ending in an accented letter never found
+  its word boundary: `mobilità` scored 0 against "mobilità urbana", `qualità` against
+  "qualità dell'aria". On a catalog that is mostly not in English this silently sank
+  every accented query. Replaced with Unicode lookarounds.
+
+Looking for more of the same family turned up three more, all cases of a local filter
+running over a truncated or wrongly-normalised set:
+
+- `ckan_find_relevant_datasets` never called the parser probe. `portals.json` used to
+  cover it; removing `force_text_field` left it sending boolean queries to the parser that
+  ignores them. On dati.comune.milano.it `aria OR acqua` returned 0 there against 87 from
+  `ckan_package_search`. Same probe now applies to both.
+- `ckan_organization_search` builds a Solr wildcard, which bypasses the analysis chain, so
+  the pattern must be pre-normalised the way CKAN builds a name slug. It lowercased but did
+  not fold accents: `città` returned 0 while `citta` matched 135 datasets.
+- `ckan_tag_list` applied `tag_query` after faceting, with `facet.limit` set to the
+  caller's `limit`. On dati.gov.it 53 tags contain "citta" and none is in the top 100, so
+  the filter answered "no tags" while they existed. The facet is now widened when a filter
+  is given.
+
+`openspec/specs/ckan-search/spec.md` described the parser as a property of
+`ckan_package_search` alone, which is what let the second caller go unnoticed — and after
+yesterday it was also wrong, still describing the per-portal default that was removed.
+Rewritten as a property of the query-building path, naming every tool that shares it.
+
+Also raised the candidate window to at least 50: the local ranking only sees what Solr
+returns first, and `limit: 3` shrank it to 15 — enough when a search returned a handful
+of results, not enough now.
+
+`defibrillatori Comune di Lecce` puts the Lecce dataset first again. 532 tests, 5 added.
+
+How it was missed: yesterday's verification checked result **counts** through
+`ckan_package_search`, never the ranked output of `ckan_find_relevant_datasets` — the
+third most used tool in the telemetry, and the second caller of `resolveSearchQuery`.
+Counting results proves recall, not usefulness.
+
 ### v0.4.122 - Solr parser fix
 
 Ships #534: the `text:(...)` wrapper is reserved for the queries dismax cannot serve, the escaping preserves unary operators and balanced grouping, and the parser probe measures two terms taken from the catalog on every portal. `force_text_field` is gone from `portals.json`.

openspec/specs/ckan-search/spec.md

@@ -1,18 +1,69 @@
 # ckan-search Specification
 
 ## Purpose
-TBD - created by archiving change update-search-parser-config. Update Purpose after archive.
-## Requirements
-### Requirement: Package search parser override
-The system SHALL support a per-portal default and a per-request override to force package search queries through the `text` field when needed, and SHALL escape Solr/Lucene special characters when wrapping queries in `text:(...)`.
+How this server turns a caller's query into a CKAN `package_search` request: which Solr
+parser it reaches, how the query is escaped, and how results are ranked.
 
-#### Scenario: Portal default applies
-- **WHEN** a portal is configured to force the text-field parser
-- **THEN** `ckan_package_search` uses `text:(...)` for non-fielded queries by default with escaped query content
+## Requirements
+### Requirement: Solr parser selection
+
+Every tool that builds a Solr query for `package_search` SHALL resolve the parser through
+`resolveSearchQuery`, and SHALL use the same portal probe when the query may need wrapping.
+This is a property of the query-building path, not of one tool: today the path is shared by
+`ckan_package_search` and `ckan_find_relevant_datasets`, any tool added later belongs on
+that list, and a change touching the parser SHALL be verified against every tool on it.
+
+Background: CKAN sends a colon-free query to Solr's dismax parser with `q.op=AND`,
+`mm='2<-1 5<80%'` and `qf='name^4 title^4 tags^2 groups^2 text'`. dismax has no boolean
+syntax, so `A OR B` collapses into `A AND B`; a colon takes the query off dismax, which is
+what `text:(...)` exploits. The same switch discards the `qf` boosts and ANDs every term on
+one field, so the wrapper helps a boolean query and harms every other shape.
+
+#### Scenario: Boolean query on a portal that ignores booleans
+- **WHEN** a query carries `AND`, `OR` or `NOT`, or punctuation inside a word, and the
+  portal probe finds the default parser does not honour a disjunction
+- **THEN** the query is wrapped in `text:(...)` with its content escaped
+- **AND** the wrapper is applied identically by `ckan_package_search` and
+  `ckan_find_relevant_datasets`
+
+#### Scenario: Plain keyword query
+- **WHEN** a query carries no boolean operator, which is the shape an LLM client generates
+  from a user's request
+- **THEN** the query reaches the portal's own parser unwrapped, keeping the `qf` boosts
+  and `mm`, and no probe is issued
+
+#### Scenario: Unary operator
+- **WHEN** a query carries a `+`, `-` or `!` in operator position
+- **THEN** the character survives the escaping, because dismax honours it natively and
+  escaping it inverts the caller's intent
 
 #### Scenario: Request override applies
 - **WHEN** a client explicitly requests the text-field parser
-- **THEN** `ckan_package_search` uses `text:(...)` regardless of portal defaults with escaped query content
+- **THEN** the wrapper is applied regardless of what the probe found
+
+#### Scenario: Portal where the wrapper does not work
+- **WHEN** the probe finds the wrapped form returns fewer results than the plain one
+- **THEN** no wrapping is applied on that portal, and the verdict is cached only if it was
+  actually measured
+
+### Requirement: Relevance ranking
+
+`ckan_find_relevant_datasets` ranks locally over the candidates `package_search` returns
+first, so its answer depends on both the recall of the query and the size of the candidate
+window. The tool SHALL score a field by the share of query terms it carries, SHALL match
+terms on Unicode word boundaries, and SHALL score at least 50 candidates whatever the
+requested limit.
+
+#### Scenario: Field scoring
+- **WHEN** a field contains some of the query's terms
+- **THEN** it scores in proportion to the share it carries, never the full weight for a
+  single term
+
+#### Scenario: Non-English text
+- **WHEN** a query term ends in an accented letter, or is a stopword of the catalog's
+  language
+- **THEN** term matching respects Unicode word boundaries, and the stopword does not
+  contribute to any field's score
 
 ### Requirement: List Dataset Resources
 

src/tools/organization.ts

@@ -7,6 +7,7 @@ import { ResponseFormat, ResponseFormatSchema, CkanOrganization } from "../types
 import { makeCkanRequest, formatCkanError, CkanApiError } from "../utils/http.js";
 import { truncateText, formatDate, addDemoFooter, wrapUntrusted, formatError, jsonToolResult, sanitizeInline } from "../utils/formatting.js";
 import { getOrganizationViewUrl } from "../utils/url-generator.js";
+import { stripAccents } from "../utils/search.js";
 import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 
 type OrgFacetItem = { name: string; display_name?: string; count: number };
@@ -374,8 +375,11 @@ Typical workflow: ckan_organization_search → ckan_organization_show (get detai
     },
     async (params) => {
       try {
-        // Build Solr query with wildcards (lowercase: Solr org names are always lowercase)
-        const query = `organization:*${params.pattern.toLowerCase()}*`;
+        // Build Solr query with wildcards. A wildcard term bypasses Solr's analysis
+        // chain, so the pattern has to be pre-normalised the way CKAN builds the name
+        // slug: lowercase and without accents. Otherwise `città` finds nothing while
+        // `citta` finds 135 organizations, `citta-metropolitana-di-*` among them.
+        const query = `organization:*${stripAccents(params.pattern.toLowerCase())}*`;
 
         // Search using package_search with faceting
         const result = await makeCkanRequest<any>(

src/tools/package.ts

@@ -175,6 +175,12 @@ const DEFAULT_RELEVANCE_WEIGHTS: RelevanceWeights = {
 };
 
 const QUERY_STOPWORDS = new Set([
+  // Italian: without these, `defibrillatori Comune di Lecce` scored a full holder
+  // match against "Provincia Autonoma di Trento" on the strength of "di" alone.
+  "di", "del", "dello", "della", "dei", "degli", "delle",
+  "il", "lo", "la", "i", "gli", "le", "un", "uno", "una",
+  "e", "ed", "per", "con", "su", "da", "dal", "dalla", "nel", "nella", "al", "alla",
+  "che", "non", "come", "dove", "sono",
   "a",
   "an",
   "the",
@@ -219,21 +225,62 @@ const QUERY_STOPWORDS = new Set([
 ]);
 
 export const extractQueryTerms = (query: string): string[] => {
-  const matches = query.toLowerCase().match(/[\p{L}\p{N}]+/gu) ?? [];
-  const terms = matches.filter((term) => term.length > 1 && !QUERY_STOPWORDS.has(term));
+  const raw = query.normalize("NFC").match(/[\p{L}\p{N}]+/gu) ?? [];
+  // An all-caps token is an acronym, not an article: the stopword list is there for
+  // `defibrillatori Comune di Lecce`, and must not swallow the `UN` of `UN population`
+  // on a catalog in another language.
+  const terms = raw
+    .filter((token) => {
+      const term = token.toLowerCase();
+      if (term.length <= 1) return false;
+      if (!QUERY_STOPWORDS.has(term)) return true;
+      return token.length > 1 && token === token.toUpperCase() && token !== term;
+    })
+    .map((token) => token.toLowerCase());
   return Array.from(new Set(terms));
 };
 
 export const escapeRegExp = (value: string): string => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
 
+/**
+ * Word-boundary matcher for a single term.
+ *
+ * `\b` is ASCII-only in JavaScript, so an accented letter counts as a non-word
+ * character and a term ending in one never finds its boundary: `mobilità` failed to
+ * match "mobilità urbana". Unicode lookarounds fix it, which matters on catalogs that
+ * are mostly not in English.
+ */
+const termPattern = (term: string): RegExp =>
+  new RegExp(`(?<![\\p{L}\\p{N}])${escapeRegExp(term.normalize("NFC"))}(?![\\p{L}\\p{N}])`, "iu");
+
+/** Same Unicode form on both sides: `mobilità` written as NFD would not match NFC. */
+const normalizeForMatch = (text: string): string =>
+  text.normalize("NFC").toLowerCase().replace(/_/g, " ");
+
 export const textMatchesTerms = (text: string | undefined, terms: string[]): boolean => {
   if (!text || terms.length === 0) return false;
-  const normalized = text.toLowerCase().replace(/_/g, " ");
-  return terms.some((term) => new RegExp(`\\b${escapeRegExp(term)}\\b`, "i").test(normalized));
+  const normalized = normalizeForMatch(text);
+  return terms.some((term) => termPattern(term).test(normalized));
+};
+
+/** How many of the query's terms this text contains. */
+export const countMatchingTerms = (text: string | undefined, terms: string[]): number => {
+  if (!text || terms.length === 0) return 0;
+  const normalized = normalizeForMatch(text);
+  return terms.filter((term) => termPattern(term).test(normalized)).length;
 };
 
+/**
+ * Score a field by the share of query terms it carries, not by whether any of them
+ * appears. With the all-or-nothing rule a single common word earned the whole field:
+ * on `defibrillatori Comune di Lecce`, "Comune di Martina Franca" and "Comune di Lecce"
+ * both scored a full holder match, so the catalog's only Lecce dataset could not
+ * outrank the others and fell out of the top results.
+ */
 export const scoreTextField = (text: string | undefined, terms: string[], weight: number): number => {
-  return textMatchesTerms(text, terms) ? weight : 0;
+  const matched = countMatchingTerms(text, terms);
+  if (matched === 0) return 0;
+  return Math.round((weight * matched / terms.length) * 10) / 10;
 };
 
 /**
@@ -1128,11 +1175,25 @@ Typical workflow: ckan_find_relevant_datasets → ckan_package_show (inspect top
           ...(params.weights ?? {})
         };
 
-        const rows = Math.min(Math.max(params.limit * 5, params.limit), 100);
+        // At least 50 candidates to score: the local ranking only sees what Solr
+        // returns first, and a small limit used to shrink the window to 15 — enough
+        // when a search returned a handful of results, not enough now that it returns
+        // hundreds.
+        const rows = Math.min(Math.max(params.limit * 5, 50), 100);
+
+        // Same probe as ckan_package_search: without it this tool sends a boolean
+        // query to the parser that ignores booleans. On dati.comune.milano.it
+        // `aria OR acqua` returned 0 here against 87 there.
+        let parserOverride = params.query_parser;
+        if (!parserOverride && mayNeedTextWrapping(params.query)) {
+          const needsText = await probePortalParser(params.server_url);
+          if (needsText) parserOverride = "text";
+        }
+
         const { effectiveQuery } = resolveSearchQuery(
           params.server_url,
           params.query,
-          params.query_parser
+          parserOverride
         );
 
         const searchResult = await makeCkanRequest<any>(

src/tools/tag.ts

@@ -7,6 +7,7 @@ import { ResponseFormat, ResponseFormatSchema } from "../types.js";
 import { makeCkanRequest } from "../utils/http.js";
 import { truncateText, addDemoFooter, formatError, jsonToolResult, sanitizeInline } from "../utils/formatting.js";
 import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { stripAccents } from "../utils/search.js";
 
 type TagItem = {
   name: string;
@@ -91,7 +92,13 @@ Typical workflow: ckan_tag_list → ckan_package_search with fq="tags:tag_name"
           q: params.q,
           rows: 0,
           'facet.field': JSON.stringify(['tags']),
-          'facet.limit': params.limit
+          // The filter runs on whatever the facet returned, so with a tag_query the
+          // facet has to be wider than the caller's limit: on dati.gov.it 53 tags
+          // contain "citta" and none is in the top 100, so asking for 100 answered
+          // "no tags" while they existed. CKAN rejects Solr's own `facet.contains`
+          // ("Invalid search parameters"), so the filtering cannot move server-side;
+          // this window is the compromise, and WIDE_FACET below is the escape hatch.
+          'facet.limit': params.tag_query ? Math.max(params.limit * 20, 500) : params.limit
         };
 
         if (params.fq) apiParams.fq = params.fq;
@@ -105,8 +112,32 @@ Typical workflow: ckan_tag_list → ckan_package_search with fq="tags:tag_name"
         let tags = normalizeTagFacets(result);
 
         if (params.tag_query) {
-          const needle = params.tag_query.toLowerCase();
-          tags = tags.filter(tag => tag.name.toLowerCase().includes(needle));
+          // Both sides accent-folded: CKAN builds tag names as slugs, so `città`
+          // would never match `citta-metropolitana` — while portals that do keep
+          // accented tags still match either spelling.
+          const needle = stripAccents(params.tag_query.toLowerCase());
+          const matching = (candidates: TagItem[]) =>
+            candidates.filter(tag => stripAccents(tag.name.toLowerCase()).includes(needle));
+
+          tags = matching(tags);
+
+          // Only a filter that came up short pays for the whole facet: on
+          // dati.gov.it that is 14138 tags and 1.4 MB, against ~50 KB for the
+          // bounded window above.
+          if (tags.length < params.limit) {
+            const wide = await makeCkanRequest<any>(
+              params.server_url,
+              'package_search',
+              { ...apiParams, 'facet.limit': -1 }
+            ).catch(() => null);
+            if (wide) tags = matching(normalizeTagFacets(wide));
+          }
+
+          // Solr sorts by count only while facet.limit is positive: asked for -1 it
+          // returns the tags in index order, so the widened set arrives alphabetical
+          // ("zuglio", "zucs", ...). CKAN rejects `facet.sort` too, so the ordering
+          // the tool documents has to be restored here.
+          tags = tags.sort((a, b) => b.count - a.count).slice(0, params.limit);
         }
 
         tags = tags