fix(search): relevance scoring, shared parser probe and accent-safe filters

[aborruso]

What happened

Testing v0.4.122 through an MCP client rather than curl showed ckan_find_relevant_datasets answering its own documented example badly.

	v0.4.121	v0.4.122	this PR
total results	1	679	679
1st	Defibrillatori DAE — Comune Cardioprotetto (Lecce)	Defibrillatori (Trento)	Defibrillatori DAE — Comune Cardioprotetto (Lecce)
2nd	—	Martina Franca	Elenco patrocini Comune di Lecce 2015
3rd	—	Desio	Elenco patrocini Comune di Lecce 2016

defibrillatori Comune di Lecce is the example the tool's own description gives for holder weighting. It used to work for the wrong reason: the search returned exactly one result, so there was nothing to rank.

The wrapping fix in #534 did not cause this. It removed the cover from defects that were already there — and looking for more of the same family turned up six in total.

Relevance scoring

Any term earned the whole field. scoreTextField returned the full weight if any query term matched, so "Comune di Martina Franca" and "Comune di Lecce" both scored a full holder match of 4 and the dataset actually held by Lecce could not outrank the others. It now scores the share of terms the field carries: 2 of 3 → 2.7, 1 of 3 → 1.3.

The stopword list was English-only. di counted as a query term, so "Provincia Autonoma di Trento" earned a full holder match on a query asking for Lecce. That is why nearly every candidate scored 13.

\b is ASCII-only in JavaScript. A term ending in an accented letter never found its word boundary: mobilità scored 0 against "mobilità urbana", qualità against "qualità dell'aria". On a catalog that is mostly not in English this silently sank every accented query. Replaced with Unicode lookarounds, which still reject immobilità for mobilità.

The candidate window was too small. Now at least 50 (Math.min(Math.max(limit * 5, 50), 100)): the local ranking only scores what Solr returns first, and limit: 3 shrank the window to 15.

Three more of the same family

ckan_find_relevant_datasets never called the parser probe. portals.json used to cover it; removing force_text_field in #534 left it sending boolean queries to the parser that ignores them.

query	ckan_package_search	ckan_find_relevant_datasets before	after
aria OR acqua (Milano)	87	0	87
bandiera blu OR spiagge	7	1	7

ckan_organization_search did not fold accents. It builds a Solr wildcard, which bypasses the analysis chain, so the pattern has to be pre-normalised the way CKAN builds a name slug. It lowercased but stopped there: città returned 0 while citta matched citta-metropolitana-di-napoli and 134 others.

ckan_tag_list filtered a truncated list. tag_query is applied after faceting, with facet.limit set to the caller's limit. On dati.gov.it 53 tags contain "citta" and none is in the top 100, so the filter answered "no tags" while they existed. The facet is widened when a filter is given, and the limit applied to the filtered list.

The spec

openspec/specs/ckan-search/spec.md named ckan_package_search alone in both parser scenarios. That is what let the second caller go unnoticed: it described the parser as a property of one tool, when it is a property of the query-building path. After #534 it was also wrong, still describing the per-portal default that was removed.

Rewritten around the invariant — every tool that builds a Solr query for package_search resolves the parser through resolveSearchQuery and the same probe, today ckan_package_search and ckan_find_relevant_datasets, and a change touching the parser is verified against every tool on that list. Scenarios added for plain, boolean and unary shapes, for the portal where the wrapper does not work, and for relevance ranking. openspec validate --specs --strict passes.

Verification

• 532 tests, 5 added, all anchored to the measured cases above
• e2e through the MCP client and a local HTTP server: the Lecce dataset is first again; qualità dell'aria Milano returns air-quality monitoring datasets; the two search tools now agree on boolean queries; città finds 4 organizations and 5 tags

How it was missed

Yesterday's verification checked result counts through ckan_package_search, never the ranked output of ckan_find_relevant_datasets — the third most used tool in the telemetry, and the second caller of resolveSearchQuery, which I had listed and not exercised. Counting results proves recall, not usefulness.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MEpSWpAwuMaGkfnMpQdqK2

[greptile-apps]

Greptile Summary

This PR improves CKAN search relevance and consistency across package, organization, and tag tools.

• Scores text fields proportionally across matching query terms and handles accented text across Unicode normalization forms.
• Applies shared parser probing to both package-search paths and expands the local ranking candidate window.
• Accent-folds organization and tag filters.
• Uses bounded tag-facet retrieval first, propagates exhaustive fallback failures, and restores count-based ordering locally.
• Updates the search specification, regression tests, and change log.

Important Files Changed

Filename	Overview
src/tools/package.ts	Improves multilingual term extraction, proportional relevance scoring, Unicode matching, parser resolution, and candidate selection.
src/tools/tag.ts	Adds accent-safe filtering, bounded-first facet retrieval, explicit fallback failure propagation, and deterministic count ordering.
src/tools/organization.ts	Normalizes accented organization search patterns before constructing Solr wildcard queries.
tests/unit/package-scoring.test.ts	Adds focused regressions for stopwords, acronyms, proportional scoring, boundaries, and Unicode normalization.
openspec/specs/ckan-search/spec.md	Defines shared parser-selection and relevance-ranking invariants for package-search tools.

_{Reviews (5): Last reviewed commit: "fix(tags): let an escalation failure sur..." | Re-trigger Greptile}