Skip to content

perf(search): use one postings path without hidden ranking retries #597

Description

@eric-tramel

Summary

Interactive search exposes two production strategies, and MCP may rerun postings ranking up to 16 times when deduplication removes candidates. That hidden refill loop multiplies the expensive part of one request and eventually reports a generic “narrow the query” error.

Consolidate production search on search_postings, remove hidden fallback/refill reruns, deduplicate before applying the result limit, and read full content only for selected events.

“One search path” describes the work, not a stopwatch. A broad query may take time and should finish while it stays within #644's resource limits and its caller is still waiting.

Scope

  • Remove optimized versus oracle_exact from the public runtime API; retain an exact oracle only in tests/benchmarks if useful.
  • Use one shared search_postings implementation for interactive event-text search.
  • Apply scope and filters during ranking so excluded rows do not affect scores or hydration.
  • Do not keep complete common-term posting lists in an unbounded Rust cache.
  • Do not revisit postings through empty-result fallback, dedup refill, or larger-offset passes.
  • Apply existing duplicate rules before the final top-K limit.
  • Distinguish normal top-K truncation from ranking stopped by resource exhaustion.
  • Hydrate text/payload only for selected canonical event versions.

Acceptance criteria

  • Public interactive APIs expose one production postings strategy.
  • One logical search does not restart ranking for fallback, refill, or a larger offset; any chunks are non-overlapping.
  • A duplicate-heavy fixture returns the correct K unique results without a fixed candidate allowance or generic “narrow the query” error.
  • Common-term search does not keep complete posting lists in an unbounded Rust cache, scan transcript text for candidates, or hydrate non-selected events.
  • Responses distinguish normal result-limit truncation from incomplete/resource-limited ranking.
  • Existing relevance, scope, filtering, ordering, deduplication, freshness, and full-payload fixtures pass.
  • A slow search can exceed former Moraine defaults and complete while its caller remains connected.

Chain

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical priorityarea/clickhouseClickHouse integrationarea/searchSearch indexing and retrievalbugSomething isn't workingstatus/merge-trainAutomation lock label for merge-train repair work

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions