Commit 041b991
committed
feat(ck-engine): report chunk-level line spans for lexical search results
Lexical (BM25) hits previously spanned the whole file (line 1 to EOF), so a
match deep in a large file reported line 1 with a first-lines preview. Results
now locate the best-matching chunk — using the same chunker the semantic path
uses — and report its real span and text, falling back to the whole-file span
when no chunk contains a query term. Document granularity, scoring, and result
order are unchanged (covered by a ranking-invariance test).
The refinement runs only after the top-k documents are selected. For each hit
the stored content is chunked (language resolved as the indexer does: PDF check,
else Language::from_path) and the winning chunk is the one with the most
whole-token matches for the query terms. The terms are taken from the parsed
tantivy query (its content-field terms), not the raw query string, so field
prefixes, quoted phrases, and boolean operators — which tantivy strips before
matching — don't leak into span selection. This consumes whatever query object
the parser produces, so it composes with lenient parsing: the span step reads
the content-field terms of the leniently-parsed query, and dropped fragments
never leak into span selection. The chunk is tokenized the same way tantivy
tokenizes (lowercase, split on non-alphanumeric) and matches are whole-token,
so a short term like "in" doesn't tally hits inside larger words. Ties favor the
earliest chunk. The tantivy document, its score, and top-k selection are
untouched, so an existing index produces identical rankings.
The default lexical preview changes from the file's first three lines to the
winning chunk's text (--full-section still returns the whole file). If the
shorter preview is preferred as the default, this is easy to gate.
Spans reflect the content the index was built from, so they are only as fresh as
the index; the corpus fingerprint triggers a rebuild on ordinary edits, but a
span can point at shifted lines after a change that preserves a file's mtime and
size.1 parent df46105 commit 041b991
1 file changed
Lines changed: 357 additions & 12 deletions
0 commit comments