Skip to content

Commit 61e44fb

Browse files
authored
feat(ck-engine): report chunk-level line spans for lexical search results (#167)
Lexical (BM25) hits previously spanned the whole file (line 1 to EOF), so a match deep in a large file reported line 1 with a first-lines preview. Results now locate the best-matching chunk — using the same chunker the semantic path uses — and report its real span and text, falling back to the whole-file span when no chunk contains a query term. Document granularity, scoring, and result order are unchanged (covered by a ranking-invariance test). The refinement runs only after the top-k documents are selected. For each hit the stored content is chunked (language resolved as the indexer does: PDF check, else Language::from_path) and the winning chunk is the one with the most whole-token matches for the query terms. The terms are taken from the parsed tantivy query (its content-field terms), not the raw query string, so field prefixes, quoted phrases, and boolean operators — which tantivy strips before matching — don't leak into span selection. This consumes whatever query object the parser produces, so it composes with lenient parsing: the span step reads the content-field terms of the leniently-parsed query, and dropped fragments never leak into span selection. The chunk is tokenized the same way tantivy tokenizes (lowercase, split on non-alphanumeric) and matches are whole-token, so a short term like "in" doesn't tally hits inside larger words. Ties favor the earliest chunk. The tantivy document, its score, and top-k selection are untouched, so an existing index produces identical rankings. The default lexical preview changes from the file's first three lines to the winning chunk's text (--full-section still returns the whole file). If the shorter preview is preferred as the default, this is easy to gate. Spans reflect the content the index was built from, so they are only as fresh as the index; the corpus fingerprint triggers a rebuild on ordinary edits, but a span can point at shifted lines after a change that preserves a file's mtime and size.
1 parent bb307fb commit 61e44fb

1 file changed

Lines changed: 357 additions & 12 deletions

File tree

0 commit comments

Comments
 (0)