feat(ck-engine): report chunk-level line spans for lexical search results - #167
Merged
Conversation
…ults Lexical (BM25) hits previously spanned the whole file (line 1 to EOF), so a match deep in a large file reported line 1 with a first-lines preview. Results now locate the best-matching chunk — using the same chunker the semantic path uses — and report its real span and text, falling back to the whole-file span when no chunk contains a query term. Document granularity, scoring, and result order are unchanged (covered by a ranking-invariance test). The refinement runs only after the top-k documents are selected. For each hit the stored content is chunked (language resolved as the indexer does: PDF check, else Language::from_path) and the winning chunk is the one with the most whole-token matches for the query terms. The terms are taken from the parsed tantivy query (its content-field terms), not the raw query string, so field prefixes, quoted phrases, and boolean operators — which tantivy strips before matching — don't leak into span selection. This consumes whatever query object the parser produces, so it composes with lenient parsing: the span step reads the content-field terms of the leniently-parsed query, and dropped fragments never leak into span selection. The chunk is tokenized the same way tantivy tokenizes (lowercase, split on non-alphanumeric) and matches are whole-token, so a short term like "in" doesn't tally hits inside larger words. Ties favor the earliest chunk. The tantivy document, its score, and top-k selection are untouched, so an existing index produces identical rankings. The default lexical preview changes from the file's first three lines to the winning chunk's text (--full-section still returns the whole file). If the shorter preview is preferred as the default, this is easy to gate. Spans reflect the content the index was built from, so they are only as fresh as the index; the corpus fingerprint triggers a rebuild on ordinary edits, but a span can point at shifted lines after a change that preserves a file's mtime and size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lexical (BM25) hits previously spanned the whole file (line 1 to EOF), so a match deep in a large file reported line 1 with a first-lines preview. Results now locate the best-matching chunk — using the same chunker the semantic path uses — and report its real span and text, falling back to the whole-file span when no chunk contains a query term. Document granularity, scoring, and result order are unchanged (covered by a ranking-invariance test).
The refinement runs only after the top-k documents are selected. For each hit the stored content is chunked (language resolved as the indexer does: PDF check, else
Language::from_path) and the winning chunk is the one with the most whole-token matches for the query terms. The terms are taken from the parsed tantivy query (its content-field terms), not the raw query string, so field prefixes, quoted phrases, and boolean operators — which tantivy strips before matching — don't leak into span selection. The chunk is tokenized the same way tantivy tokenizes (lowercase, split on non-alphanumeric) and matches are whole-token, so a short term like "in" doesn't tally hits inside larger words. Ties favor the earliest chunk. The tantivy document, its score, and top-k selection are untouched, so an existing index produces identical rankings.The default lexical preview changes from the file's first three lines to the winning chunk's text (
--full-sectionstill returns the whole file). If you'd rather keep the shorter preview as the default, this is easy to gate.Spans reflect the content the index was built from, so they are only as fresh as the index; the corpus fingerprint triggers a rebuild on ordinary edits, but a span can point at shifted lines after a change that preserves a file's mtime and size.