[Enhancement] Support filtered vector search with pre/post-filter strategies#74715
Open
sevev wants to merge 16 commits into
Open
[Enhancement] Support filtered vector search with pre/post-filter strategies#74715sevev wants to merge 16 commits into
sevev wants to merge 16 commits into
Conversation
Evaluate a residual scalar predicate (one not resolved by an exact index) against the vector ANN search in SegmentIterator: pre-filter by early-evaluating it into the candidate id set when the index supports efficient filtered search, otherwise post-filter by oversampling and letting the read-time predicate path filter the result. Add VectorIndexReader::supports_efficient_filtered_search() and the enable_vector_index_residual_prefilter / vector_index_residual_post_filter_oversample configs. Signed-off-by: sevev <qiangzh95@gmail.com>
RewriteToVectorPlanRule used to disable the vector index (falling back to a brute-force scan) whenever a scan predicate could not be fully parsed into a vector distance range. Split the predicate instead: distance-range conjuncts fold into the ANN range, and the residual scalar conjuncts stay on the scan where the BE pre/post-filters them against the ANN. A predicate involving the vector distance but not a valid range bound still disables the index. Signed-off-by: sevev <qiangzh95@gmail.com>
Design for WHERE <scalar predicate> ORDER BY approx_*_distance(vec, q) LIMIT k on internal tables, including predicates on un-indexed columns. FE unconditionally rewrites to an ANN plan; the BE runtime resolver picks PRE/POST/BRUTE per segment so a residual filter is never silently under-returned. Signed-off-by: sevev <qiangzh95@gmail.com>
Route a vector top-k query to exact brute-force whenever the residual filter cannot be safely served by a segment-level ANN k-limit, so results are never silently under-returned.
- vector_filter_strategy.{h,cpp}: pure resolve_ann_filter_strategy() returning PRE/POST/BRUTE from bitmap-free gates (user choice, config, residual, above-iterator predicate, exactness, reader capability).
- has_predicate_above_iterator: computed in OlapChunkSource and LakeDataSource, plumbed via TabletReaderParams -> RowsetReadOptions -> SegmentReadOptions.
- _init_ann_reader: run the resolver before column iterators; route to brute (may add the embedding column to _schema) on BRUTE. Disable late materialization when brute is active so the brute-only embedding column is not materialized into the output chunk.
- Tests: resolver truth table; residual queries route to brute on an above-iterator predicate and with the kill-switch off.
Signed-off-by: sevev <qiangzh95@gmail.com>
The residual pre-filter only walked the immediate per-column predicate map and skipped expr predicates, so OR / compound / single-column-expr residuals were silently dropped: the ANN ran effectively unfiltered and the read-time predicate then cut the result below k (under-return). Evaluate the entire pred_tree into the candidate bitmap instead. - _evaluate_residual_to_bitmap: read all pred_tree columns into a per-batch chunk keyed by column id and call PredicateTree::evaluate (AND/OR/compound/expr). A predicate column that is not readable is now a hard error, not a silent skip. - has_residual uses !pred_tree.empty() (whole tree) in the resolver and the PRE path; drop the dead post-filter oversample (AUTO routes to brute, not POST; explicit POST returns later). - Test: residual_or_predicate_prefilters_ann (filter_col=4 OR filter_col=6). Signed-off-by: sevev <qiangzh95@gmail.com>
Wire VectorSearchOption.filter_strategy (the ann_filter_strategy override) into the resolver and record the resolved strategy on the vector index context so the search dispatches on it.
- _init_ann_reader passes filter_strategy as the resolver's user_choice (was hardcoded AUTO) and stores the result; BRUTE routes to the read-loop fallback.
- _get_row_ranges_by_vector_index: PRE narrows the scan range to the exact residual bitmap before the search (unchanged); POST (explicit opt-in) skips the bitmap and over-fetches, leaving the read-time predicate to drop non-matching rows (approximate, may return < k).
- Tests: explicit_brute_force_returns_all_matching ({4,5,6,7}); explicit_post_filter_may_underreturn ({} -- the 3 nearest the query all fail the filter).
Signed-off-by: sevev <qiangzh95@gmail.com>
Expose the per-query override that drives the BE filter-strategy resolver. - TQueryOptions.ann_filter_strategy (field 153) + SessionVariable ann_filter_strategy (auto|pre|post|brute_force, default auto), mapped to the AnnFilterStrategy ordinal (0/1/2/3) and passed via toThrift(). - OlapChunkSource / LakeDataSource read query_options().ann_filter_strategy into VectorSearchOption::filter_strategy (the resolver's user_choice). - Test: SessionVariableTest.testAnnFilterStrategyToThrift (mapping, case-insensitive, invalid->auto, thrift passthrough). Signed-off-by: sevev <qiangzh95@gmail.com>
- vector-filter-strategy-test-coverage.md: enumerate every covered scenario (resolver truth table, PRE/POST/BRUTE + Gap2 execution, brute fallback, FE session-var mapping), map each to a correctness invariant, and list what is deferred (selectivity gate, execution-layer Strategy classes, SQL e2e). - vector-filter-strategy.md: add an implementation-status section correcting the in-scope abstraction (resolver decision-layer implemented; polymorphic VectorFilterStrategy/VectorFilterContext execution layer and selectivity gate deferred). Signed-off-by: sevev <qiangzh95@gmail.com>
Cover {scalar, fulltext (GIN MATCH), scalar AND/OR fulltext, above-iterator} x {high/medium/low/very-low selectivity} x {auto, pre, post, brute_force} on DUPLICATE and PRIMARY KEY tables. Each exact strategy's ANN result (approx_l2_distance, FE-rewritten) is asserted equal to the full-scan ground truth (l2_distance); POST asserts soundness; EXPLAIN asserts VECTORINDEX: ON; the kill-switch toggles enable_vector_index_residual_prefilter. R/ is recorded on first cluster run.
Signed-off-by: sevev <qiangzh95@gmail.com>
…tionIterator A vector-index scan (ANN or brute-force) with a residual scalar/fulltext predicate plus global late materialization crashed the BE with a SIGSEGV in ColumnRef::evaluate_checked under ProjectOperator::push_chunk. Root cause: with a residual predicate, new_segment_iterator wraps the SegmentIterator in a ProjectionIterator. SegmentIterator appends the ANN distance column to the output chunk at runtime (append_vector_column), but ProjectionIterator builds its column index map from the static schemas, so the runtime-appended distance column is never copied to the output and is dropped. OlapChunkSource then cannot find the distance slot, stores SIZE_MAX as its column index, and the upstream ColumnRef dereferences _columns[SIZE_MAX]. Fix: in ProjectionIterator::do_get_next, after the index-map projection, re-attach any columns the child appended beyond its static output schema by their slot id. No-op for non-vector reads (no extra columns); for vector reads it carries the distance column through by a pointer swap (no row-data copy), leaving an empty clone in the internal chunk so the next reset() is safe. Also broaden the predicate-column late-materialization guard in SegmentIterator::_init_context from brute-force-only to all non-IVFPQ vector-index reads: ANN appends a synthetic distance column and brute additionally appends the embedding column to _schema, neither of which the late-materialization rowid/output assumptions handle. Tests: - projection_iterator_test: ProjectionIterator carries a runtime-appended slot-tagged column to the output (without the fix the output has 1 column instead of 2). - vector_search_test: residual predicate + predicate-column late materialization + multi-output read keeps every output column == num_rows. Signed-off-by: sevev <qiangzh95@gmail.com>
…urn, POST) Correctness fixes for vector ANN search with scalar/fulltext residual predicates, found by an exhaustive cluster matrix (41 predicate shapes over DUP + PRIMARY KEY tables, scalar + GIN fulltext, AUTO/PRE/POST/BRUTE vs an exact brute-force ground truth). 1. Multi-column-leaf residual returned wrong (empty) results. A residual leaf referencing >1 column (e.g. cat + tag > 50) cannot be pushed into the ANN exact pre-filter bitmap; the planner keeps it as a filter above the segment k-limit, so a k-limited PRE search under-returns. The FE now flags such residuals (RewriteToVectorPlanRule.hasMultiColumnLeaf -> TVectorSearchOptions.has_complex_residual) and the BE resolver treats them like an above-iterator predicate, routing the scan to the exact BRUTE path. 2. Selectivity gate: HNSW filtered search under-returns on selective+scattered residuals. Even with an exact pre-filter bitmap, HNSW graph traversal can miss candidates scattered far apart in vector space under a very selective filter (e.g. cat=1 AND tag=51 -> 7 of 10 rows). After the filtered search, if it found fewer survivors than the bitmap could supply, the BE now rescans the (small) candidate set with exact distances (_exact_search_over_candidates), restoring completeness. The brute distance math is shared via _brute_force_distance_column. 3. POST under-returned by default. POST post-filters after the search but did not over-fetch (search_k = k), so a selective residual dropped most results. Wire the existing vector_index_residual_post_filter_oversample config into search_k for POST. 4. SIGSEGV on inverted-index fallback predicate in the PRE bitmap. A residual with a MATCH inside an OR (e.g. doc MATCH 'x' OR cat < 2) is kept as an InvertedIndexFallbackPredicate whose evaluate() maps chunk rows via the iterator's rowid buffer. _evaluate_residual_to_bitmap did not publish that buffer, so the predicate dereferenced an empty buffer and crashed. Publish each batch's contiguous rowids before evaluating the tree. Tests: - FE: VectorIndexTest.testHasMultiColumnLeaf (direct unit test). - BE: vector_search_test explicit_post_filter_oversamples (POST over-fetch). - Cluster: 17 scalar + 11 fulltext shapes on DUP, 13 on PK; AUTO/PRE == exact ground truth, POST sound (approximate by design), no crashes. Signed-off-by: sevev <qiangzh95@gmail.com>
The multi-column-leaf residual fix previously relied on an FE rewrite-time heuristic (RewriteToVectorPlanRule.hasMultiColumnLeaf -> has_complex_residual thrift flag). The vector rewrite runs BEFORE the predicate split, so the rule had to guess which residuals would later land in a SELECT above the ANN scan. That guess assumed single-column predicates always push into the scan -- true in the current planner but not an invariant. Replace the guess with an observation of the real execution tree. After FragmentExecutor builds the ExecNode tree, a one-time walk flags any ScanNode that has a row-filtering operator (SELECT) above it but below the TopN limit; the ANN filter resolver then treats it exactly like an above-iterator predicate and uses the exact brute-force path. This is robust to how single-column predicates are placed, and mirrors Doris's BE-side decision -- Doris reads it from the scan's own conjuncts because Doris pushes every predicate onto the scan, whereas StarRocks keeps a non-pushable residual in a separate SELECT operator, so it walks the tree. - ScanNode: _filtered_above_iterator + setter/getter. - FragmentExecutor::mark_filtered_above_scans: reset at SORT_NODE (TopN is the limit boundary; a filter above it is post-limit), set at SELECT_NODE, flag ScanNodes in scope. Project and other passthrough nodes are transparent. - OlapChunkSource / LakeDataSource: OR the flag into has_predicate_above_iterator. - segment_iterator: derive both resolver inputs from has_predicate_above_iterator. Removes the FE heuristic and its plumbing: hasMultiColumnLeaf, has_complex_residual (VectorSearchOptions field, TVectorSearchOptions thrift field 12 [retired, ordinal not reused], BE VectorSearchOption field + read), and the FE unit test. Performance: the walk is a one-time O(plan-nodes) pass at fragment setup, off any per-row path; the set of queries routed to brute is unchanged. Signed-off-by: sevev <qiangzh95@gmail.com>
- Add R/test_vector_filter_strategy: recorded and validated on a shared-nothing cluster. The case asserts the ANN result (approx_*_distance, FE-rewritten) equals the exact full-scan ground truth (l2_distance) across scalar / fulltext (GIN MATCH) / combined / above-iterator residuals x selectivities x auto/pre/post/brute_force, plus POST soundness, the residual-prefilter kill-switch, and a PRIMARY KEY table. It is @Sequential (run with -a sequential). - Remove docs/design/vector-filter-strategy{,-test-coverage}.md: internal design notes, not intended for upstream. Signed-off-by: sevev <qiangzh95@gmail.com>
|
@codex review |
Contributor
|
No new undocumented parameters detected by the param-drift check. |
sevev
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87d7aedd8c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| -- =========================================================================== | ||
| -- B1. medium selectivity (cat < 5 ~ 50%): auto == pre == brute_force == exact | ||
| set ann_filter_strategy = 'auto'; | ||
| function: assert_equal_result("select id from vt where cat < 5 order by approx_l2_distance(vec, [0,0,0,0,0]), id limit 10", "select id from vt where cat < 5 order by l2_distance(vec, [0,0,0,0,0]), id limit 10") |
There was a problem hiding this comment.
These correctness checks do not exercise the new vector-filter paths because the left query orders by two expressions (approx_l2_distance(...), id). RewriteToVectorPlanRule.check() returns false unless topNOp.getOrderByElements().size() == 1, so these assert_equal_result calls bypass VECTORINDEX entirely and the preceding set ann_filter_strategy values are not tested. Use a single vector-distance order expression here, or add a separate assertion that the exact query being compared contains VECTORINDEX: ON.
Useful? React with 👍 / 👎.
Comment on lines
+1089
to
+1090
| // ANN scalar-filter strategy override: auto | pre | post | brute_force. | ||
| public static final String ANN_FILTER_STRATEGY = "ann_filter_strategy"; |
There was a problem hiding this comment.
This adds a user-settable session variable, and the new SQL tests use SET ann_filter_strategy = ..., but the docs changed in this commit only cover enable_vector_index_refine; I could not find ann_filter_strategy in docs/en or docs/zh. The repo AGENTS.md requires user-facing config/session changes to update the matching English and Chinese docs, otherwise users have no documented accepted values (auto, pre, post, brute_force) or semantics.
Useful? React with 👍 / 👎.
Contributor
🌎 Translation Required?Thanks for your doc contribution! The following languages are missing or outdated for your changes. 🤖 Automated TranslationsIf you are fluent in any of the missing languages you can add them. If not, a maintainer can generate the translations below. Maintainer: Check the ones you wish to generate:
|
github-actions
Bot
added
the
docs-maintainer
The quantization-keyed refine option itself merged upstream (StarRocks#74533). This keeps only the parts that PR does not cover: - Rename the remaining use_ivfpq gates in the v3 filter-strategy code paths (segment_iterator resolver/late-mat gates, vector_filter_strategy, BE UT fixture) to refine_distance, matching the upstream rename. - test_vector_index_hnsw_lazymat: SET enable_vector_index_refine = true so the IVFPQ-eager check keeps refine semantics under the new default (refine off). Signed-off-by: sevev <qiangzh95@gmail.com>
sevev
force-pushed
the
feat/vector-filter-strategy-v3
branch
from
d2b6012 to
348ad82
Compare
github-actions
Bot
added
behavior_changed
and removed
documentation
Contributor
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
Contributor
[FE Incremental Coverage Report]✅ pass : 1 / 1 (100.00%) file detail
|
Contributor
[BE Incremental Coverage Report]❌ fail : 69 / 143 (48.25%) file detail
|
…E bitmaps After the PRE strategy evaluates the residual predicate into an exact candidate bitmap, skip the filtered HNSW search and score the candidates exactly (_exact_search_over_candidates, the same kernel the post-search count gate uses) when the search cannot pay for itself: - cardinality <= k: a top-k over <= k candidates must return every candidate, so the search is a logical no-op. Always on. - cardinality <= vector_index_brute_selectivity_threshold * segment rows (new mutable BE config, default 0.01, 0 disables): a bitmap this sparse relative to the graph makes the HNSW traversal slow and likely to under-return, paying the exact rescan on top of the wasted search. The denominator is the segment's total rows -- the graph the traversal walks -- not the pruned scan range. This is routing only: both sides are exact, so a mis-set threshold costs speed, never correctness. Completeness remains owned by the post-search count gate, which is unchanged. The gate applies to top-k PRE only: range search keeps its own path and refine (which never ran the resolver) is explicitly excluded, leaving both behaviors untouched. Tests: 4 new BE UT cases (cardinality < k, == k boundary, ratio leg with a 0.5 threshold, and an anti-overfire case asserting the search still runs above both thresholds), keyed on vector_search_timer == 0 -- the timer accumulates only inside the search block, so it is a deterministic skipped-search marker, not a timing assertion. All 33 vector suites green under ASAN. Signed-off-by: sevev <qiangzh95@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
Today, when a vector ANN query carries a scalar predicate that the index rewrite cannot fully resolve (a "residual" predicate, e.g.
WHERE category = 3 ORDER BY approx_l2_distance(...) LIMIT 10), the FE gives up the ANN rewrite and the query degrades to a brute-force scan over all matching rows. This makes filtered vector search — the most common production shape — pay full-scan cost even when an HNSW/IVFPQ index exists, and pushing only part of the predicate down can silently return fewer thankcorrect results.The decision of how to combine a filter with ANN (pre-filter the candidate set, post-filter with oversampling, or fall back to brute force) depends on runtime information the FE plan cannot see: whether the index supports efficient filtered search, whether a residual SELECT operator sits above the scan in the execution tree, and how selective the filter actually is. So the FE should keep the ANN rewrite and let the BE decide.
What I'm doing:
The FE now unconditionally rewrites eligible ANN queries while keeping the residual predicate in the plan. The BE adds a filter-strategy resolver (
be/src/storage/index/vector/vector_filter_strategy.{h,cpp}) that picks one of three execution paths per segment at runtime: PRE (evaluate the whole predicate tree into a row bitmap and run filtered ANN over it, with a count-based selectivity gate that self-corrects to exact search over the candidates when the filter is too sparse for HNSW), POST (oversample the ANN byvector_index_residual_post_filter_oversample, default 3x, then let the normal read-time predicate path filter), and BRUTE (in-iterator exact search) as the safe fallback.FragmentExecutorwalks the execution tree and forces BRUTE when a SELECT operator with complex residuals sits above the ANN scan, since a partial pushdown there would under-return. A new session variableann_filter_strategy(auto/pre/post/brute_force, defaultauto) allows overriding the resolver, andenable_vector_index_residual_prefilter(BE config, default true) is the kill switch back to the old behavior. A latent bug whereProjectionIteratordropped the runtime-appended distance column is also fixed.Verified with FE UT (
VectorIndexTest,SessionVariableTest), BE UT (vector_search_test,projection_iterator_test, ASAN-clean), new SQL testtest_vector_index/test_vector_filter_strategy, and a cluster matrix of 41 query shapes (duplicate/PK tables, single/multi-column predicates, MATCH-in-OR, all four strategies) whereauto/pre/postresults match brute-force ground truth with no crashes. User documentation forann_filter_strategywill follow in a separate doc PR.What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: