feat(vector_io): add classifier reranker type for post-retrieval chunk filtering

[varshaprasad96]

Summary

Add "classifier" as a new reranker type that scores chunks by quality or answerability (rather than relevance) and filters below a confidence threshold. Uses the same inference rerank API as neural reranking — the difference is the objective model and the filtering behavior.

Refs: #5728

Motivation

Vector search ranks by embedding similarity, but similarity ≠ usefulness. Rerankers ("neural") re-score by relevance but don't assess chunk quality — whether a chunk is complete, non-redundant, and likely to produce a correct answer. A classifier model trained on "did this chunk lead to a correct answer?" captures a different signal than "is this chunk similar to the query?"

As discussed in #5728, a classifier with probability output is functionally a reranker. Rather than introducing a separate abstraction, this implements the classifier as a reranker type — same mechanism, different objective.

Usage

POST /v1/vector_stores/{id}/search
{
  "query": "quarterly revenue",
  "ranking_options": {
    "ranker": "classifier",
    "model": "my-org/chunk-quality-classifier",
    "score_threshold": 0.7
  }
}

Changes

• vector_store.py: Add RERANKER_TYPE_CLASSIFIER constant, apply_classifier_rerank() method, wire into query_chunks() dispatch. Extract _extract_chunk_texts() helper to deduplicate text extraction between neural and classifier rerankers.
• openai_vector_store_mixin.py: Handle ranker="classifier" in _build_reranker_params().
• models.py: Document "classifier" as valid ranker value in SearchRankingOptions.

Design decisions

• Runs after neural rerank (if both enabled) as a final quality gate
• Falls back to default_reranker_model from config when no explicit model provided
• Gracefully handles inference errors (returns original results)
• Logs warning when all chunks are filtered out
• Backward compatible — no change when ranker is not "classifier"

Test plan

uv run pytest tests/unit/providers/vector_io/test_classifier_reranker.py -v

9 tests covering: threshold filtering, keeps above threshold, zero threshold, no model fallback, inference error, correct model passthrough, out-of-bounds index, all-filtered empty response, constant definition.

============================== 9 passed in 0.05s ==============================

Signed-off-by: Varsha Prasad Narsing vnarsing@redhat.com

🤖 Generated with Claude Code

[franciscojavierarceo]

can you test the retrieval benchmark suite to see the performance?

[github-actions]

This pull request has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

[varshaprasad96]

Classifier Reranker Benchmark Analysis

I ran a comparative analysis of the classifier reranker concept using BEIR/nfcorpus (50 queries, ~3600 documents) to determine whether classifier-based chunk filtering provides retrieval quality improvements.

Setup

• Dataset: BEIR/nfcorpus (medical domain, 3633 docs, 323 judged queries)
• Baseline: Hybrid search (dense + sparse + RRF) with no classifier
• Embedding model: nomic-ai/nomic-embed-text-v1.5
• Vector store: sqlite-vec
• Metrics: nDCG@10, Recall@10, MAP@10 (via pytrec_eval against BEIR ground truth)

Two models were tested as the classifier:

Model	Type	Params	Architecture
Qwen/Qwen3-Reranker-0.6B (current)	Causal LM	600M	yes/no token classification via `log_softmax`
cross-encoder/ms-marco-MiniLM-L6-v2	CrossEncoder	22M	Bi-directional attention over query+passage pair

---

Finding 1: Score Distribution

The current Qwen3-Reranker produces an extremely narrow score range, making threshold-based filtering impossible. The CrossEncoder uses the full 0–1 range.

Metric	Qwen3 (current)	CrossEncoder
Min	0.0125	0.0000
Max	0.0328	0.9995
Mean	0.0194	0.1390
Std	0.0061	0.2758
P10	0.0132	0.0001
P25	0.0145	0.0004
P50	0.0161	0.0054
P75	0.0245	0.0978
P90	0.0290	0.6256
P95	0.0311	0.9467

Qwen3's 0.006 std means all scores are essentially the same value. The default score_threshold=0.3 in the PR filters out 100% of results because no score exceeds 0.033.

---

Finding 2: Relevance Discrimination

Does the classifier actually assign higher scores to relevant chunks (per BEIR ground truth)?

Metric	Qwen3 (current)	CrossEncoder
Relevant chunks (mean score)	0.0216	0.3222
Irrelevant chunks (mean score)	0.0189	0.0990
Score separation ratio	1.15x	3.26x

The CrossEncoder discriminates 3x better between relevant and irrelevant chunks. Qwen3's 1.15x separation is essentially noise.

---

Finding 3: Retrieval Quality (Threshold Sweep)

How does filtering at various thresholds affect retrieval metrics?

Qwen3-Reranker (current classifier):

Threshold	nDCG@10	Recall@10	MAP@10	Avg Results
baseline	0.3517	0.1709	0.1298	--
0.0000	0.3517	0.1709	0.1298	18.7
P10 (0.013)	0.3517	0.1709	0.1298	17.1
P25 (0.014)	0.3502	0.1702	0.1292	14.1
P50 (0.016)	0.2807	0.1391	0.1142	9.6
0.30 (PR default)	0.0000	0.0000	0.0000	0.0

⚠️ The current implementation with score_threshold=0.3 returns zero results for every query.

CrossEncoder/ms-marco-MiniLM-L6-v2:

Threshold	nDCG@10	Recall@10	MAP@10	Avg Results
baseline	0.3517	0.1709	0.1298	--
0.0000	0.3822 (+8.7%)	0.1626	0.1400 (+7.9%)	18.6
0.0001	0.3791	0.1617	0.1397	16.7
0.005	0.3508	0.1424	0.1313	9.3
0.098	0.2952	0.1205	0.1123	4.7
0.30	0.2218	0.0754	0.0715	2.8
0.50	0.1738	0.0545	0.0518	2.2

✅ CrossEncoder at threshold=0 (reranking only, no filtering) improves nDCG@10 by +8.7% over baseline.

---

Finding 4: Root Cause

The current classifier reranker calls inference_api.rerank() with the same model used for neural reranking (Qwen3-Reranker-0.6B). This means:

1. Identical scores: Classifier produces the exact same scores as ranker="neural" — the ranking is unchanged from baseline.
2. Uncalibrated output: The Qwen3 model uses exp(log_softmax(yes_logit)) which produces values in the 0.01–0.03 range, not the 0–1 range needed for threshold filtering.
3. Architecturally redundant: The only functional difference between ranker="classifier" and ranker="neural" is the threshold filter, which doesn't work because scores are too low.

---

Recommendations

1. The classifier concept has merit — but only with a model that produces well-calibrated, well-separated scores. The CrossEncoder (ms-marco-MiniLM-L6-v2) shows a real +8.7% nDCG improvement.

2. The classifier needs a different model than the reranker. Reusing the same model is architecturally a no-op. Candidate models:

   • cross-encoder/ms-marco-MiniLM-L6-v2 (22M params, fast, tested above)
   • cross-encoder/qnli-electra-base (trained specifically for answerability classification)
   • BAAI/bge-reranker-v2-m3 (multilingual, modern)

---

Reproducing these results

# Start OGX with local config (sqlite-vec + sentence-transformers)
OPENAI_API_KEY=<key> uv run ogx run benchmarking/rag/config-local.yaml --insecure

# Run baseline BEIR benchmark
uv run python benchmarking/rag/run_benchmark.py \
  --benchmark beir --dataset nfcorpus --search-mode hybrid \
  --provider ogx --base-url http://localhost:8321/v1 --no-resume

# Run comparative analysis
uv run python benchmarking/rag/analyze_classifier.py

[franciscojavierarceo]

nice!