Dedupe blog search results so a post appears only once - #8081
Merged
Conversation
Mixedbread returns chunk-level matches, and the search route mapped every chunk to its own page result. A post matching a query in several chunks (e.g. "Neon" for the Neon pricing post) showed up 2-4 times in the search dialog. Keep only the first (highest-scoring) chunk per file_id; this also frees up topK slots for more distinct posts.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughChangesBlog search deduplication
Estimated code review effort: 2 (Simple) | ~5 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
|
The latest updates on your projects. Learn more about Argos notifications ↗︎
|
Member
Author
|
@coderabbitai approve |
AmanVarshney01
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Searching the blog shows the same post multiple times. For example, searching "Neon" returns Prisma Postgres vs Neon Pricing 2026 four times:
The blog search is backed by a Mixedbread vector store, which searches at the chunk level. The route's
transformmapped every matching chunk to its ownpageresult (keyed byfile_id-chunk_index), so any post matching the query in several chunks rendered as several identical cards. Live evidence:/blog/api/search?query=Neonreturned the samefile_idwith chunk indices 0, 1, 2, and 3 as four separate results.The more relevant a post is to a query, the more chunks match, and the more it duplicates. The duplicates also crowd out other posts inside the
topK: 20window.Fix
Track seen
file_ids in the transform and skip chunks from files already emitted. Results arrive in score order, so each post keeps its best-ranked position and appears exactly once. Query-time only, no reindexing of the store needed.Verification
No
MIXEDBREAD_API_KEYis available locally, so I replayed the exact live chunk sequence for "Neon" (4 chunks of the pricing post interleaved with 2 other posts) through the real routeGEThandler with only the SDK network call stubbed:types:check(tsc) passes; oxlint reports 0 errors.Summary by CodeRabbit