feat: department-filtered search for local document indexes (+2 PDF pipeline fixes)#115
Merged
andylizf merged 2 commits intoJul 16, 2026
Conversation
- pixelrag index build records a "department" field in articles.json, derived from the first sub-directory under the local source root (docs/<department>/<file> layout) - serve: new optional "department" field on POST /search pre-filters inside FAISS via IDSelectorBatch (SearchParametersIVF on IVF indexes), so n_docs results are guaranteed without over-fetching; unknown department returns 404 with the list of valid names; indexes built without department metadata return a clear 400 - new GET /departments endpoint listing departments with document counts - fix: rename PDF tile dirs to the pipeline position index. render_pdf names tile dirs after the PDF filename stem, and non-numeric stems made embed fall back to salted hash() article ids, breaking the positional articles.json mapping for every local PDF source - fix: serve now resolves PDF page images (tile_XXXX.jpg) when materialized chunk files do not exist, and /tile returns the correct MIME type for JPEG tiles - tests: build-side department mapping and serve-side selector filtering (flat + IVF), 404/400 error paths
|
@locphamnguyen is attempting to deploy a commit to the andylizf's projects Team on Vercel. A member of the Team first needs to authorize it. |
# Conflicts: # index/src/pixelrag_index/pipelines.py
andylizf
added a commit
to Anush008/PixelRAG
that referenced
this pull request
Jul 16, 2026
… contract tests Conflict resolution against main (StarTrail-org#115/StarTrail-org#118/StarTrail-org#119 landed since this PR branched): - Department filtering moved into the backend interface: raw_search now takes article_ids (+ filter_cache_key). FaissBackend implements it as the same IDSelectorBatch/SearchParametersIVF pre-filter main used, with the positions scan cached per key; QdrantBackend as a payload MatchAny(article_id) filter. api.py keeps only the backend-agnostic department -> article-ids lookup. - PIXELRAG_INDEX_MMAP support (added on main) ported into FaissBackend.__init__. - pyproject: qdrant extra layered onto main's post-StarTrail-org#118 extras. Fixes found while testing: - FaissBackend assumed IndexIVF: `.nprobe` access and make_direct_map() crash on flat indexes. nprobe is now a no-op/0 on flat indexes and reconstruct only builds a direct map when the index has one. tests/test_serve_backends.py runs the SAME contract suite against both backends (FAISS on a synthetic index; Qdrant fully in-process via QdrantClient(":memory:")): hit shape/ranking, article_ids pre-filter, k-guarantee within a filtered set, filter-position caching, reconstruct round-trip. Skips cleanly when the optional heavy extras are absent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds metadata-filtered search for indexes built from local document trees, and fixes two bugs in the local-PDF pipeline discovered while implementing it. Motivation: indexing an internal SOP corpus organized as
docs/<department>/<file>.pdfand serving per-department filtered retrieval (RBAC-style) from one index.1. Department metadata + filtered search
pixelrag index buildnow records adepartmentfield per article inarticles.json, derived from the first sub-directory under the local source root. Web/kiwix sources and files directly under the root get""— fully backward compatible (additive key).POST /searchaccepts an optionaldepartment: vector positions for that department are pre-filtered inside FAISS viaIDSelectorBatch(SearchParametersIVFon IVF indexes, plainSearchParametersotherwise), son_docsresults are guaranteed without over-fetch + post-filter. Position lists are cached per department.GET /departmentsendpoint returns departments with document counts.2. Fix: unstable article ids for local PDF sources
render_pdfnames tile dirs after the PDF filename stem, while embed/serve map vectors toarticles.jsonby the sequential position index. Non-numeric stems fell back tohash(stem) % 2**31(salted per process), silently breaking URL/path resolution for every local-PDF index. The pipeline now renames PDF tile dirs to the position index, matching the URL/text branches.3. Fix: tile serving for PDF page images
_resolve_pathonly looked for materializedchunk_XXXX_YY.pngfiles (Wikipedia layout). PDF pages are stored as wholetile_XXXX.jpgfiles, soinclude_imagesreturned null and/tile/...404'd for PDF indexes. Added a fallback to the page image, and/tilenow returns the correct MIME type for JPEG.Tests
tests/test_department_filter.py: build-side directory→department mapping (nested dirs, root files, file:// URLs, out-of-root paths) and serve-side selector filtering on bothIndexFlatIPandIndexIVFFlat, plus the 404/400 error paths. Full suite: 37 passed.include_imagesand/tileserve PDF pages correctly.