fix(deps): move tqdm, pyyaml, numpy from heavy extras into core#118
Merged
Conversation
tqdm, pyyaml and numpy are light, pure-Python utilities that the umbrella CLI (`pixelrag chunk`, `pixelrag index`) and the torch-free stage code import at module load: chunk.py, index/config.py, and embed.py (which deliberately lazy-imports torch but imports numpy/tqdm at the top). Gating them behind the heavy `embed`/`index` extras forced a torch install for light operations and made those modules fail to import on a core-only install — which is why the test suite (deliberately torch-free; no test imports torch/faiss/transformers) hit ModuleNotFoundError at collection time on `uv sync --extra dev`. Move the three into core dependencies and drop the now-redundant copies from the embed/serve/index/eval extras. "Core stays light (no torch)" still holds — these are not torch/GPU deps — and the suite now imports cleanly with no skips or CI changes.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
andylizf
added a commit
to Anush008/PixelRAG
that referenced
this pull request
Jul 16, 2026
… contract tests Conflict resolution against main (StarTrail-org#115/StarTrail-org#118/StarTrail-org#119 landed since this PR branched): - Department filtering moved into the backend interface: raw_search now takes article_ids (+ filter_cache_key). FaissBackend implements it as the same IDSelectorBatch/SearchParametersIVF pre-filter main used, with the positions scan cached per key; QdrantBackend as a payload MatchAny(article_id) filter. api.py keeps only the backend-agnostic department -> article-ids lookup. - PIXELRAG_INDEX_MMAP support (added on main) ported into FaissBackend.__init__. - pyproject: qdrant extra layered onto main's post-StarTrail-org#118 extras. Fixes found while testing: - FaissBackend assumed IndexIVF: `.nprobe` access and make_direct_map() crash on flat indexes. nprobe is now a no-op/0 on flat indexes and reconstruct only builds a direct map when the index has one. tests/test_serve_backends.py runs the SAME contract suite against both backends (FAISS on a synthetic index; Qdrant fully in-process via QdrantClient(":memory:")): hit shape/ranking, article_ids pre-filter, k-guarantee within a filtered set, filter-position caching, reconstruct round-trip. Skips cleanly when the optional heavy extras are absent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Several PRs fail CI's
Testsjob at import/collection time withModuleNotFoundError(tqdm,yaml,numpy) — not on any assertion. The suite is deliberately torch-free (no test importstorch/faiss/transformers), and it runs onuv sync --extra dev(core + pytest only).Root cause: three light, pure-Python utilities are gated behind the heavy
embed/indexextras, even though modules that are meant to run without torch import them at module load:embed/chunk.py— pure image slicing (stdlib + PIL), usestqdmindex/config.py— config parsing, usesyamlembed/embed.py— deliberately lazy-importstorchinside functions, but importsnumpy/tqdmat the topSo
pixelrag chunk/pixelrag index --helpand these tests can't import on a core-only install without dragging in torch/faiss.Fix
Move
tqdm,pyyaml,numpyinto coredependencies; drop the now-redundant copies from theembed/serve/index/evalextras. No CI changes, nopytest.skipworkarounds — the light stages simply become importable light again."Core stays light — no torch" still holds: these are not torch/GPU/heavy-ML deps.
uv lock --lockedconfirms the lock is consistent (no re-resolution; no version changes — pure regrouping).Unblocks (once merged, these go genuinely green — no skips)
pixelrag index build#73 —--device auto/mps: itsindex --helptest needsyaml, now in coreembed.embed(numpy/tqdm), now in coretqdmimport tochunk.py, now in core