feat: add LiteParseLoader (LiteParse optional extra)#534
Conversation
jexp
force-pushed
the
feature/liteparse-loader
branch
from
82300a5 to
0b4241c
Compare
Adds LiteParseLoader to neo4j_graphrag.experimental.components.data_loader as an alternative PDF/document loader using LiteParse — a local, zero-cloud Rust-based parser with optional Tesseract OCR support. - New liteparse optional extra: pip install "neo4j-graphrag[liteparse]" - Lazy-imports liteparse; raises a clear ImportError with install hint when absent - Parser instance cached per loader to avoid repeated Rust init overhead - Supports local FS fast-path (file path) and non-local FS (bytes) via fsspec - 9 unit tests (fully mocked, no liteparse install required) - 5 integration tests (real liteparse; auto-skipped when not installed) - Usage example in examples/customize/build_graph/components/loaders/
Extracts LiteParseLoader from data_loader.py into a dedicated liteparse_loader.py, keeping the optional-dependency import guard and all liteparse-specific logic isolated from the core loaders. data_loader.py is restored to only DataLoader, PdfLoader, MarkdownLoader.
Pass output_format='markdown' to get #/##-style headers in the parsed text, enabling clean chaining with HierarchicalTextSplitter(header_strategy='markdown'). Bumps the liteparse version pin to >=2.1.0 to guarantee the param is available.
jexp
force-pushed
the
feature/liteparse-loader
branch
from
41b8fde to
fb54b43
Compare
AmirLayegh
left a comment
AmirLayegh
left a comment
There was a problem hiding this comment.
Nice work! 🙌
Left a few comments worth addressing before merging.
| metadata=self.get_document_metadata(text, metadata), | ||
| document_type=DocumentType.PDF, | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Considering LiteParse also parses other document types such as DOCX, XLSX, PPTX, and images, loading a non-PDF file will result in document_type="pdf" in the returned DocumentInfo. Since the field is optional, it would be more accurate to infer the type from the file extension and fall back to None for unsupported types.
| return self._parser | ||
|
|
||
| def load_file(self, filepath: str, fs: AbstractFileSystem) -> str: | ||
| """Parse a document and return the full extracted text.""" |
There was a problem hiding this comment.
PdfLoader.load_file and MarkdownLoader.load_file both use file as the parameter name, but LiteParseLoader.load_file uses filepath. This can be renamed to file to stay consistent.
There was a problem hiding this comment.
The lock file pins liteparse==2.0.3 while pyproject.toml specifies >=2.1.0,<3.0.0. I guess the lock file should be regenerated accordingly.
2 participants
Summary
LiteParseLoaderin a newneo4j_graphrag.experimental.components.liteparse_loadermodule as an alternative PDF/document loader backed by LiteParse — a local, zero-cloud Rust parser with optional Tesseract OCR supportdata_loader.pyis unchanged —LiteParseLoaderlives in its own module alongside the optional-dependency import guardliteparseoptional extra:pip install "neo4j-graphrag[liteparse]"PdfLoader; pass asfile_loader=LiteParseLoader()toSimpleKGPipelineChanges
src/neo4j_graphrag/experimental/components/liteparse_loader.py— new module withLiteParseLoaderpyproject.toml—liteparse = ["liteparse>=2.0.0,<3.0.0"]optional extra + mypy overrideCHANGELOG.md— entry under## Nextexamples/customize/build_graph/components/loaders/liteparse_loader.py— usage exampletests/unit/experimental/components/test_liteparse_loader.py— 9 unit tests (fully mocked)tests/unit/experimental/components/test_liteparse_loader_integration.py— 5 integration tests (real liteparse; auto-skipped when absent)Usage
Test plan
uv run pytest tests/unit/experimental/components/test_liteparse_loader.py— 9 unit tests pass without liteparse installedpip install "neo4j-graphrag[liteparse]" && uv run pytest tests/unit/experimental/components/test_liteparse_loader_integration.py— 5 integration tests pass with liteparse installedtest_data_loader.pytests unaffected