Skip to content

Commit e1e7c51

Browse files
committed
docs: expand extraction and openalex documentation
Add detailed sections for PDF extraction via Datalab/marker and OpenAlex rate limiting. Includes CLI usage examples and local caching instructions.
1 parent 9882741 commit e1e7c51

1 file changed

Lines changed: 44 additions & 3 deletions

File tree

README.md

Lines changed: 44 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,7 @@
77
Search, manage, and explore academic papers from the terminal. Or run as an [MCP](https://github.com/mmgeorge/papers/tree/main?tab=readme-ov-file#mcp-server) to query your papers with an LLM. Queries 240M+ works via [OpenAlex](https://openalex.org), integrates with your [Zotero](https://www.zotero.org) library, and builds a local vector index over your papers with [LanceDB](https://github.com/lancedb/lancedb) so you can semantically search across sections and figures. Embedding accelerated with DirectML and CoreML on Windows and macOS.
88

99
> [!NOTE]
10-
> Even the best analytical PDF-extraction methods mangle LaTeX and tables for technical papers. This project uses vision-model-based OCR via [Datalab](https://www.datalab.to/) (requires API key) to produce clean markdown with math and tables preserved. Extracted results (JSON, markdown, images) sync back to your Zotero library.
11-
>
12-
> You can also run [marker](https://github.com/datalab-to/marker) locally if you meet its [license requirements](https://github.com/datalab-to/marker?tab=readme-ov-file#commercial-usage) — just place the output in the cache directory.
10+
> Even the best analytical PDF-extraction methods mangle LaTeX and tables for technical papers. This project uses vision-model-based OCR via [Datalab](https://www.datalab.to/) (requires API key) to produce clean markdown with math and tables preserved. Extracted results (JSON, markdown, images) sync back to your Zotero library. You can also [run marker locally](#using-marker-locally).
1311
1412
## Install
1513

@@ -33,6 +31,8 @@ Commands accepts `--json` for machine-readable output.
3331

3432
## OpenAlex
3533

34+
OpenAlex works without authentication but is rate-limited. Set `OPENALEX_KEY` for higher rate limits ([openalex.org/pricing](https://openalex.org/pricing)).
35+
3636
### Search and filter
3737

3838
```sh
@@ -81,6 +81,47 @@ papers rag get-section <paper> <section>
8181
papers rag outline <paper>
8282
```
8383

84+
## Extraction
85+
86+
PDF extraction uses vision-model-based OCR via marker to produce clean markdown with LaTeX math and tables preserved. Extracted results (markdown, JSON, images) are cached locally and synced back to your Zotero library as attachments.
87+
88+
Requires `DATALAB_API_KEY` ([datalab.to](https://www.datalab.to/)).
89+
90+
```sh
91+
papers zotero work extract <item_key> # Extract a Zotero item (default: balanced)
92+
papers zotero work extract <item_key> -m accurate # Highest quality, slowest
93+
papers zotero work extract <item_key> -m fast # Fastest, lower layout accuracy
94+
papers work text <work_id> --advanced balanced # Extract an OpenAlex work via Datalab
95+
```
96+
97+
Processing modes: `fast`, `balanced` (default), `accurate`.
98+
99+
### Managing cached extractions
100+
101+
```sh
102+
papers zotero extract list # List items with cached extractions
103+
papers zotero extract text <query> # Print cached markdown
104+
papers zotero extract json <query> # Print cached JSON
105+
papers zotero extract get <query> # Print cache directory path
106+
papers zotero extract upload [--dry-run] # Upload local cache to Zotero
107+
papers zotero extract download [--dry-run] # Download Zotero cache to local
108+
```
109+
110+
Cache location: `~/.cache/papers/datalab/` (Linux/macOS) or `%APPDATA%\papers\datalab\` (Windows). Override with `PAPERS_DATALAB_CACHE_DIR`.
111+
112+
### Using marker locally
113+
114+
You can run [marker](https://github.com/datalab-to/marker) locally instead of using the Datalab API if you meet its [license requirements](https://github.com/datalab-to/marker?tab=readme-ov-file#commercial-usage). Place the output files in the cache directory:
115+
116+
```
117+
~/.cache/papers/datalab/<item_key>/
118+
├── <item_key>.md
119+
├── <item_key>.json # optional
120+
└── images/ # optional
121+
```
122+
123+
The extraction will be picked up automatically from the local cache.
124+
84125
## MCP server
85126

86127
Exposes CLI commands as MCP tools for LLMs. Currently only --stdio is supported.

0 commit comments

Comments
 (0)