Comparison with Markdown-based pipelines and efficiency vs text-only RAG

[xabd]

Hi! I came across PixelRAG and shared it with the Digital Escape Tools community:

https://www.reddit.com/r/DigitalEscapeTools/s/jT9TryU3FX

One of our community members, who mentioned they're a researcher, raised some interesting questions that I thought were worth forwarding here.

They asked:

Have you compared PixelRAG against a Markdown-based workflow instead of HTML-to-plain-text?

Since Markdown preserves document structure (headings, lists, tables) while reducing token usage, could the reported accuracy gains partly reflect the text baseline rather than a limitation of text representations?

Have you considered comparisons using Markdown generated by tools such as Microsoft MarkItDown, get-md, or Pandoc?

Do you have benchmarks comparing token usage, inference cost, and overall efficiency between a text-only embedding/RAG pipeline and PixelRAG's vision-based pipeline?

These seem like valuable comparisons for understanding the trade-offs between a "render everything as pixels" approach and a "well-structured text" approach.

I'm not the original author of these questions, I'm simply forwarding them on behalf of a community member who was interested in the project. If helpful, the original Reddit discussion is linked above.

Thanks for your work!

[andylizf]

Thanks for forwarding. Most of this is in the paper (arXiv:2606.28344), so pointers:

On the baseline: we compared six HTML-to-text extractors before picking one (Appendix D.1) and went with the best of them, Trafilatura, plus mwparserfromhell as a second baseline. Same embedding model, same FAISS index, and the inputs are token-matched: 1024-token text chunks vs ~875 visual tokens per tile (§3.2). Worth knowing that swapping Trafilatura for mwparserfromhell alone moves SimpleQA by about 10 points (§5.2), which is part of the argument: parser choice is a large, brittle variable that text RAG inherits and pixels don't have.

On Markdown: we didn't run MarkItDown or Pandoc, but Table 3 has the structure-preserving extreme, feeding raw HTML to the reader. It does worse than flat text. The markup inflates context about 3.8x and the reader spends its budget on tags instead of evidence (Appendix D.5); indexing raw HTML directly drops accuracy by up to 29% (D.6). Markdown carries less markup than HTML, so it would land somewhere between the two measured points. Honestly I'd expect it to help a little on tables and not much else. The failure analysis (Table 2, §5.2) shows where text actually loses: tables that don't survive extraction at all (evidence recall 23.8% vs 34.8%), and linearized infobox text displacing the answer paragraph at retrieval time. Markdown headings fix neither; it's still linearized text in embedding space.

Token usage and cost are measured, not asserted. Figure 3 plots accuracy against input tokens across four readers. Table 5 lists per-model token counts for both modalities. §5.5 covers image compression: a fine-tuned 2x-compressed reader roughly matches native-resolution accuracy, and reaches the uncompressed k=5 ceiling at a third of the token cost. In the agentic setting (§5.4), pixel retrieval used 3.6M prompt tokens vs 37.5M for text on MoNaCo.

That said, the Markdown baseline is a fair experiment and the eval harness is public and self-contained (eval/ in this repo). If your community member wants to run it: swap the extraction step, rebuild the text index, run the same cells. Happy to answer questions if anyone tries it.