Document to RAG-Ready Markdown

Turn PDFs into clean markdown and retrieval-ready chunks. Built for people feeding institutional documents into RAG pipelines and LLMs.

Why this instead of a plain text extractor

Naive PDF text extraction produces text that looks fine and then quietly poisons a RAG index: running headers repeat on every chunk, tables collapse into unreadable word soup, and chunks get cut in the middle of sentences.

Measured against naive extraction (PyMuPDF get_text) on the same documents:

Document	Metric	Naive	This actor
IRS Form 1040 instructions (126 pp)	chunks cut mid-sentence	468 / 733	37 / 740
IRS Form 1040 instructions	tables recovered as markdown	0	89
IRS Form 1040 instructions	fragmented lines	20.5%	14.7%
NIST AI RMF (48 pp)	fragmented lines	14.0%	6.2%
ResNet paper, 2-column (12 pp)	fragmented lines	42.5%	18.7%
ResNet paper	tables recovered	0	21
YOLO paper, 2-column (10 pp)	fragmented lines	37.9%	20.5%
BERT paper, 2-column (16 pp)	fragmented lines	50.4%	31.8%
IRS Form 1040 instructions	token recall vs source	-	99.2%

Sentence integrity (a stricter check: share of sentences that read as complete and unbroken) is at parity with naive extraction on two-column papers — 96.5% vs 96.6% (ResNet), 92.1% vs 92.2% (YOLO), 85.5% vs 89.0% (BERT). We do not claim to improve reading order; we claim cleaner structure at no cost to it.

Where it does not help: clean single-column academic papers. On one arXiv paper our fragmentation was 8.2% vs naive 5.0% — if your corpus is single-column preprints, a simple extractor is fine and you should save your money.

What it does

• Reading order - detects two-column layouts and reads columns in order.
• Boilerplate removal - drops headers/footers that repeat across pages.
• Tables as markdown - table regions are excluded from body text, so tabular numbers don't contaminate prose.
• Headings - inferred from font size (not bold, which is used for emphasis).
• De-hyphenation - rejoins words and sentences broken across lines.
• Chunking - breaks on headings, never mid-sentence, with section, page_start, page_end metadata and configurable overlap.

Limitations (please read before paying)

• No OCR. Scanned PDFs without a text layer produce little or no text. The run reports this in its status message.
• Complex multi-column tables and heavy visual layouts extract imperfectly.
• Verify output against the source before relying on it for anything consequential. See TERMS.md.

Input

Field	Type	Default	Notes
pdfUrls	array	-	Direct PDF links you have the right to process
chunkSize	int	1800	Target characters per chunk
chunkOverlap	int	150	Character overlap between chunks
extractTables	bool	true	Emit tables as markdown
removeBoilerplate	bool	true	Strip repeating headers/footers
emitChunks	bool	true	Chunks (true) or whole pages (false)
maxPages	int	0	0 = all pages

Output

Chunk mode: {source, section, page_start, page_end, chars, text} Page mode: {source, page, columns, markdown, tables[]} Failures push {source, error} and are not charged.

Pricing

Pay per event: $0.02 per document + $0.001 per page. You are charged only for documents that extract successfully; failed URLs cost nothing.

Licence

AGPL-3.0. This is a network service, so you are entitled to the Corresponding Source: the complete source is at https://github.com/petergrillo/doc-to-rag-markdown — see also LICENSE and NOTICE. Uses PyMuPDF (AGPL-3.0 / Artifex commercial); not affiliated with Artifex.