PDF Parser Picker: compare Python PDF libraries on your files

I gave five Python PDF libraries the same five odd test files from Mozilla's pdf.js repo. On issue10175.pdf, an insurance letter, pypdfium2 returned J,I, <cl /33/. ) Q]‘a]b[hcb while pypdf read it cleanly. On tracemonkey.pdf, a 14-page paper, pdfplumber and markitdown dropped so many spaces that 62% of the letters ended up in run-together tokens like AndreasGal∗+,BrendanEich∗,MikeShaver∗. On an Arabic sample the three libraries built on pdfminer.six returned the words with their letters in reverse order. None of the five got a word out of filled-background.pdf, and pypdf raised an error on it.

This Actor runs that comparison on your documents, because the best library for research papers is not the best one for invoices. That is the run behind this table:

File	Pages	pypdf	pypdfium2	pdfplumber	pdfminer.six	markitdown
issue10175.pdf	1	98.5	33.4	93	92.7	92.7
filled-background.pdf (no text in any parser, not ranked)	3	fail	empty	empty	empty	empty
tracemonkey.pdf	14	98.3	97.8	57.2	98.3	57.2
alphatrans.pdf	1	88.3	88.3	88.3	88.3	88.3
ArabicCIDTrueType.pdf	1	100	100	40	40	40

Rank	Parser	Mean score	Failures	Total seconds
1	pypdf 6.21.4	96.3	0	1.3
2	pypdfium2 5.14.0	79.9	0	0.11
3	pdfminer.six 20260107	79.8	0	2.6
4	markitdown 0.1.9	69.5	0	4.3
5	pdfplumber 0.11.10	69.6	0	4.8

Recommendation for that set: "Use pypdf: mean score 96.3/100, 0 failure(s), 1.296 s in total." On the two clean IRS files in the default input every library scores about 98 and the pick comes down to speed (pypdfium2). Ordinary text PDFs rarely separate these libraries. The odd ones do, so test with the files you actually have.

What it runs

Library	Licence	Engine
pypdf	BSD-3-Clause	pure Python
pypdfium2	Apache-2.0 / BSD-3-Clause	PDFium, the PDF engine in Chrome
pdfminer.six	MIT	pure Python
pdfplumber	MIT	built on pdfminer.six
markitdown	MIT	Microsoft's converter, built on pdfplumber and pdfminer.six for PDFs

Each library uses the plain recipe from its own docs (for example page.extract_text() for every page), the same code as the Reader board. Each one runs in its own process, one file and one parser at a time, with a hard timeout (default 60 s) and a memory cap of 75% of the run's memory. A parser that hangs is killed from outside and counted as a failure for that file. The rest of the run carries on.

Two well-known libraries are left out on purpose. PyMuPDF is fast and good, but it is AGPL-3.0, so using it inside a hosted service can oblige you to publish your own code or buy a licence from Artifex; I kept this tool to permissive licences, so its advice is something you can ship without a lawyer. Docling needs PyTorch and its layout models, a far bigger install and far more CPU per page than the other five, which is too heavy for a quick per-file comparison. Both may come back as options later.

How the score works

There is no answer key for your files, so the score measures agreement between libraries, not correctness. Failed or empty output scores 0. Otherwise the score is out of 100:

Part	Weight	What it measures
Agreement	60%	Share of the "consensus words" this parser also found. A consensus word is one that at least two other independent engines found.
Cleanliness	25%	Share of tokens that look like words (letters only, 2 to 20 long; numbers don't count either way), minus 5% for each 1% of junk characters: (cid:12) codes, U+FFFD, control and private-use characters.
Spacing	15%	Penalises letters inside tokens longer than 25 characters, which is what lost spaces look like (ThisIsWhatGluedTextLooksLike).

pdfplumber and markitdown both run on pdfminer.six, so when they vote on the consensus words, the three count as one engine. Without that, the pdfminer.six family would outvote pypdf and pypdfium2 on everything. Words are compared after Unicode NFKC normalisation, so ligatures and Arabic presentation forms match plain letters. If only one parser produces text, there is nothing to agree with and the score uses cleanliness and spacing alone.

The weakness is easy to state. If every engine makes the same mistake, nobody is penalised for it. If one parser finds text that all the others miss, such as form field values, it gets no credit for that. Read the preview field (first 200 characters of each parser's text) before you trust a close result.

The recommendation is the parser with the highest mean score across the files you gave it. Scores within about a point count as a tie; then fewer failures wins, then the faster parser. Files where no parser found any text (usually scans) are listed but not ranked, because they say nothing about the libraries, only that you need OCR. Seconds are parse time with the library already imported.

Input

Field	Default	What it does
PDF URLs (startUrls)	two IRS forms	Direct links to PDFs.
More PDF URLs (fileUrls)		The same as a plain list of strings, handy from code.
Parsers to compare (parsers)	all five	Keep at least three, or the agreement check has little to work with.
Timeout per parser per file (timeoutPerParserSecs)	60	A parser still working after this is killed and counted as failed for that file.
Max files (maxFiles)	20	Only the first this-many URLs are checked. Also caps the cost.

Output

One dataset item per file, then one summary item. The dataset has three views: scores, seconds and the recommendation. A readable REPORT.md with the ranking, a score table and notes goes to the key-value store.

{
  "type": "file",
  "url": "https://example.com/issue10175.pdf",
  "status": "ok",
  "pages": 1,
  "bestParser": "pypdf",
  "bestScore": 98.5,
  "scores": {"pypdf": 98.5, "pypdfium2": 33.4, "pdfplumber": 93.0, "pdfminer": 92.7, "markitdown": 92.7},
  "parsers": {
    "pypdfium2": {
      "ok": true, "error": null, "chars": 996, "seconds": 0.005, "score": 33.4,
      "agreement": 0.082, "cleanliness": 0.547, "spacing": 0.988,
      "notes": ["found 8% of the 122 words the other engines agree on",
                "many tokens don't look like words (letter-spaced or garbled text)"],
      "preview": "J,I, <cl /33/. ) Q]‘a]b[hcb >? ) /7664+7030 ..."
    }
  }
}

The summary item has "type": "summary", the ranking (mean score, failures, empty results, files won, total and median seconds per parser), the recommendation sentence and the exact library versions.

Pricing

You pay $0.004 for each checked file, with all five parsers included, plus $0.00005 when the run starts. A download that fails, a file that isn't a PDF, or a PDF none of the parsers can open costs nothing. A scan with no text layer is charged, since every parser did run on it. When your run reaches its spending limit, the Actor stops starting new files.

Use it from code

curl -X POST "https://api.apify.com/v2/acts/prometheus-maker~pdf-parser-picker/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"fileUrls": ["https://www.irs.gov/pub/irs-pdf/fw9.pdf", "https://arxiv.org/pdf/1706.03762"]}'

Limits

PDFs only for now; Word files are planned. No OCR, so scans come back empty for every parser. Files up to 100 MB. Text from scripts without spaces (Chinese, Japanese, Thai) is split per character for the agreement check, which works but is coarser than for English. The score says nothing about tables, headings or reading order beyond what shows up in the words and spacing.

For how the same libraries hold up on more than 1,000 deliberately strange PDFs, see the Reader board, the public benchmark this tool is built on. If you then want clean Markdown out of your PDFs, PDF & Document to Markdown does that.