PDF Harvester

Finds PDFs linked from a list of URLs and extracts their text and metadata — point it at pages, not PDF links.

You give it pages (a listing, a publications page, a report index — anything with links on it), and it finds every PDF linked from them, downloads each one, and returns its text and metadata as a clean dataset you can download as JSON, CSV or Excel, or pull through the API. HTTP-only, no browser, so runs are fast and cheap.

The incumbents compared for this Actor all expect you to already have the PDF URLs. This one does the step before that: you give it the pages a PDF is normally linked from, and it finds the PDF itself — including PDFs served from URLs with no .pdf extension at all (arXiv is a well-known example) and PDFs hosted on a different domain from the page that links them (as on Wikimedia Commons).

What data you get

For every PDF found, the Actor returns:

Field	Description
url	Same as pdfUrl below, kept for consistency with this catalogue's other Actors
scrapedAt	Same as fetchedAt below
pdfUrl	The direct URL of the PDF file
foundOn	The page this PDF was linked from, or null when the PDF URL was given directly as input
title	The PDF's own metadata title, or the link text it was found under, or null
pages	Number of pages in the PDF
sizeBytes	Size of the downloaded PDF file, in bytes
text	Extracted text, truncated to maxTextLength characters (see Limitations)
metadata	{ producer, created, modified } from the PDF's own info dictionary
fetchedAt	ISO 8601 timestamp of when this PDF was fetched and extracted

Input

Option	Type	Default	Description
urls	array	-	Pages to scan for linked PDFs. A URL that is itself a PDF is also accepted directly.
maxDepth	integer	0	0: only the given URLs. 1: also follow same-host links found on those pages, once.
extractText	boolean	true	Read each PDF's text. Page count and metadata are always read regardless of this option.
maxTextLength	integer	20000	Truncate each PDF's extracted text to this many characters.
maxItems	integer	100	Stop after this many PDFs are extracted (max 10,000). Also your cost cap.
proxyConfiguration	object	Apify datacenter	Proxy settings
maxPdfs	integer	100	Advanced. Hard cap on distinct PDF links this run will ever try to fetch.
maxPdfSizeBytes	integer	20971520 (20 MB)	Advanced. PDFs larger than this are skipped, not downloaded in full.
maxRequestsPerCrawl	integer	a formula	Advanced. Hard cap on pages fetched while looking for PDF links.

Example input:

{
    "urls": [
        { "url": "https://commons.wikimedia.org/wiki/File:Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf" }
    ],
    "maxItems": 50
}

Output example

{
    "url": "https://upload.wikimedia.org/wikipedia/commons/a/ad/Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf?utm_source=commons.wikimedia.org&utm_campaign=index&utm_content=original",
    "scrapedAt": "2026-09-22T20:29:22.643Z",
    "pdfUrl": "https://upload.wikimedia.org/wikipedia/commons/a/ad/Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf?utm_source=commons.wikimedia.org&utm_campaign=index&utm_content=original",
    "foundOn": "https://commons.wikimedia.org/wiki/File:Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf",
    "title": "Ornamental AGM Plants April 2023",
    "pages": 129,
    "sizeBytes": 1116900,
    "text": "AGM Plants April 2023 (c) RHS - ORNAMENTAL  Abelia (Caprifoliaceae) ...",
    "metadata": {
        "producer": "Adobe PDF Library 23.1.125",
        "created": "2023-04-06T10:57:13.000Z",
        "modified": "2023-04-11T09:32:28.000Z"
    },
    "fetchedAt": "2026-09-22T20:29:22.643Z"
}

A run summary (items pushed, items skipped by category, pages fetched, failed requests by category, stop reason) is stored in the run's key-value store under the key OUTPUT.

Pricing

Pay per result. You are charged per result and nothing else: no platform usage, no compute units. Pages fetched while discovering PDFs, retries and failed requests are all free. The current rate is on the Pricing tab of this Actor — that is the only place it is set, so no figure is repeated here.

• maxItems caps the number of PDFs extracted, and therefore the cost, of a run.
• The run also stops when it reaches the maximum total charge you set for the run in Apify Console or through the API.

Tips

• Start with maxItems: 10 and maxDepth: 0 to check the fields before a large run.
• If your pages don't have PDFs linked directly but link to other pages that do (a publications index that links year pages, say), set maxDepth: 1.
• If you only need to know a PDF exists (title, page count, size) and not its text, set extractText: false — it skips the slower per-page text extraction.

Limitations

• Pages that require a login are not accessible and are not scraped.
• PDFs with no text layer — a scanned page saved as an image, for instance — return an empty text. This Actor does not perform OCR.
• PDF discovery relies on a link's URL ending in .pdf, an explicit type="application/pdf" attribute, or its visible text/title containing the word "PDF". A PDF linked with none of these signals will not be found. Every candidate is confirmed by inspecting the downloaded bytes before it is parsed, so a link that merely looks like a PDF never produces a wrong result — at most it costs one extra, cheap request that shows up under skipped in the run summary, not in your data.
• At maxDepth: 1, at most 25 same-host links per given page are followed, to keep the crawl bounded.
• PDFs over maxPdfSizeBytes (20 MB by default) are skipped, not truncated.

Source caveats. This Actor has no single source — you choose the pages. What each PDF says about itself (draft status, publication date, whether it is the final version) is for you to judge from its content; this Actor does not interpret it.

Data and compliance

You choose the pages, and you are responsible for what you point this Actor at. This Actor fetches only what you give it (plus, at maxDepth: 1, same-host pages linked from those), and:

• It reads and honours robots.txt for every host it touches — the pages you list and every host a PDF is downloaded from, which is often a different host from the page that links it. It does not bypass logins, paywalls or other access controls.
• It identifies itself with a plain openrows/pdf-harvester User-Agent and sends no headers that imitate a browser.
• It sends few requests in parallel and waits between requests to the same host.
• Fields that could name a person are dropped. A PDF's Author metadata field is never copied into the output — only producer (the software that produced the file), created and modified are. title and text, in contrast, pass through unfiltered: title is either the PDF's own Title metadata field or the link text it was found under, and text is the PDF's own body content — both are what you asked this Actor to fetch, your chosen content, not data this Actor collected about people. If the pages you choose to point this Actor at contain personal data in their PDFs — a CV, a scanned form, a document whose title happens to be a person's name — that is a property of your input, not of this Actor's behaviour, and is your responsibility to handle.

How robots.txt is handled

Before the first page on a host is fetched, the Actor fetches that host's robots.txt once and obeys it for the rest of the run — for every host, including the ones your PDFs turn out to be hosted on rather than the pages you listed. Rules are read for the product token openrows, falling back to the * group when the file does not name us, with * and $ wildcards and the standard "longest matching rule wins" precedence. A Crawl-delay set for us is honoured, up to 10 seconds. When a page redirects, the URL the redirect actually leads to is checked again against its own host's rules before anything is read from it, so a redirect cannot carry the Actor onto a site, or a path, that its owner puts off limits — the normal shape for a PDF download, not a corner case.

• A disallowed URL is never fetched for its own sake, and its body is never read. A URL you gave directly, or a PDF link discovered on a page, is checked before it is requested at all. A URL that is only reached because something else redirected to it is a partial exception: the redirect is already followed by the time its target's rules can be checked, so that one response is fetched — but discarded unparsed, before anything is read from it, the moment the check fails. Either way it is reported under skipped.robotsDisallowed in the run summary, so you can see exactly how many of your input URLs (or the PDFs linked from them) the site puts off limits.
• No robots.txt (404 or 410) means no rules, and the host is crawled normally. It is counted as robotsAbsent in the summary.
• A robots.txt we cannot read means the host is skipped, not crawled. Any other response — 401, 403, 429, a 5xx, a timeout, a network failure, or a 200 that turns out to be an error or login page — leaves us without the site's rules, and this Actor will not guess. Every URL on that host is skipped and counted under skipped.robotsUnknown, and the host and the status that stopped us are listed under robots.unknownHosts in the summary. If a host you supplied returns no results, look there first.

Removal requests. If you believe this Actor exposes data it should not, open an issue on the Actor's Issues tab and state the URL(s) concerned. Requests are answered there, and the fields or pages in question are removed.

You are responsible for using the extracted data in line with the terms of the sites you point this Actor at and the laws that apply to you.

Support

Report bugs and request fields on the Issues tab of this Actor. Include the run ID.