# PDF Harvester (`openrows/pdf-harvester`) Actor

Finds PDFs linked from a list of URLs and extracts their text and metadata — point it at pages, not PDF links.

- **URL**: https://apify.com/openrows/pdf-harvester.md
- **Developed by:** [openrows](https://apify.com/openrows) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$6.00 / 1,000 pdf documents

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## PDF Harvester

Finds PDFs linked from a list of URLs and extracts their text and metadata — point it at pages,
not PDF links.

You give it pages (a listing, a publications page, a report index — anything with links on it), and
it finds every PDF linked from them, downloads each one, and returns its text and metadata as a clean
dataset you can download as JSON, CSV or Excel, or pull through the API. HTTP-only, no browser, so
runs are fast and cheap.

The incumbents compared for this Actor all expect you to already have the PDF URLs. This one does the
step before that: you give it the pages a PDF is normally *linked from*, and it finds the PDF itself —
including PDFs served from URLs with no `.pdf` extension at all (arXiv is a well-known example) and
PDFs hosted on a different domain from the page that links them (as on Wikimedia Commons).

### What data you get

For every PDF found, the Actor returns:

| Field       | Description                                                                               |
| ----------- | ----------------------------------------------------------------------------------------- |
| `url`       | Same as `pdfUrl` below, kept for consistency with this catalogue's other Actors           |
| `scrapedAt` | Same as `fetchedAt` below                                                                 |
| `pdfUrl`    | The direct URL of the PDF file                                                            |
| `foundOn`   | The page this PDF was linked from, or `null` when the PDF URL was given directly as input |
| `title`     | The PDF's own metadata title, or the link text it was found under, or `null`              |
| `pages`     | Number of pages in the PDF                                                                |
| `sizeBytes` | Size of the downloaded PDF file, in bytes                                                 |
| `text`      | Extracted text, truncated to `maxTextLength` characters (see Limitations)                 |
| `metadata`  | `{ producer, created, modified }` from the PDF's own info dictionary                      |
| `fetchedAt` | ISO 8601 timestamp of when this PDF was fetched and extracted                             |

### Input

| Option                | Type    | Default            | Description                                                                              |
| --------------------- | ------- | ------------------ | ---------------------------------------------------------------------------------------- |
| `urls`                | array   | -                  | Pages to scan for linked PDFs. A URL that is itself a PDF is also accepted directly.     |
| `maxDepth`            | integer | `0`                | `0`: only the given URLs. `1`: also follow same-host links found on those pages, once.   |
| `extractText`         | boolean | `true`             | Read each PDF's text. Page count and metadata are always read regardless of this option. |
| `maxTextLength`       | integer | `20000`            | Truncate each PDF's extracted text to this many characters.                              |
| `maxItems`            | integer | `100`              | Stop after this many PDFs are extracted (max 10,000). Also your cost cap.                |
| `proxyConfiguration`  | object  | Apify datacenter   | Proxy settings                                                                           |
| `maxPdfs`             | integer | `100`              | Advanced. Hard cap on distinct PDF links this run will ever try to fetch.                |
| `maxPdfSizeBytes`     | integer | `20971520` (20 MB) | Advanced. PDFs larger than this are skipped, not downloaded in full.                     |
| `maxRequestsPerCrawl` | integer | a formula          | Advanced. Hard cap on pages fetched while looking for PDF links.                         |

Example input:

```json
{
    "urls": [
        { "url": "https://commons.wikimedia.org/wiki/File:Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf" }
    ],
    "maxItems": 50
}
```

### Output example

```json
{
    "url": "https://upload.wikimedia.org/wikipedia/commons/a/ad/Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf?utm_source=commons.wikimedia.org&utm_campaign=index&utm_content=original",
    "scrapedAt": "2026-09-22T20:29:22.643Z",
    "pdfUrl": "https://upload.wikimedia.org/wikipedia/commons/a/ad/Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf?utm_source=commons.wikimedia.org&utm_campaign=index&utm_content=original",
    "foundOn": "https://commons.wikimedia.org/wiki/File:Award-of-Garden-Merit-Plants-April-2023-Ornamentals.pdf",
    "title": "Ornamental AGM Plants April 2023",
    "pages": 129,
    "sizeBytes": 1116900,
    "text": "AGM Plants April 2023 (c) RHS - ORNAMENTAL  Abelia (Caprifoliaceae) ...",
    "metadata": {
        "producer": "Adobe PDF Library 23.1.125",
        "created": "2023-04-06T10:57:13.000Z",
        "modified": "2023-04-11T09:32:28.000Z"
    },
    "fetchedAt": "2026-09-22T20:29:22.643Z"
}
```

A run summary (items pushed, items skipped by category, pages fetched, failed requests by category,
stop reason) is stored in the run's key-value store under the key `OUTPUT`.

### Pricing

**Pay per result.** You are charged per result and nothing else: no platform usage, no compute units.
Pages fetched while discovering PDFs, retries and failed requests are all free. The current rate is on
the *Pricing* tab of this Actor — that is the only place it is set, so no figure is repeated here.

- `maxItems` caps the number of PDFs extracted, and therefore the cost, of a run.
- The run also stops when it reaches the *maximum total charge* you set for the run in Apify Console
  or through the API.

### Tips

- Start with `maxItems: 10` and `maxDepth: 0` to check the fields before a large run.
- If your pages don't have PDFs linked directly but link to *other* pages that do (a publications
  index that links year pages, say), set `maxDepth: 1`.
- If you only need to know a PDF exists (title, page count, size) and not its text, set
  `extractText: false` — it skips the slower per-page text extraction.

### Limitations

- Pages that require a login are not accessible and are not scraped.
- PDFs with no text layer — a scanned page saved as an image, for instance — return an empty `text`.
  This Actor does not perform OCR.
- PDF discovery relies on a link's URL ending in `.pdf`, an explicit `type="application/pdf"`
  attribute, or its visible text/title containing the word "PDF". A PDF linked with none of these
  signals will not be found. Every candidate is confirmed by inspecting the downloaded bytes before
  it is parsed, so a link that merely *looks* like a PDF never produces a wrong result — at most it
  costs one extra, cheap request that shows up under `skipped` in the run summary, not in your data.
- At `maxDepth: 1`, at most 25 same-host links per given page are followed, to keep the crawl bounded.
- PDFs over `maxPdfSizeBytes` (20 MB by default) are skipped, not truncated.

**Source caveats.** This Actor has no single source — you choose the pages. What each PDF says about
itself (draft status, publication date, whether it is the final version) is for you to judge from its
content; this Actor does not interpret it.

### Data and compliance

**You choose the pages, and you are responsible for what you point this Actor at.** This Actor fetches
only what you give it (plus, at `maxDepth: 1`, same-host pages linked from those), and:

- It reads and honours `robots.txt` for **every host it touches** — the pages you list and every host
  a PDF is downloaded from, which is often a different host from the page that links it. It does not
  bypass logins, paywalls or other access controls.
- It identifies itself with a plain `openrows/pdf-harvester` User-Agent and sends no headers that
  imitate a browser.
- It sends few requests in parallel and waits between requests to the same host.
- **Fields that could name a person are dropped.** A PDF's `Author` metadata field is never copied into
  the output — only `producer` (the software that produced the file), `created` and `modified` are.
  `title` and `text`, in contrast, pass through unfiltered: `title` is either the PDF's own `Title`
  metadata field or the link text it was found under, and `text` is the PDF's own body content —
  both are what you asked this Actor to fetch, your chosen content, not data this Actor collected
  about people. If the pages you choose to point this Actor at contain personal data in their PDFs —
  a CV, a scanned form, a document whose title happens to be a person's name — that is a property of
  your input, not of this Actor's behaviour, and is your responsibility to handle.

#### How robots.txt is handled

Before the first page on a host is fetched, the Actor fetches that host's `robots.txt` once and obeys
it for the rest of the run — for every host, including the ones your PDFs turn out to be hosted on
rather than the pages you listed. Rules are read for the product token `openrows`, falling back to the
`*` group when the file does not name us, with `*` and `$` wildcards and the standard "longest matching
rule wins" precedence. A `Crawl-delay` set for us is honoured, up to 10 seconds. When a page redirects,
the URL the redirect actually leads to is checked again against its own host's rules before anything is
read from it, so a redirect cannot carry the Actor onto a site, or a path, that its owner puts off
limits — the normal shape for a PDF download, not a corner case.

- **A disallowed URL is never fetched for its own sake, and its body is never read.** A URL you gave
  directly, or a PDF link discovered on a page, is checked before it is requested at all. A URL that
  is only reached because something else redirected to it is a partial exception: the redirect is
  already followed by the time its target's rules can be checked, so that one response is fetched —
  but discarded unparsed, before anything is read from it, the moment the check fails. Either way it
  is reported under `skipped.robotsDisallowed` in the run summary, so you can see exactly how many of
  your input URLs (or the PDFs linked from them) the site puts off limits.
- **No `robots.txt` (404 or 410) means no rules**, and the host is crawled normally. It is counted as
  `robotsAbsent` in the summary.
- **A `robots.txt` we cannot read means the host is skipped, not crawled.** Any other response — 401,
  403, 429, a 5xx, a timeout, a network failure, or a 200 that turns out to be an error or login page —
  leaves us without the site's rules, and this Actor will not guess. Every URL on that host is skipped
  and counted under `skipped.robotsUnknown`, and the host and the status that stopped us are listed
  under `robots.unknownHosts` in the summary. If a host you supplied returns no results, look there
  first.

**Removal requests.** If you believe this Actor exposes data it should not, open an issue on the Actor's
*Issues* tab and state the URL(s) concerned. Requests are answered there, and the fields or pages in
question are removed.

You are responsible for using the extracted data in line with the terms of the sites you point this
Actor at and the laws that apply to you.

### Support

Report bugs and request fields on the *Issues* tab of this Actor. Include the run ID.

# Actor input Schema

## `urls` (type: `array`):

Pages to scan for linked PDFs. A URL that is itself a PDF is also accepted and extracted directly. You choose the sources and are responsible for them (see the README's Data and compliance section).

## `maxDepth` (type: `integer`):

0: only scan the given URLs for PDF links. 1: also follow same-host links found on those pages, one level deep, and scan those pages for PDF links too. No other values are used.

## `extractText` (type: `boolean`):

Read each PDF's text content. Page count and metadata (producer, created, modified) are always read; turning this off skips only the (slower) per-page text extraction, so text comes back null.

## `maxTextLength` (type: `integer`):

Truncate each PDF's extracted text to this many characters. Ignored when Extract text is off.

## `maxItems` (type: `integer`):

Stop after this many PDFs are extracted. You are charged per result, so this is also your cost cap. Maximum 10,000 per run; split a bigger job across runs.

## `proxyConfiguration` (type: `object`):

Proxy settings. Apify datacenter proxies are the default and are enough for most targets.

## `maxPdfs` (type: `integer`):

Hard cap on how many PDFs this run will process once confirmed to actually be a PDF (by content, not just by URL). A link that looked like a PDF but wasn't, or that robots.txt disallowed, does not count against this cap. Keeps a large, PDF-heavy site from being fetched beyond what Max results actually needs.

## `maxPdfSizeBytes` (type: `integer`):

PDFs larger than this are skipped (counted under skipped.filteredOut in the run summary) rather than downloaded and parsed in full.

## `maxRequestsPerCrawl` (type: `integer`):

Hard cap on pages fetched while looking for PDF links (separate from Max PDFs to fetch). Defaults to a formula based on the number of URLs given and Max PDFs to fetch.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxDepth": 0,
  "extractText": true,
  "maxTextLength": 20000,
  "maxItems": 100,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxPdfs": 100,
  "maxPdfSizeBytes": 20971520
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("openrows/pdf-harvester").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [{ "url": "https://docs.apify.com/" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("openrows/pdf-harvester").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call openrows/pdf-harvester --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,openrows/pdf-harvester"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6Zot9QSCykzwGEoEw/builds/lhXgjDmoY9lPMbbSP/openapi.json