# Website PDF Finder & Text Extractor (`metallic_pastry/website-pdf-extractor`) Actor

Find every PDF on a website and extract clean text, markdown, metadata and RAG-ready chunks. Also takes direct PDF links and Google Drive or Dropbox shares. Tracks new, changed and removed PDFs between runs. Scanned PDFs without text are free.

- **URL**: https://apify.com/metallic_pastry/website-pdf-extractor.md
- **Developed by:** [Simon Liarsgård](https://apify.com/metallic_pastry) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 pdf extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website PDF Finder & Text Extractor

Give it a website, a page or a PDF link. It finds every PDF linked from the site and turns each one into clean text, with metadata and optional RAG-ready chunks. Other PDF extractors need you to collect the PDF URLs yourself. This one crawls the site for you, so `https://www.example.com/investors` is all you need to get the annual reports, presentations and policies linked there.

Each PDF is one dataset row:

- **Text:** markdown with detected headings and lists, or plain text. Running headers, footers and page numbers are removed.
- **Basics:** title, page count, word count, file size and SHA-256.
- **Document properties:** author, dates, producer, language and form fields.
- **Provenance:** the page the PDF was found on and its link text.

Turn on change tracking and a scheduled run tells you which PDFs are new, changed or removed since the previous run. Scanned PDFs with no text layer, broken files and links that turn out not to be PDFs are free.

### What you can use it for

- **RAG and AI knowledge bases.** Load a company's manuals, datasheets, policies or research into a vector database. Chunks split at headings, and each one carries its page range and heading path, so answers can cite the page.
- **Investor relations and competitor research.** Pull annual reports, sustainability reports and presentations from IR pages and read them, search them or summarise them with an LLM.
- **Document monitoring.** Run weekly on tender portals, regulator pages, price lists, terms and conditions or safety data sheet libraries, and get only the PDFs that are new or changed.
- **Document inventories and audits.** Switch text off to list every PDF on a site with its size, source page and last-modified header, at a tenth of the price.
- **Lead generation and sales research.** Extract the text of brochures, catalogues and case studies to find products, partners and contacts.
- **Direct PDF text.** Paste PDF links, or Google Drive and Dropbox share links, and get the text back with no crawling.

### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | required | One per line. **Web pages and websites** are crawled for PDF links. **Direct PDF links** are extracted as they are. Google Drive and Dropbox share links are converted to direct downloads. Duplicates are removed. Invalid lines get a free error row. |
| `extractText` | `true` | Off = listing mode. PDFs are found and listed with their URL, link text, source page, size and SHA-256, but not read. |
| `textFormat` | `markdown` | `markdown` keeps headings (`#`, `##`, `###`, detected from font size) and bullet lists. `plain` gives the same paragraphs without markup. |
| `chunkSize` | `0` | Set to 200-20000 to also get a `chunks` list (see Output). 1000-2000 suits most embedding models. |
| `includePages` | `false` | Also return the text of each page in a `pages` list. |
| `removeHeadersFooters` | `true` | Drop lines that repeat at the top or bottom of most pages, and page numbers. |
| `crawlDepth` | `1` | How many clicks from the start page to look for PDFs. 0 = only the start page. Pages that look like reports, downloads, publications or investors are visited first. |
| `maxPagesPerSite` | `25` | Max HTML pages fetched per start URL. Crawling is free. |
| `maxPdfsPerSite` | `50` | Max PDFs taken from one start URL. |
| `includeKeywords` | empty | Keep only PDFs whose URL or link text contains one of these words, e.g. `annual`, `2025`, `datasheet`. |
| `excludeKeywords` | empty | Skip PDFs whose URL or link text contains any of these words. |
| `allowOtherHosts` | `true` | Include PDFs hosted on another domain, such as a CDN or document host. Pages are always crawled on the same site only. |
| `maxPagesPerPdf` | `500` | Stop reading a PDF after this many pages. |
| `maxPdfSizeMb` | `50` | Skip larger files (free `too_large` row). |
| `trackChanges` | `false` | Remember every PDF by SHA-256 and mark it `new`, `changed` or `unchanged` on the next run. Report PDFs that disappeared. |
| `onlyNewOrChanged` | `false` | With change tracking: skip the text of unchanged PDFs and charge only the listing price for them. |
| `stateKey` | `default` | Separate change history per schedule or list. |
| `concurrency` | `5` | Start URLs processed in parallel. |

Example: annual and sustainability reports from an investor page, as markdown chunks.

```json
{
    "startUrls": ["https://www.example.com/investors/reports"],
    "includeKeywords": ["annual", "sustainability"],
    "crawlDepth": 1,
    "chunkSize": 1500
}
```

### Output

One row per PDF. A start URL that failed or had no PDFs also gets one row, which is free.

```json
{
    "input": "https://www.irs.gov/forms-pubs/about-form-w-9",
    "status": "extracted",
    "pdfUrl": "https://www.irs.gov/pub/irs-pdf/fw9.pdf",
    "sourcePage": "https://www.irs.gov/forms-pubs/about-form-w-9",
    "linkText": "Form W-9 PDF",
    "fileName": "fw9.pdf",
    "title": "Form W-9 (Rev. March 2024)",
    "titleSource": "metadata",
    "pageCount": 6,
    "pagesExtracted": 6,
    "extractionStoppedEarly": null,
    "charCount": 37828,
    "wordCount": 6268,
    "pagesWithoutText": 0,
    "likelyScanned": false,
    "headerFooterLinesRemoved": 5,
    "textFormat": "markdown",
    "text": "Form W-9(Rev. March 2024)\n\n## Request for Taxpayer Identification Number and Certification\n\n...",
    "chunkCount": 33,
    "chunks": [
        { "index": 0, "text": "...", "pageStart": 1, "pageEnd": 1, "heading": "Request for Taxpayer Identification Number and Certification", "charCount": 1480 }
    ],
    "metadata": {
        "title": "Form W-9 (Rev. March 2024)",
        "author": "SE:W:CAR:MP",
        "subject": "Request for Taxpayer Identification Number and Certification",
        "keywords": "Fillable",
        "creator": "Designer 6.5",
        "producer": "Designer 6.5",
        "language": "en-US",
        "createdAt": "2024-03-06T13:18:13.000Z",
        "modifiedAt": "2024-03-06T13:18:13.000Z",
        "pdfVersion": "1.7",
        "hasForm": true
    },
    "fileSizeBytes": 140815,
    "sha256": "2d420cbb4123dcf1fb82595b2359cfbb5d81f00b9df9d359fcc7af361d093f53",
    "contentType": "application/pdf",
    "lastModified": null,
    "changeStatus": null,
    "httpStatus": 200,
    "errorCode": null,
    "checkedAt": "2026-10-30T12:00:00.000Z"
}
```

- **`title`** comes from the PDF's own title. Junk titles such as "Microsoft Word - draft3.docx" are skipped. After that, the actor tries the first large heading, then the link text, then the file name. `titleSource` says which one was used.
- **`chunks`** are split at headings and paragraphs, never mid-sentence unless a single sentence is longer than the chunk size. A new chunk starts at a heading once the current one is reasonably full. `heading` is the path of headings above the chunk, joined with `>`.
- **`likelyScanned`** is `true` when there is almost no text per page, which usually means a scanned document. Rows with no text at all get status `no_text` and are free. OCR is not included.
- **`extractionStoppedEarly`** is one of:
  - `page_limit`: hit `maxPagesPerPdf`;
  - `time_limit`: very slow files are cut off after about two minutes of parsing;
  - `charge_limit`: the run's max charge was reached;
  - `size_limit`: the text was cut to fit the 9 MB dataset row limit, after per-page text and chunks were dropped.
- **With change tracking on:**
  - `changeStatus` is `new`, `changed` or `unchanged`;
  - `previousSha256`, `firstSeenAt` and `previousSeenAt` show the history;
  - PDFs that disappeared from a fully crawled site get a free row with status `removed`.

The dataset has views for **PDFs**, **Text**, **Metadata**, **Changes** and **Problems**. The `RUN_SUMMARY` record in the key-value store counts:

- pages crawled;
- PDFs found, extracted and listed;
- failures;
- changes;
- charged events.

#### Status values

| Status | Meaning | Charged |
|---|---|---|
| `extracted` | Text and metadata extracted. | yes |
| `listed` | Listing mode: found and checked, not read. | listing price |
| `unchanged` | Change tracking with `onlyNewOrChanged`: same file as last run, text skipped. | listing price |
| `no_text` | No text layer (scan or images only). | no |
| `encrypted` | Password protected. | no |
| `invalid_pdf` | Broken or unreadable file. | no |
| `not_pdf` | The link did not return a PDF. | no |
| `too_large` | Larger than `maxPdfSizeMb`. | no |
| `http_error`, `unreachable`, `blocked` | The server returned an error, did not answer, or refused automated access. | no |
| `blocked_by_robots` | The site's robots.txt asks crawlers to stay out. | no |
| `no_pdfs_found` | The start URL was crawled but had no matching PDF links. | no |
| `removed` | Change tracking: a PDF from the previous run is no longer linked. | no |
| `invalid_input` | The line is not a usable URL. | no |

Links that only hint at a PDF, such as a "Download (PDF)" button pointing at `/download?id=7`, are checked by reading the file header. If one turns out to be something else, it is skipped silently. The `RUN_SUMMARY` counts these skipped links.

### Pricing

Pay per result. No subscription, and crawling is free.

| Event | Price | When |
|---|---|---|
| PDF extracted | $0.003 | One PDF read, including its first 50 pages. |
| Extra PDF page | $0.00005 | Each page beyond 50, for example $0.0075 for a 200-page report. |
| PDF listed | $0.0003 | Listing mode, or an unchanged PDF skipped with `onlyNewOrChanged`. |

Examples:

- 1,000 short PDFs cost about $3.
- One 300-page annual report costs about $0.016.
- A weekly check of a 200-PDF library where 5 files changed costs about $0.07.

Nothing is charged for scans without text, errors, non-PDF links, robots.txt opt-outs, removed rows or invalid input. The actor stops cleanly when the run's maximum charge is reached and lists what it did not process in `RUN_SUMMARY`.

### How it works and what it respects

- The actor fetches only public pages and files from the URLs you give it, and the pages they link to on the same site, up to your limits. It never logs in and never fills in forms.
- robots.txt is honoured for every page and every PDF host, with the user agent `WebsitePdfExtractor`. If a site's robots.txt answers with a server error, the site is treated as closed.
- Requests to one host are spaced out, and only a few PDFs download at a time.
- PDFs are parsed with PDF.js. Script execution and font loading are switched off.

### Limits

- **No OCR.** Scanned PDFs are detected and returned free as `no_text`. Run them through an OCR tool if you need their text.
- **Pages rendered with JavaScript.** PDF links on these pages are found when they appear in the page's HTML or embedded JSON data, which covers most Next.js and React sites. Links that only appear after a click or a search form are not found. Paste the PDF URLs directly in that case.
- **Layout.** Headings are detected from font size, so documents set in one font size come out as paragraphs only. Tables come out as text rows, and multi-column layouts are read in the PDF's own text order.
- **Change tracking.** Changes are detected from file contents (SHA-256). A PDF re-exported with identical text but new internal timestamps counts as changed.
- **Bot protection.** Sites behind Cloudflare challenges or similar return a free `blocked` row.

### Support

Open an issue on the actor's Issues tab with the start URL and what you expected. Runs, input and output stay in your Apify account.

# Changelog

This Actor's version history is a separate document: https://apify.com/metallic_pastry/website-pdf-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A web page or website ("https://www.example.com/investors") is crawled for PDF links; a direct PDF link is extracted as is. Google Drive and Dropbox share links are turned into direct downloads. Duplicates are removed; invalid lines get a free error row.

## `extractText` (type: `boolean`):

On: download every PDF and return its text, metadata and page count. Off: only list the PDFs found (URL, link text, source page, size, SHA-256) at the much lower listing price, for example to build an inventory first.

## `textFormat` (type: `string`):

Markdown keeps headings (#, ##, ###, detected from font size) and bullet lists, which works well for LLMs and RAG. Plain text has the same paragraphs without markup.

## `chunkSize` (type: `integer`):

0 = off. Otherwise also return a chunks list split at headings and paragraphs, each with its page range and heading path ("Annual report > Risks > Climate"). 1000 to 2000 suits most embedding models. Minimum 200.

## `includePages` (type: `boolean`):

Also return a pages list with the text of each page, for citations or page-level search.

## `removeHeadersFooters` (type: `boolean`):

Drops lines that repeat at the top or bottom of most pages (company name, report title, confidentiality notes) and page numbers, so they do not pollute the text or the chunks.

## `crawlDepth` (type: `integer`):

How many clicks away from each start page to look for PDF links. 0 = only the start page, 1 = the start page and the pages it links to on the same site. Pages that look like reports, downloads, publications or investors are visited first.

## `maxPagesPerSite` (type: `integer`):

Upper limit on HTML pages fetched while looking for PDFs. Crawling is never charged.

## `maxPdfsPerSite` (type: `integer`):

Upper limit on PDFs taken from one start URL. Use the keyword filters to get the right ones first.

## `includeKeywords` (type: `array`):

Keep only PDFs whose URL or link text contains at least one of these words (case-insensitive), for example "annual", "2025", "datasheet". Pages whose links contain them are also crawled first.

## `excludeKeywords` (type: `array`):

Skip PDFs whose URL or link text contains any of these words, for example "cookie", "privacy", "terms".

## `allowOtherHosts` (type: `boolean`):

Many sites keep their PDFs on a CDN or document host (for example a separate investor-relations domain). Off = only PDFs on the start URL's own domain. Web pages are always crawled on the same site only.

## `maxPagesPerPdf` (type: `integer`):

Stop reading a PDF after this many pages. The first 50 pages are included in the per-PDF price; each page beyond that is charged as an extra page.

## `maxPdfSizeMb` (type: `integer`):

Larger files are skipped (free row with status too_large). Raise memory to 2048 MB or more for files above 100 MB.

## `concurrency` (type: `integer`):

How many start URLs are processed at the same time. Requests to one host are always spaced out.

## `trackChanges` (type: `boolean`):

Remember every PDF (by SHA-256) in a named key-value store and mark each one new, changed or unchanged on the next run. PDFs that disappeared from a fully crawled site get a free removed row. Schedule the Actor to monitor document libraries, tenders, policies or price lists.

## `onlyNewOrChanged` (type: `boolean`):

With change tracking on: unchanged PDFs are returned as a short unchanged row at the listing price instead of being extracted again.

## `stateKey` (type: `string`):

Keeps separate change history for separate schedules (letters, digits, - and \_). Leave empty for the default history.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.irs.gov/forms-pubs/about-form-w-9",
    "https://arxiv.org/pdf/1706.03762"
  ],
  "extractText": true,
  "textFormat": "markdown",
  "chunkSize": 0,
  "includePages": false,
  "removeHeadersFooters": true,
  "crawlDepth": 1,
  "maxPagesPerSite": 25,
  "maxPdfsPerSite": 50,
  "includeKeywords": [],
  "excludeKeywords": [],
  "allowOtherHosts": true,
  "maxPagesPerPdf": 500,
  "maxPdfSizeMb": 50,
  "concurrency": 5,
  "trackChanges": false,
  "onlyNewOrChanged": false
}
```

# Actor output Schema

## `pdfs` (type: `string`):

Default dataset, PDFs view.

## `text` (type: `string`):

Default dataset, Text view with the extracted markdown or plain text.

## `metadata` (type: `string`):

Default dataset, Metadata view.

## `changes` (type: `string`):

Default dataset, Changes view (with change tracking on).

## `runSummary` (type: `string`):

RUN_SUMMARY record: pages crawled, PDFs found and extracted, failures, changes and charged events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.irs.gov/forms-pubs/about-form-w-9",
        "https://arxiv.org/pdf/1706.03762"
    ],
    "includeKeywords": [],
    "excludeKeywords": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("metallic_pastry/website-pdf-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "https://www.irs.gov/forms-pubs/about-form-w-9",
        "https://arxiv.org/pdf/1706.03762",
    ],
    "includeKeywords": [],
    "excludeKeywords": [],
}

# Run the Actor and wait for it to finish
run = client.actor("metallic_pastry/website-pdf-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.irs.gov/forms-pubs/about-form-w-9",
    "https://arxiv.org/pdf/1706.03762"
  ],
  "includeKeywords": [],
  "excludeKeywords": []
}' |
apify call metallic_pastry/website-pdf-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,metallic_pastry/website-pdf-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BPvcN5ItNUHChttBr/builds/QCexCT9Ceee7Wi85d/openapi.json