# PDF to RAG-Ready Dataset (`utilityforgelab/pdf-to-rag-ready-dataset`) Actor

Convert public text-based PDFs into clean, token-aware RAG chunks with Markdown, page references, metadata, stable document IDs, and ready-to-use Apify Dataset output.

- **URL**: https://apify.com/utilityforgelab/pdf-to-rag-ready-dataset.md
- **Developed by:** [UtilityForgeLab](https://apify.com/utilityforgelab) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $30.00 / 1,000 document processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## PDF to RAG-Ready Dataset

Convert public, native-text PDFs into clean, token-aware chunks for retrieval-augmented generation (RAG), LLM pipelines, semantic search, document Q\&A, and knowledge-base ingestion.

Each successful PDF produces one Apify Dataset row per chunk with normalized text, conservative Markdown, page references, token counts, PDF metadata, and a stable SHA-256 document ID.

> **V1 supports native-text PDFs only. OCR is not included. Each PDF is limited to 100 pages and 25 MB.**

### What this Actor does

- Downloads 1-20 public HTTP(S) PDF URLs per run.
- Extracts native PDF text with `pypdf`.
- Normalizes whitespace, ligatures, and soft hyphens.
- Splits text into deterministic, token-aware chunks using `tiktoken`.
- Tracks the first and last source page represented in every chunk.
- Returns normalized plain text, conservative Markdown, or both.
- Adds PDF metadata and a stable `documentId` based on the PDF file bytes.
- Skips duplicate PDF content within the same run.
- Emits structured error rows for rejected or failed PDFs when `failFast` is `false`.

The Actor does **not** generate embeddings, call an LLM, write to a vector database, or perform OCR. Its Dataset output is designed to feed those downstream steps.

### Input

#### Example

```json
{
  "pdfUrls": [
    "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
  ],
  "chunkSizeTokens": 1000,
  "chunkOverlapTokens": 150,
  "tokenEncoding": "cl100k_base",
  "maxPagesPerPdf": 100,
  "maxFileSizeMb": 25,
  "includeMarkdown": true,
  "includePlainText": true,
  "failFast": false
}
```

#### Fields

| Field | Type | Default | Allowed values | Description |
| --- | --- | --- | --- | --- |
| `pdfUrls` | array of strings | required | 1-20 public HTTP(S) URLs | PDFs to process. Embedded credentials and private-network destinations are rejected. |
| `chunkSizeTokens` | integer | `1000` | 500-2000 | Maximum target size of each chunk in tokenizer tokens. |
| `chunkOverlapTokens` | integer | `150` | 0-400 | Tokens repeated between adjacent chunks. |
| `tokenEncoding` | string | `cl100k_base` | `cl100k_base`, `o200k_base` | Tokenizer used for chunk sizing and `tokenCount`. |
| `maxPagesPerPdf` | integer | `100` | 1-100 | Reject a PDF when its page count exceeds this value. V1 cannot exceed 100 pages. |
| `maxFileSizeMb` | integer | `25` | 1-25 | Reject a download when it exceeds this value. V1 cannot exceed 25 MB. |
| `includeMarkdown` | boolean | `true` | `true`, `false` | Include conservative Markdown for each chunk. |
| `includePlainText` | boolean | `true` | `true`, `false` | Include normalized plain text for each chunk. |
| `failFast` | boolean | `false` | `true`, `false` | Stop at the first failed PDF instead of adding an error row and continuing. |

### Output

Results are written to the run's default Apify Dataset. A successful PDF produces one row per chunk.

#### Successful chunk example

```json
{
  "documentId": "a-stable-sha256-hash-of-the-pdf-bytes",
  "sourceUrl": "https://example.com/document.pdf",
  "filename": "document.pdf",
  "title": "Example document",
  "author": null,
  "pageCount": 12,
  "chunkIndex": 0,
  "chunkCount": 8,
  "pageStart": 1,
  "pageEnd": 2,
  "tokenCount": 987,
  "text": "Normalized text for this chunk...",
  "markdown": "<!-- pages:1-2 -->\n\nNormalized text for this chunk...",
  "metadata": {
    "contentType": "application/pdf",
    "fileSizeBytes": 12345,
    "pdfProducer": null,
    "pdfCreationDate": null,
    "tokenEncoding": "cl100k_base"
  },
  "error": null
}
```

#### Output fields

| Field | Description |
| --- | --- |
| `documentId` | SHA-256 hash of the downloaded PDF bytes. Identical PDF content receives the same ID. |
| `sourceUrl` | Final validated PDF URL after permitted redirects. |
| `filename` | Filename derived from the URL. |
| `title`, `author` | PDF metadata when available; otherwise `null`. |
| `pageCount` | Number of pages in the PDF. |
| `chunkIndex` | Zero-based position of this chunk within the document. |
| `chunkCount` | Total chunks produced for the document. |
| `pageStart`, `pageEnd` | Inclusive one-based page range represented in the chunk. |
| `tokenCount` | Token count under the selected `tokenEncoding`. |
| `text` | Normalized plain text, or `null` when disabled. |
| `markdown` | Conservative Markdown with a deterministic page comment, or `null` when disabled. |
| `metadata` | Content type, file size, PDF producer/date when available, and tokenizer encoding. |
| `error` | `null` for successful chunks; a structured error object for failed inputs. |

#### Failed PDF example

With `failFast: false`, a rejected or failed PDF produces a structured error row and the Actor continues with the next URL.

```json
{
  "sourceUrl": "https://example.com/not-a-pdf.pdf",
  "pageCount": 0,
  "chunkCount": 0,
  "tokenCount": 0,
  "text": null,
  "markdown": null,
  "metadata": {},
  "error": {
    "code": "NOT_A_PDF",
    "message": "The downloaded resource is not a valid PDF."
  }
}
```

Possible error categories include security rejection, invalid or non-PDF content, file-size or page-limit rejection, password protection, extraction failure, OCR required, and general processing failure.

### Pricing and billing behavior

- **$0.03 per successfully processed unique PDF** through the custom `document_processed` event.
- The standard low-priced Apify Actor-start event remains enabled.
- Dataset rows are chunks, so `apify-default-dataset-item` is intentionally removed and chunks are **not** billed individually.
- Duplicate PDF content within the same run is skipped without a `document_processed` charge.
- Rejected or failed PDFs are designed to produce error rows without a `document_processed` charge.
- Successful chunk rows are stored before the custom document event is charged.

Apify displays the applicable event prices and run limits before execution.

### Common RAG and LLM use cases

- Prepare public reports, manuals, papers, and documentation for embedding pipelines.
- Build page-aware document Q\&A and citation workflows.
- Create source records for semantic search or a vector database.
- Supply chunked context to agents and LLM applications.
- Preprocess batches of PDFs for summarization, classification, or extraction in a separate downstream step.
- Create repeatable Dataset exports for ETL and knowledge-base workflows.

### Limitations

- **Native-text PDFs only. OCR is not included in V1.** Scanned or image-only PDFs return `OCR_REQUIRED` when too little text can be extracted.
- Maximum **100 pages per PDF**.
- Maximum **25 MB per PDF**.
- Maximum **20 PDF URLs per run**.
- Only public HTTP(S) URLs are supported. Private-network URLs and URLs containing credentials are rejected.
- Password-protected PDFs are not supported.
- Markdown conversion is intentionally conservative. It does not reconstruct complex tables, figures, or page layouts.
- Multi-column documents and complex PDF reading order may not extract perfectly because PDF text order depends on the source file.
- Deduplication applies within a single run and is based on identical downloaded PDF bytes.
- No OCR, embeddings, vector-database writes, LLM calls, or semantic enrichment are included.

### Tips

- Start with the defaults (`1000` tokens and `150` overlap) for general RAG ingestion.
- Use `o200k_base` only when your downstream tokenizer expects it.
- Keep `failFast` disabled for batch jobs so one bad URL does not prevent later PDFs from being processed.
- Use `documentId`, `chunkIndex`, and page fields as stable downstream identifiers and citation metadata.

### Support

If you find a reproducible edge case, open an issue on the Actor and include a public test URL when possible. Do not post private documents or credentials.

# Actor input Schema

## `pdfUrls` (type: `array`):

One to twenty public HTTP(S) PDF URLs. Private-network URLs and embedded credentials are rejected.

## `chunkSizeTokens` (type: `integer`):

Maximum target size of each text chunk in tokenizer tokens.

## `chunkOverlapTokens` (type: `integer`):

Number of tokens repeated between adjacent chunks.

## `tokenEncoding` (type: `string`):

Tokenizer encoding used for chunk sizing and token counts.

## `maxPagesPerPdf` (type: `integer`):

Reject PDFs with more pages than this safety cap.

## `maxFileSizeMb` (type: `integer`):

Reject downloads larger than this safety cap in megabytes.

## `includeMarkdown` (type: `boolean`):

Include conservative Markdown for each chunk.

## `includePlainText` (type: `boolean`):

Include normalized plain text for each chunk.

## `failFast` (type: `boolean`):

Stop the run on the first rejected or failed PDF instead of emitting an error record and continuing.

## Actor input object example

```json
{
  "chunkSizeTokens": 1000,
  "chunkOverlapTokens": 150,
  "tokenEncoding": "cl100k_base",
  "maxPagesPerPdf": 100,
  "maxFileSizeMb": 25,
  "includeMarkdown": true,
  "includePlainText": true,
  "failFast": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

Default Dataset items produced by this Actor run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("utilityforgelab/pdf-to-rag-ready-dataset").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("utilityforgelab/pdf-to-rag-ready-dataset").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call utilityforgelab/pdf-to-rag-ready-dataset --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,utilityforgelab/pdf-to-rag-ready-dataset"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/P710RM7J7Hf7nOtt7/builds/13j27PksS5zIb3u0E/openapi.json