PDF to RAG-Ready Dataset

Convert public, native-text PDFs into clean, token-aware chunks for retrieval-augmented generation (RAG), LLM pipelines, semantic search, document Q&A, and knowledge-base ingestion.

Each successful PDF produces one Apify Dataset row per chunk with normalized text, conservative Markdown, page references, token counts, PDF metadata, and a stable SHA-256 document ID.

V1 supports native-text PDFs only. OCR is not included. Each PDF is limited to 100 pages and 25 MB.

What this Actor does

• Downloads 1-20 public HTTP(S) PDF URLs per run.
• Extracts native PDF text with pypdf.
• Normalizes whitespace, ligatures, and soft hyphens.
• Splits text into deterministic, token-aware chunks using tiktoken.
• Tracks the first and last source page represented in every chunk.
• Returns normalized plain text, conservative Markdown, or both.
• Adds PDF metadata and a stable documentId based on the PDF file bytes.
• Skips duplicate PDF content within the same run.
• Emits structured error rows for rejected or failed PDFs when failFast is false.

The Actor does not generate embeddings, call an LLM, write to a vector database, or perform OCR. Its Dataset output is designed to feed those downstream steps.

Input

Example

{
  "pdfUrls": [
    "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
  ],
  "chunkSizeTokens": 1000,
  "chunkOverlapTokens": 150,
  "tokenEncoding": "cl100k_base",
  "maxPagesPerPdf": 100,
  "maxFileSizeMb": 25,
  "includeMarkdown": true,
  "includePlainText": true,
  "failFast": false
}

Fields

Field	Type	Default	Allowed values	Description
pdfUrls	array of strings	required	1-20 public HTTP(S) URLs	PDFs to process. Embedded credentials and private-network destinations are rejected.
chunkSizeTokens	integer	1000	500-2000	Maximum target size of each chunk in tokenizer tokens.
chunkOverlapTokens	integer	150	0-400	Tokens repeated between adjacent chunks.
tokenEncoding	string	cl100k_base	cl100k_base, o200k_base	Tokenizer used for chunk sizing and tokenCount.
maxPagesPerPdf	integer	100	1-100	Reject a PDF when its page count exceeds this value. V1 cannot exceed 100 pages.
maxFileSizeMb	integer	25	1-25	Reject a download when it exceeds this value. V1 cannot exceed 25 MB.
includeMarkdown	boolean	true	true, false	Include conservative Markdown for each chunk.
includePlainText	boolean	true	true, false	Include normalized plain text for each chunk.
failFast	boolean	false	true, false	Stop at the first failed PDF instead of adding an error row and continuing.

Output

Results are written to the run's default Apify Dataset. A successful PDF produces one row per chunk.

Successful chunk example

{
  "documentId": "a-stable-sha256-hash-of-the-pdf-bytes",
  "sourceUrl": "https://example.com/document.pdf",
  "filename": "document.pdf",
  "title": "Example document",
  "author": null,
  "pageCount": 12,
  "chunkIndex": 0,
  "chunkCount": 8,
  "pageStart": 1,
  "pageEnd": 2,
  "tokenCount": 987,
  "text": "Normalized text for this chunk...",
  "markdown": "<!-- pages:1-2 -->\n\nNormalized text for this chunk...",
  "metadata": {
    "contentType": "application/pdf",
    "fileSizeBytes": 12345,
    "pdfProducer": null,
    "pdfCreationDate": null,
    "tokenEncoding": "cl100k_base"
  },
  "error": null
}

Output fields

Field	Description
documentId	SHA-256 hash of the downloaded PDF bytes. Identical PDF content receives the same ID.
sourceUrl	Final validated PDF URL after permitted redirects.
filename	Filename derived from the URL.
title, author	PDF metadata when available; otherwise null.
pageCount	Number of pages in the PDF.
chunkIndex	Zero-based position of this chunk within the document.
chunkCount	Total chunks produced for the document.
pageStart, pageEnd	Inclusive one-based page range represented in the chunk.
tokenCount	Token count under the selected tokenEncoding.
text	Normalized plain text, or null when disabled.
markdown	Conservative Markdown with a deterministic page comment, or null when disabled.
metadata	Content type, file size, PDF producer/date when available, and tokenizer encoding.
error	null for successful chunks; a structured error object for failed inputs.

Failed PDF example

With failFast: false, a rejected or failed PDF produces a structured error row and the Actor continues with the next URL.

{
  "sourceUrl": "https://example.com/not-a-pdf.pdf",
  "pageCount": 0,
  "chunkCount": 0,
  "tokenCount": 0,
  "text": null,
  "markdown": null,
  "metadata": {},
  "error": {
    "code": "NOT_A_PDF",
    "message": "The downloaded resource is not a valid PDF."
  }
}

Possible error categories include security rejection, invalid or non-PDF content, file-size or page-limit rejection, password protection, extraction failure, OCR required, and general processing failure.

Pricing and billing behavior

• $0.03 per successfully processed unique PDF through the custom document_processed event.
• The standard low-priced Apify Actor-start event remains enabled.
• Dataset rows are chunks, so apify-default-dataset-item is intentionally removed and chunks are not billed individually.
• Duplicate PDF content within the same run is skipped without a document_processed charge.
• Rejected or failed PDFs are designed to produce error rows without a document_processed charge.
• Successful chunk rows are stored before the custom document event is charged.

Apify displays the applicable event prices and run limits before execution.

Common RAG and LLM use cases

• Prepare public reports, manuals, papers, and documentation for embedding pipelines.
• Build page-aware document Q&A and citation workflows.
• Create source records for semantic search or a vector database.
• Supply chunked context to agents and LLM applications.
• Preprocess batches of PDFs for summarization, classification, or extraction in a separate downstream step.
• Create repeatable Dataset exports for ETL and knowledge-base workflows.

Limitations

• Native-text PDFs only. OCR is not included in V1. Scanned or image-only PDFs return OCR_REQUIRED when too little text can be extracted.
• Maximum 100 pages per PDF.
• Maximum 25 MB per PDF.
• Maximum 20 PDF URLs per run.
• Only public HTTP(S) URLs are supported. Private-network URLs and URLs containing credentials are rejected.
• Password-protected PDFs are not supported.
• Markdown conversion is intentionally conservative. It does not reconstruct complex tables, figures, or page layouts.
• Multi-column documents and complex PDF reading order may not extract perfectly because PDF text order depends on the source file.
• Deduplication applies within a single run and is based on identical downloaded PDF bytes.
• No OCR, embeddings, vector-database writes, LLM calls, or semantic enrichment are included.

Tips

• Start with the defaults (1000 tokens and 150 overlap) for general RAG ingestion.
• Use o200k_base only when your downstream tokenizer expects it.
• Keep failFast disabled for batch jobs so one bad URL does not prevent later PDFs from being processed.
• Use documentId, chunkIndex, and page fields as stable downstream identifiers and citation metadata.

Support

If you find a reproducible edge case, open an issue on the Actor and include a public test URL when possible. Do not post private documents or credentials.