# Website & PDF to RAG JSONL Crawler (`orbiscribe/linked-pdf-website-rag-crawler`) Actor Paste webpage and PDF URLs and get Markdown, JSONL chunks, PDF inventory, source warnings, and RAG-ready records. - **URL**: https://apify.com/orbiscribe/linked-pdf-website-rag-crawler.md - **Developed by:** [Orbiscribe Labs](https://apify.com/orbiscribe) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing $1.00 / 1,000 web or pdf source processeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website & PDF to RAG JSONL Crawler Use this Actor when the useful knowledge is split between webpages and linked PDF manuals, whitepapers, policy documents, reports, or help files. It fetches public pages, optionally discovers linked PDFs, extracts machine-readable PDF text, and writes one mixed-source RAG dataset with source type and warnings. ### Why use this instead of a generic crawler? Generic website crawlers often stop at HTML or hide PDF extraction failures. This Actor makes PDFs first-class sources, keeps a PDF inventory, and emits warnings when a file has no machine-readable text. - paste webpage and PDF URLs - keep the first crawl small with low live defaults - filter web paths with `includeUrlPatterns` - export `MIXED_RAG_CHUNKS_JSONL` for vector pipelines - inspect `PDF_INVENTORY` and `PDF_WARNINGS` before trusting the corpus ### What you get - Dataset rows for web pages, PDF documents, and chunks. - Source type, discovered-from URL, Markdown, main text, content hash, word count, and extraction warnings. - Key-value outputs: `RAG_CHUNKS_JSONL`, `MIXED_RAG_CHUNKS_JSONL`, `DOCUMENTS_JSONL`, `PDF_INVENTORY`, `PDF_WARNINGS`, `SOURCE_INVENTORY`, `MARKDOWN_BUNDLE`, `BUYER_BRIEF`, and `RUN_SUMMARY`. ### Common workflows - Build a knowledge base from product docs plus linked PDF manuals. - Convert vendor compliance pages and policy PDFs into one dataset. - Audit which PDFs were discovered and which lacked machine-readable text. - Export mixed-source JSONL for retrieval with source-type filtering. ### Input Provide `startUrls`, direct `pdfUrls`, or both. Keep `discoverLinkedPdfs` enabled to follow PDF links from fetched pages. `maxPdfs` is enforced globally across direct and discovered PDFs. The default input runs a tiny live webpage and PDF sample: ```json { "startUrls": [{ "url": "https://docs.apify.com/academy/getting-started" }], "pdfUrls": [ "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf" ], "includeUrlPatterns": ["/academy/"], "excludeUrlPatterns": [], "discoverLinkedPdfs": true, "maxPages": 1, "maxPdfs": 1, "dryRun": false } ```` Use `dryRun: true` when you want bundled demo records without fetching live sources or calling custom pay-per-event charges. ### Pricing Recommended monetization: Pay per Event at **$0.001** per `web-pdf-rag-source`. That is **$1 per 1,000** processed webpages or PDFs, plus normal Apify platform usage. When pay-per-event pricing is enabled, dry runs are uncharged and free-plan callers get the first 25 processed sources without this Actor's custom event charge. Users should still set Apify spending limits before large crawls. ### Limits and compliance Public webpages and PDFs only. This Actor does not bypass logins, paywalls, robots policies, or access controls. PDF extraction is for machine-readable text; OCR is not included in this MVP. # Actor input Schema ## `startUrls` (type: `array`): Public website pages to crawl for HTML content and PDF links. ## `includeUrlPatterns` (type: `array`): Only crawl webpage URLs containing at least one of these strings. Leave empty to allow all start URLs. ## `excludeUrlPatterns` (type: `array`): Skip webpage or discovered PDF URLs containing any of these strings. ## `pdfUrls` (type: `array`): Optional direct URLs to machine-readable PDFs. ## `discoverLinkedPdfs` (type: `boolean`): Discover PDF links from fetched webpages and process them up to maxPdfs. ## `maxPages` (type: `integer`): Maximum number of HTML pages to fetch in one run. ## `maxPdfs` (type: `integer`): Maximum number of PDFs to process globally in one run. ## `chunkSizeChars` (type: `integer`): Target character length for generated text chunks. ## `chunkOverlapChars` (type: `integer`): Character overlap between adjacent chunks. ## `requestTimeoutMs` (type: `integer`): Per-request timeout in milliseconds. ## `dryRun` (type: `boolean`): Emit bundled demo web/PDF records without fetching or charging events. ## Actor input object example ```json { "startUrls": [ { "url": "https://docs.apify.com/academy/getting-started" } ], "includeUrlPatterns": [ "/academy/" ], "excludeUrlPatterns": [], "pdfUrls": [ "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf" ], "discoverLinkedPdfs": true, "maxPages": 1, "maxPdfs": 1, "chunkSizeChars": 2500, "chunkOverlapChars": 250, "requestTimeoutMs": 20000, "dryRun": false } ``` # Actor output Schema ## `results` (type: `string`): No description ## `ragChunksJsonl` (type: `string`): No description ## `mixedChunksJsonl` (type: `string`): No description ## `documentsJsonl` (type: `string`): No description ## `pdfInventory` (type: `string`): No description ## `pdfWarnings` (type: `string`): No description ## `sourceInventory` (type: `string`): No description ## `markdownBundle` (type: `string`): No description ## `buyerBrief` (type: `string`): No description ## `runSummary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("orbiscribe/linked-pdf-website-rag-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("orbiscribe/linked-pdf-website-rag-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call orbiscribe/linked-pdf-website-rag-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=orbiscribe/linked-pdf-website-rag-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website & PDF to RAG JSONL Crawler", "description": "Paste webpage and PDF URLs and get Markdown, JSONL chunks, PDF inventory, source warnings, and RAG-ready records.", "version": "0.1", "x-build-id": "53KaboboBwJHJAAJJ" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/orbiscribe~linked-pdf-website-rag-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-orbiscribe-linked-pdf-website-rag-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/orbiscribe~linked-pdf-website-rag-crawler/runs": { "post": { "operationId": "runs-sync-orbiscribe-linked-pdf-website-rag-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/orbiscribe~linked-pdf-website-rag-crawler/run-sync": { "post": { "operationId": "run-sync-orbiscribe-linked-pdf-website-rag-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Website start URLs", "type": "array", "description": "Public website pages to crawl for HTML content and PDF links.", "default": [ { "url": "https://docs.apify.com/academy/getting-started" } ], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "includeUrlPatterns": { "title": "Include URL patterns", "type": "array", "description": "Only crawl webpage URLs containing at least one of these strings. Leave empty to allow all start URLs.", "default": [ "/academy/" ], "items": { "type": "string" } }, "excludeUrlPatterns": { "title": "Exclude URL patterns", "type": "array", "description": "Skip webpage or discovered PDF URLs containing any of these strings.", "default": [], "items": { "type": "string" } }, "pdfUrls": { "title": "PDF URLs", "type": "array", "description": "Optional direct URLs to machine-readable PDFs.", "default": [ "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf" ], "items": { "type": "string" } }, "discoverLinkedPdfs": { "title": "Discover linked PDFs", "type": "boolean", "description": "Discover PDF links from fetched webpages and process them up to maxPdfs.", "default": true }, "maxPages": { "title": "Max webpages", "minimum": 0, "maximum": 1000, "type": "integer", "description": "Maximum number of HTML pages to fetch in one run.", "default": 1 }, "maxPdfs": { "title": "Max PDFs", "minimum": 0, "maximum": 200, "type": "integer", "description": "Maximum number of PDFs to process globally in one run.", "default": 1 }, "chunkSizeChars": { "title": "Chunk size characters", "minimum": 500, "maximum": 12000, "type": "integer", "description": "Target character length for generated text chunks.", "default": 2500 }, "chunkOverlapChars": { "title": "Chunk overlap characters", "minimum": 0, "maximum": 2000, "type": "integer", "description": "Character overlap between adjacent chunks.", "default": 250 }, "requestTimeoutMs": { "title": "Request timeout ms", "minimum": 1000, "maximum": 60000, "type": "integer", "description": "Per-request timeout in milliseconds.", "default": 20000 }, "dryRun": { "title": "Dry run", "type": "boolean", "description": "Emit bundled demo web/PDF records without fetching or charging events.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```