# RAG Website Crawler — Markdown & AI Chunks (`joaosbp/website-content-crawler`) Actor Crawl websites into clean Markdown, deterministic RAG chunks, canonical metadata, content hashes, and deduplicated AI-ready datasets for vector databases and agents. - **URL**: https://apify.com/joaosbp/website-content-crawler.md - **Developed by:** [João Victor](https://apify.com/joaosbp) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## RAG Website Crawler — Markdown & AI Chunks Turn websites into **clean Markdown and deterministic, deduplicated chunks** ready for RAG pipelines, vector databases, AI agents, search indexes, and knowledge bases. > HTTP-first for low cost. Browser rendering, proxies, and asset downloads remain explicit opt-ins. ### Why use this Actor? A generic crawler returns pages. This Actor returns a practical ingestion contract: | Crawl | Clean | Prepare for AI | Control cost | |---|---|---|---| | Same-site links and depth limits | Main-content Markdown | Stable chunk IDs and metadata | HTTP mode by default | | Include/exclude URL patterns | Canonical URL and language | SHA-256 content hash | Proxy disabled by default | | Per-item error rows | Author and publication metadata | Duplicate detection | Byte, page, depth, and concurrency caps | ### Best for - **RAG and AI teams** building reliable ingestion pipelines. - **Developers** syncing documentation or help centers into vector databases. - **Agencies** exporting clean client-site content through Dataset/API/webhooks. - **Knowledge-base owners** who need canonical metadata and duplicate suppression. ### Output highlights Every page produces a traceable Dataset row with: - `url` and `canonicalUrl`; - clean `markdown`; - title, description, author, language, and publication date; - deterministic `contentHash`; - `isDuplicate` and `duplicateOf`; - chunks with stable IDs and page metadata; - optional downloaded assets; - machine-readable `errorType` for failed pages. Duplicate pages remain visible for auditing but do not create redundant chunks. ### Quick start ```json { "startUrls": [{ "url": "https://docs.example.com" }], "maxCrawlDepth": 2, "maxPages": 100, "chunkForRag": true, "chunkSize": 500, "chunkOverlap": 50, "deduplicateContent": true, "useBrowser": false, "downloadAssets": false, "proxyConfiguration": { "useApifyProxy": false } } ```` Recommended workflow: ```text Start URLs → safe HTTP crawl → main-content extraction → Markdown → canonical metadata → hash/dedupe → RAG chunks → Dataset/API ``` ### Input | Field | Default | Purpose | |---|---:|---| | `startUrls` | required | Seed pages | | `maxCrawlDepth` | `3` | Link depth from seeds | | `maxPages` | `100` | Hard page budget | | `includeUrlPatterns` | `[]` | Optional URL regex allowlist | | `excludeUrlPatterns` | `[]` | Optional URL regex blocklist | | `chunkForRag` | `true` | Create vector-ready chunks | | `chunkSize` | `500` | Approximate target tokens | | `chunkOverlap` | `50` | Approximate overlap tokens | | `deduplicateContent` | `true` | Suppress duplicate chunks | | `useBrowser` | `false` | Render JavaScript-heavy pages | | `downloadAssets` | `false` | Download up to 20 images/PDFs per page | | `proxyConfiguration` | disabled | Apify Proxy settings | | `maxConcurrency` | `10` | Parallel HTTP requests; browser mode is capped at 5 | | `requestDelay` | `500` ms | Delay between requests | | `maxContentBytes` | `1,000,000` | Maximum page body size | | `allowPrivateUrls` | `false` | Explicit internal-network override | Invalid or risky regexes are rejected before crawling. `chunkOverlap` must be smaller than `chunkSize`. ### Example page result ```json { "url": "https://docs.example.com/getting-started", "canonicalUrl": "https://docs.example.com/getting-started", "title": "Getting started", "markdown": "# Getting started\n\nInstall the SDK...", "language": "en", "contentHash": "sha256...", "isDuplicate": false, "duplicateOf": null, "useful": true, "chunks": [ { "id": ":0", "index": 0, "text": "# Getting started...", "tokenCount": 186, "metadata": { "url": "https://docs.example.com/getting-started", "title": "Getting started", "language": "en" } } ], "chunkCount": 1, "error": null } ``` `OUTPUT` separately reports attempted, technically successful, useful, duplicate, and failed pages so empty/error rows cannot inflate success metrics. ### Security and reliability Public URL fetchers can become SSRF relays. This Actor blocks local/private/special IP ranges, metadata endpoints, credentialed URLs, unsafe redirects, and unsafe browser subrequests by default. It also redacts sensitive query values in logs and caps page/asset response sizes. `allowPrivateUrls: true` is an expert override for controlled internal environments. ### Cost guidance - Keep `useBrowser: false` for static sites and documentation. - Enable browser rendering only when the raw HTML lacks useful content. - Leave proxy and assets disabled unless the target requires them. - Use URL patterns and a small `maxPages` while calibrating a new site. Actual cost depends on memory, browser use, proxy traffic, assets, and target response time. Run a small sample before scaling. ### Limitations - This Actor crawls public HTTP(S) pages; it does not log in or bypass access controls. - JavaScript-heavy sites may require browser mode. - Content quality depends on the target HTML structure. - Sites can block automated traffic or impose their own crawl rules. - DNS validation reduces SSRF risk, but absolute protection against DNS rebinding requires network-level egress controls. ### Pricing Currently kept on normal Apify platform usage to reduce adoption friction while real external demand is measured. No separate Pay-Per-Event charge is added by this Actor. ### Support For issues or feature requests, contact the author through the Apify platform and include the run ID plus a redacted sample URL. # Actor input Schema ## `startUrls` (type: `array`): Add one or more public HTTP(S) pages. The crawler follows same-host links from each starting page. ## `maxPages` (type: `integer`): Hard run-wide request budget. Start small, then increase after checking output quality. ## `maxCrawlDepth` (type: `integer`): How many same-host link levels to follow. Use 0 to process only the starting URLs. ## `chunkForRag` (type: `boolean`): Emit chunks with stable IDs and page metadata for direct vector-database ingestion. ## `chunkSize` (type: `integer`): Target maximum size of each RAG chunk. Token counts are estimated from characters. ## `chunkOverlap` (type: `integer`): Context copied between consecutive chunks. Must be smaller than chunkSize. ## `deduplicateContent` (type: `boolean`): Keep a traceable row for duplicate pages but emit chunks only for the first matching content hash. ## `includeUrlPatterns` (type: `array`): Optional case-insensitive regular expressions. A URL must match at least one when supplied (maximum 20). ## `excludeUrlPatterns` (type: `array`): Optional case-insensitive regular expressions for pages to skip (maximum 20). ## `useBrowser` (type: `boolean`): Enable only for JS-heavy sites. HTTP mode is faster and cheaper; browser concurrency is capped at 5. ## `downloadAssets` (type: `boolean`): Optionally save up to 20 discovered assets per useful page. Assets are capped at 10 MB each. ## `proxyConfiguration` (type: `object`): Optional Apify or custom proxy settings. Disabled by default for lower cost on public sites. ## `maxConcurrency` (type: `integer`): Parallel requests. Browser mode is automatically limited to 5 to control memory and cost. ## `requestDelay` (type: `integer`): Politeness delay after successfully processing a page. ## `maxContentBytes` (type: `integer`): Reject pages larger than this limit. The default is 1 MB; allowed range is 100 KB to 10 MB. ## `allowPrivateUrls` (type: `boolean`): Advanced trusted-run override. By default localhost, private networks, metadata endpoints and special IP ranges are blocked. ## Actor input object example ```json { "startUrls": [ { "url": "https://example.com" } ], "maxPages": 100, "maxCrawlDepth": 3, "chunkForRag": true, "chunkSize": 500, "chunkOverlap": 50, "deduplicateContent": true, "includeUrlPatterns": [], "excludeUrlPatterns": [], "useBrowser": false, "downloadAssets": false, "proxyConfiguration": { "useApifyProxy": false, "apifyProxyGroups": [] }, "maxConcurrency": 10, "requestDelay": 500, "maxContentBytes": 1000000, "allowPrivateUrls": false } ``` # Actor output Schema ## `datasetId` (type: `string`): ID of the Dataset containing one success or error row per attempted page. ## `datasetItems` (type: `string`): API endpoint for extracted pages, RAG chunks, duplicate markers and per-item errors. ## `output` (type: `string`): Machine-readable OUTPUT record with attempted, successful, useful, duplicate and failed counters. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("joaosbp/website-content-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("joaosbp/website-content-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call joaosbp/website-content-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=joaosbp/website-content-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "RAG Website Crawler — Markdown & AI Chunks", "description": "Crawl websites into clean Markdown, deterministic RAG chunks, canonical metadata, content hashes, and deduplicated AI-ready datasets for vector databases and agents.", "version": "0.1", "x-build-id": "v302k8tkjtZo3WNbN" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/joaosbp~website-content-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-joaosbp-website-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/joaosbp~website-content-crawler/runs": { "post": { "operationId": "runs-sync-joaosbp-website-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/joaosbp~website-content-crawler/run-sync": { "post": { "operationId": "run-sync-joaosbp-website-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Websites to crawl", "type": "array", "description": "Add one or more public HTTP(S) pages. The crawler follows same-host links from each starting page.", "default": [ { "url": "https://example.com" } ], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxPages": { "title": "Maximum pages", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Hard run-wide request budget. Start small, then increase after checking output quality.", "default": 100 }, "maxCrawlDepth": { "title": "Maximum link depth", "minimum": 0, "maximum": 10, "type": "integer", "description": "How many same-host link levels to follow. Use 0 to process only the starting URLs.", "default": 3 }, "chunkForRag": { "title": "Create RAG chunks", "type": "boolean", "description": "Emit chunks with stable IDs and page metadata for direct vector-database ingestion.", "default": true }, "chunkSize": { "title": "Target chunk size (estimated tokens)", "minimum": 100, "maximum": 2000, "type": "integer", "description": "Target maximum size of each RAG chunk. Token counts are estimated from characters.", "default": 500 }, "chunkOverlap": { "title": "Chunk overlap (estimated tokens)", "minimum": 0, "maximum": 500, "type": "integer", "description": "Context copied between consecutive chunks. Must be smaller than chunkSize.", "default": 50 }, "deduplicateContent": { "title": "Deduplicate identical content", "type": "boolean", "description": "Keep a traceable row for duplicate pages but emit chunks only for the first matching content hash.", "default": true }, "includeUrlPatterns": { "title": "Include URL patterns (advanced)", "type": "array", "description": "Optional case-insensitive regular expressions. A URL must match at least one when supplied (maximum 20).", "default": [], "items": { "type": "string" } }, "excludeUrlPatterns": { "title": "Exclude URL patterns (advanced)", "type": "array", "description": "Optional case-insensitive regular expressions for pages to skip (maximum 20).", "default": [], "items": { "type": "string" } }, "useBrowser": { "title": "Render JavaScript in a browser", "type": "boolean", "description": "Enable only for JS-heavy sites. HTTP mode is faster and cheaper; browser concurrency is capped at 5.", "default": false }, "downloadAssets": { "title": "Download images and PDFs", "type": "boolean", "description": "Optionally save up to 20 discovered assets per useful page. Assets are capped at 10 MB each.", "default": false }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Optional Apify or custom proxy settings. Disabled by default for lower cost on public sites.", "default": { "useApifyProxy": false, "apifyProxyGroups": [] } }, "maxConcurrency": { "title": "Maximum concurrency", "minimum": 1, "maximum": 50, "type": "integer", "description": "Parallel requests. Browser mode is automatically limited to 5 to control memory and cost.", "default": 10 }, "requestDelay": { "title": "Delay after each page (ms)", "minimum": 0, "maximum": 5000, "type": "integer", "description": "Politeness delay after successfully processing a page.", "default": 500 }, "maxContentBytes": { "title": "Maximum page content bytes", "minimum": 100000, "maximum": 10000000, "type": "integer", "description": "Reject pages larger than this limit. The default is 1 MB; allowed range is 100 KB to 10 MB.", "default": 1000000 }, "allowPrivateUrls": { "title": "Allow private/internal URLs (unsafe)", "type": "boolean", "description": "Advanced trusted-run override. By default localhost, private networks, metadata endpoints and special IP ranges are blocked.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```