# Website to Markdown and RAG Dataset Crawler (`orbiscribe/website-rag-dataset-builder`) Actor Crawl public websites into clean Markdown, text, metadata, links, JSON-LD, and chunks for RAG and knowledge bases. - **URL**: https://apify.com/orbiscribe/website-rag-dataset-builder.md - **Developed by:** [Orbiscribe Labs](https://apify.com/orbiscribe) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing $3.00 / 1,000 page records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website to Markdown and RAG Dataset Crawler Turn public websites into clean Markdown and RAG-ready datasets. This Actor crawls user-supplied URLs, extracts readable page content, and saves structured page records plus a JSONL chunk export with metadata for retrieval systems. Each page record includes metadata, headings, links, JSON-LD, visible emails and phones, extraction-quality stats, and text chunks. Use it for documentation sites, help centers, public knowledge bases, competitor content research, SEO audits, and AI search or chatbot projects. ### Run This First Start with a tiny docs crawl so you can inspect chunk quality before scaling: ```json { "startUrls": [{ "url": "https://docs.apify.com/academy/getting-started" }], "maxPages": 3, "maxDepth": 1, "sameDomainOnly": true, "respectRobotsTxt": true, "includeMarkdown": true, "includeHtml": false, "datasetOutputMode": "chunks", "chunkSizeChars": 1200, "chunkOverlapChars": 120, "dryRun": false } ```` Look first at `chunkId`, `text`, `canonicalUrl`, `title`, `headingPath`, and the `RAG_CHUNKS_JSONL` key-value output. A practical workflow recipe is in `docs/workflow-recipes/website-rag-dataset-pipeline.md` in the GitHub repository. ### What does this website crawler do? Website to RAG Dataset Crawler starts from one or more URLs, follows internal links up to your limits, and extracts useful page content instead of dumping raw HTML. The output is designed to be easy to export from Apify and load into a database, spreadsheet, vector store, LangChain/LlamaIndex pipeline, or internal research workflow. It does not require an LLM API key. The extraction is deterministic and keeps costs predictable. ### What data can you extract? | Field | Description | | --- | --- | | `url` | Final crawled page URL | | `canonicalUrl` | Canonical URL when present | | `title` | Page title | | `metaDescription` | Meta description | | `headings` | H1-H6 heading structure | | `mainText` | Clean readable text | | `markdown` | Markdown version of the main content | | `links` | Internal and external links found on the page | | `jsonLd` | JSON-LD/schema.org blocks | | `emails` | Email addresses visibly present on the page | | `phones` | Phone numbers visibly present on the page | | `chunks` | Text chunks with character count and token estimate | | `RAG_CHUNKS_JSONL` | Key-value output with one JSONL record per chunk | | `RAG_CHUNKS` | Key-value output with the same chunk records as JSON | | `MARKDOWN_BUNDLE` | One Markdown document combining all extracted pages | | `URL_INVENTORY` | Compact page inventory with URL, title, depth, word count, and chunk count | | `BUYER_BRIEF` | Short run brief for reviewing crawl coverage and extraction quality | | `wordCount` | Approximate word count of extracted readable text | | `markdownLength` | Character length of generated Markdown | | `linkCount` | Number of unique links included in the record | | `headingCount` | Number of extracted H1-H3 headings | | `chunkCount` | Number of generated text chunks | | `extractionMethod` | Content root used, such as `article`, `main`, or `body` | | `depth` | Crawl depth from the start URL | By default, the dataset contains one row per chunk because that is what most embedding and vector-database imports expect. Full page records are also stored in `PAGE_RECORDS`. Set `datasetOutputMode` to `pages` if you prefer one dataset row per crawled page, or `both` if you want both shapes in the dataset. ### Quick start 1. Add one or more start URLs. 2. Set `maxPages` and `maxDepth` before the first run. 3. Keep `sameDomainOnly` enabled unless you want to follow external links. 4. Keep `respectRobotsTxt` enabled for normal public-site crawling. 5. Start with `maxPages: 10`, inspect the output, then scale. For a docs site, use the docs homepage or sitemap page as the start URL. For a single-page extraction, set `maxDepth: 0`. ### Use With n8n, Make, or Zapier Run the Actor with wait-for-finish enabled, then read the default dataset items or the `RAG_CHUNKS_JSONL` key-value output. Typical workflow: 1. Trigger from a new website URL, docs URL, or scheduled refresh. 2. Run this Actor with a small `maxPages` limit. 3. Send chunk rows to your vector database, spreadsheet, or agent knowledge store. 4. Store `canonicalUrl`, `title`, and `chunkId` so answers can cite sources. ### Input example ```json { "startUrls": [{ "url": "https://example.com/" }], "maxPages": 25, "maxDepth": 2, "sameDomainOnly": true, "respectRobotsTxt": true, "includeMarkdown": true, "includeHtml": false, "datasetOutputMode": "chunks", "chunkSizeChars": 2500, "chunkOverlapChars": 250 } ``` ### Output example ```json { "url": "https://example.com/", "canonicalUrl": "https://example.com/", "title": "Example Domain", "metaDescription": "Example page used for documentation and tests.", "headings": [ { "level": 1, "text": "Example Domain" } ], "mainText": "Example Domain This domain is for use in illustrative examples in documents.", "markdown": "## Example Domain\n\nThis domain is for use in illustrative examples in documents.", "links": [ { "url": "https://www.iana.org/domains/example", "text": "More information", "internal": false } ], "jsonLd": [], "emails": [], "phones": [], "chunks": [ { "chunkId": "https://example.com/#chunk-0", "text": "## Example Domain\n\nThis domain is for use in illustrative examples in documents.", "charCount": 77, "tokenEstimate": 20 } ], "wordCount": 12, "markdownLength": 77, "linkCount": 1, "headingCount": 1, "chunkCount": 1, "extractionMethod": "main", "depth": 0 } ``` ### Pricing This Actor uses pay-per-event pricing. Dry-run examples are not charged. Apify free-plan users get the first 25 page records without this Actor's custom event charge; after that, normal pay-per-event pricing and the user's run spending limit apply. | Event | Price | What counts | | --- | --- | --- | | `page-record` | `$0.003` | One crawled page with extracted text, metadata, links, and chunks | That is `$3 per 1,000` emitted page records, plus normal Apify platform usage. Use `maxPages`, `maxDepth`, and `sameDomainOnly` to control cost. ### Tips for better crawls - Start small. A `maxPages: 10` run usually tells you whether the site structure works. - Use `maxDepth: 0` for a fixed list of URLs. - Use `sameDomainOnly: true` to avoid crawling unrelated sites. - Set `includeHtml: false` unless you need source HTML. - Shorter chunks are easier to embed; longer chunks keep more context. - Use `RAG_CHUNKS_JSONL` when your downstream pipeline wants one JSON object per line for embeddings or batch import. - Use `datasetOutputMode: "pages"` when you want a spreadsheet-style page inventory instead of chunk rows. - Check `wordCount`, `markdownLength`, and `extractionMethod` to spot thin or poorly structured pages. - Failed URLs are recorded in `RUN_SUMMARY`. ### Limits and compliance This Actor crawls public pages reachable from user-supplied URLs. It does not log in, bypass paywalls, solve CAPTCHAs, or access private systems. The `respectRobotsTxt` option applies best-effort `User-agent: *` disallow rules for start domains with a short robots.txt timeout. Buyers are responsible for checking site terms and permitted use of crawled content. # Actor input Schema ## `startUrls` (type: `array`): Web pages or sites to crawl. ## `maxPages` (type: `integer`): Maximum number of pages to crawl. ## `maxDepth` (type: `integer`): Maximum link depth from each start URL. ## `sameDomainOnly` (type: `boolean`): Only enqueue links on the same registered host as a start URL. ## `respectRobotsTxt` (type: `boolean`): Fetch and apply simple User-agent \* robots.txt disallow rules for start domains. ## `includeMarkdown` (type: `boolean`): Include markdown converted from the main page content. ## `includeHtml` (type: `boolean`): Include cleaned HTML in each dataset item. ## `datasetOutputMode` (type: `string`): Choose whether dataset rows should be RAG chunks, page records, or both. Full page records are always stored in PAGE\_RECORDS. ## `chunkSizeChars` (type: `integer`): Approximate character length for RAG chunks. ## `chunkOverlapChars` (type: `integer`): Character overlap between adjacent chunks. ## `dryRun` (type: `boolean`): Emit one deterministic example.com sample without crawling. ## Actor input object example ```json { "startUrls": [ { "url": "https://example.com/" } ], "maxPages": 25, "maxDepth": 2, "sameDomainOnly": true, "respectRobotsTxt": true, "includeMarkdown": true, "includeHtml": false, "datasetOutputMode": "chunks", "chunkSizeChars": 2500, "chunkOverlapChars": 250, "dryRun": false } ``` # Actor output Schema ## `results` (type: `string`): No description ## `ragChunksJsonl` (type: `string`): No description ## `ragChunks` (type: `string`): No description ## `pageRecords` (type: `string`): No description ## `runSummary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://example.com/" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("orbiscribe/website-rag-dataset-builder").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://example.com/" }] } # Run the Actor and wait for it to finish run = client.actor("orbiscribe/website-rag-dataset-builder").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://example.com/" } ] }' | apify call orbiscribe/website-rag-dataset-builder --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=orbiscribe/website-rag-dataset-builder", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website to Markdown and RAG Dataset Crawler", "description": "Crawl public websites into clean Markdown, text, metadata, links, JSON-LD, and chunks for RAG and knowledge bases.", "version": "0.1", "x-build-id": "395nUI35PxekWi8Yb" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/orbiscribe~website-rag-dataset-builder/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-orbiscribe-website-rag-dataset-builder", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/orbiscribe~website-rag-dataset-builder/runs": { "post": { "operationId": "runs-sync-orbiscribe-website-rag-dataset-builder", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/orbiscribe~website-rag-dataset-builder/run-sync": { "post": { "operationId": "run-sync-orbiscribe-website-rag-dataset-builder", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Web pages or sites to crawl.", "default": [ { "url": "https://example.com/" } ], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxPages": { "title": "Max pages", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Maximum number of pages to crawl.", "default": 25 }, "maxDepth": { "title": "Max depth", "minimum": 0, "maximum": 10, "type": "integer", "description": "Maximum link depth from each start URL.", "default": 2 }, "sameDomainOnly": { "title": "Same domain only", "type": "boolean", "description": "Only enqueue links on the same registered host as a start URL.", "default": true }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Fetch and apply simple User-agent * robots.txt disallow rules for start domains.", "default": true }, "includeMarkdown": { "title": "Include markdown", "type": "boolean", "description": "Include markdown converted from the main page content.", "default": true }, "includeHtml": { "title": "Include HTML", "type": "boolean", "description": "Include cleaned HTML in each dataset item.", "default": false }, "datasetOutputMode": { "title": "Dataset output mode", "enum": [ "chunks", "pages", "both" ], "type": "string", "description": "Choose whether dataset rows should be RAG chunks, page records, or both. Full page records are always stored in PAGE_RECORDS.", "default": "chunks" }, "chunkSizeChars": { "title": "Chunk size chars", "minimum": 500, "maximum": 12000, "type": "integer", "description": "Approximate character length for RAG chunks.", "default": 2500 }, "chunkOverlapChars": { "title": "Chunk overlap chars", "minimum": 0, "maximum": 2000, "type": "integer", "description": "Character overlap between adjacent chunks.", "default": 250 }, "dryRun": { "title": "Dry run", "type": "boolean", "description": "Emit one deterministic example.com sample without crawling.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```