# Website to LLM-Ready Dataset - Markdown & Chunks for RAG (`atesen-software/llm-dataset-builder`) Actor Crawl a website or docs site and get clean Markdown, token-sized semantic chunks and metadata for every page. Ready for vector databases and RAG pipelines. $1.50 per 1,000 pages. - **URL**: https://apify.com/atesen-software/llm-dataset-builder.md - **Developed by:** [Atesen Software](https://apify.com/atesen-software) (community) - **Categories:** AI, Developer tools, Agents - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $1.50 / 1,000 page extracteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Website to LLM-Ready Dataset do? **Website to LLM-Ready Dataset** crawls a website or documentation site and turns every page into a clean, structured record that you can load straight into a vector database, a RAG pipeline or a fine-tuning set. It is a **website to Markdown API** with chunking built in. For every page you get: - **Clean main content as Markdown.** Navigation, sidebars, footers, cookie banners, breadcrumbs, "edit this page" links, table-of-contents boxes and other boilerplate are removed. Headings, lists, tables and code blocks (with their language) are kept. - **Semantic chunks, ready to embed.** The Markdown is split into token-bounded chunks that follow the structure of the page instead of cutting through sentences, code blocks or tables. Chunk size and overlap are configurable. - **Metadata.** URL, title, description, headings, language, word and token counts, canonical URL, publish and modify dates, and a content hash for deduplication and change detection. It does **not** return raw HTML, and it does not bypass logins, paywalls or bot protection. It only reads public pages and always respects `robots.txt`. ### Why use Website to LLM-Ready Dataset? - **Drop-in for vector databases.** Every chunk has an `index`, its `text`, a `tokens` count and the `section` (heading path) it belongs to, so you can embed `chunk.text` and store `url`, `title` and `section` as metadata. No post-processing script needed. - **Better retrieval than raw text dumps.** Boilerplate removal means your embeddings are not polluted by menus and footers, and structure-aware chunking keeps code examples and tables in one piece. - **Polite by design.** Honest bot user agent, `robots.txt` and `Crawl-delay` support, low starting concurrency, automatic backoff on HTTP 429 and 5xx, and a shared cool-down when a server asks to slow down. - **Predictable cost.** You are charged per page that is actually delivered. Pages that are empty, duplicates, blocked or disallowed by `robots.txt` are free, and **Max pages** is a hard cap. - **Apify platform features.** Schedule runs, call it through the API, send results to your own storage with integrations and webhooks, and export the dataset as JSON, CSV, Excel or JSONL. Typical uses: AI chatbots over documentation, internal knowledge bases, RAG for support content, keeping a search index in sync with a docs site, collecting training data from sites you are allowed to use. ### What data can Website to LLM-Ready Dataset extract? One dataset item per page: | Field | Type | Description | | --------------------------- | ---------------- | ------------------------------------------------------------------- | | `url` | string | Final URL of the page after redirects | | `title` | string | Page title, taken from the main heading when available | | `description` | string | Meta description, if the page has one | | `language` | string | Language code such as `en`, from the page or detected from the text | | `headings` | array of strings | All headings in document order | | `wordCount` | integer | Words in the extracted Markdown | | `tokenCount` | integer | Tokens in the extracted Markdown (cl100k_base) | | `markdown` | string | Clean Markdown of the main content | | `chunks` | array | `{ index, text, tokens, section }` for each chunk | | `canonicalUrl` | string | Canonical URL declared by the page | | `publishedAt`, `modifiedAt` | string | ISO dates declared by the page, if any | | `contentHash` | string | Fingerprint of the Markdown, for deduplication and change detection | | `crawledAt` | string | When the page was crawled | ### How to turn a website into an LLM-ready dataset 1. Open the Actor and add one or more **Start URLs**, for example `https://docs.example.com/guide/`. 2. Set **Max pages**. This is also your cost cap. 3. Optional: choose the **Chunk size** (default 800 tokens) and **Chunk overlap** (default 100 tokens) that match your embedding model. 4. Optional: add **Include URLs** or **Exclude URLs** glob patterns to narrow the crawl. 5. Click **Start**. When the run finishes, open the **Output** tab or download the dataset as JSON, CSV or Excel. A `RUN_SUMMARY` record in the key-value store lists totals, skipped URLs and the settings used. ### How much will it cost to turn a website into an LLM-ready dataset? The Actor uses **pay-per-event** pricing: one `page` event is charged for every page that is pushed to the dataset. The current price is shown on the Pricing tab. Pages that are skipped (empty, duplicate, link-only index pages, HTTP errors, blocked by `robots.txt`) are not charged. Because **Max pages** limits how many pages are delivered, it also limits your spend, and the run stops by itself when the spending limit you set for the run is reached. In local tests a default-settings crawl of a documentation site processed roughly 90 pages per minute. Speed on the platform depends on the target site and on your politeness settings. ### Input See the **Input** tab for all options with descriptions. The main ones: | Option | Default | What it does | | -------------------------------------------- | ------------------------ | ---------------------------------------------------------------------------------------------- | | **Start URLs** | | Where the crawl starts. Only public pages are crawled. | | **Max pages** | 100 | Stop after this many pages have been pushed. | | **Crawl scope** | Under the start URL path | Keep the crawl inside the directory of each start URL, or follow the whole hostname or domain. | | **Include URLs / Exclude URLs** | none | Glob patterns for links to follow or ignore. Start URLs are always crawled. | | **Max crawl depth** | 20 | Link hops from the start URLs. `0` crawls only the start URLs. | | **Chunk size** | 800 tokens | Upper bound for the size of each chunk. | | **Chunk overlap** | 100 tokens | Context repeated at the start of the next chunk (at most half of the chunk size). | | **Minimum words per page** | 10 | Skip pages with less main content than this. | | **Keep links / images in Markdown** | links on, images off | Links add context; both add tokens. | | **Max concurrency / Delay between requests** | 5 / 0.5 s | Politeness controls. A longer `Crawl-delay` from `robots.txt` wins. | | **Proxy configuration** | none | Direct connections only; configured proxies are rejected. | Example input: ```json { "startUrls": [{ "url": "https://click.palletsprojects.com/en/stable/" }], "maxPages": 30, "chunkSize": 800, "chunkOverlap": 100 } ``` #### About the crawl scope By default the crawl stays under the path of your start URL. `https://docs.example.com/guide/` will not wander into `/blog/` or into other versions of the documentation. Add a trailing slash to pin the scope exactly; for `https://docs.example.com/guide/intro` the scope is `/guide/`. Choose **Same hostname** to follow every link on the host. ### Output You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. This is one real record (shortened) produced from the Click documentation: ```json { "url": "https://click.palletsprojects.com/en/stable/virtualenv/", "title": "Virtualenv", "description": null, "language": "en", "headings": ["Virtualenv", "Why Use Virtualenv?", "How to Use Virtualenv"], "wordCount": 145, "tokenCount": 232, "markdown": "# Virtualenv\n\n## Why Use Virtualenv?\n\nYou should use [Virtualenv](https://virtualenv.pypa.io/en/latest/) because:\n\n- It allows you to install multiple versions of the same dependency. ...", "chunks": [ { "index": 0, "text": "# Virtualenv\n\n## Why Use Virtualenv?\n\nYou should use [Virtualenv](https://virtualenv.pypa.io/en/latest/) because: ...", "tokens": 232, "section": "Virtualenv" } ], "canonicalUrl": "https://click.palletsprojects.com/en/stable/virtualenv/", "publishedAt": null, "modifiedAt": null, "contentHash": "ca18b207c0f3cadaf2c876c17f7d0f8b", "crawledAt": "2026-10-23T04:09:22.355Z" } ``` Long pages produce several chunks. If a code-fence wrapper itself consumes the token budget, the complete original code block is returned as bounded raw text fragments; concatenate those fragments without a separator to reconstruct it. Normal code fences retain their language and wrappers. Each chunk keeps whole paragraphs, lists, tables and code blocks, and `section` tells you which heading path the chunk starts in, for example `Options > Multi Value Options`. ### Load the dataset into a vector database Every chunk is a ready-to-embed unit. With the Apify client for Python: ```python from apify_client import ApifyClient client = ApifyClient("") for page in client.dataset("").iterate_items(): for chunk in page["chunks"]: record = { "id": f'{page["contentHash"]}-{chunk["index"]}', "text": chunk["text"], "metadata": { "url": page["url"], "title": page["title"], "section": chunk["section"], "language": page["language"], }, } # embed record["text"] with your model and upsert the record into your vector database ``` `contentHash` changes only when the page content changes, so you can re-run the Actor on a schedule and re-embed only the pages whose hash is new. ### How the extraction and chunking work **Main-content extraction** tries three strategies in order and reports the one used in the run summary: 1. A known content container (`main`, `article`, and the containers used by Docusaurus, MkDocs, Sphinx, VitePress, Starlight, Docsify, WordPress and similar), with navigation, sidebars, table-of-contents boxes, permalinks and edit links removed from inside it. This strategy has been verified on live Sphinx, MkDocs and Docusaurus documentation sites. 2. Mozilla Readability, for pages without a usable container such as blog posts and articles. 3. The whole body with navigation blocks and link farms stripped, as a last resort. **Chunking** measures size in tokens with the `cl100k_base` tokenizer, which is a good proxy for most embedding models. A new chunk starts at a heading once the current chunk is reasonably full, so related content stays together and small sections are merged. A block larger than the chunk size is split only on natural boundaries: code by lines (each piece keeps its fence and language), tables by rows (the header row is repeated), prose by sentences. Overlap is built from whole sentences or list lines at the end of the previous chunk; code blocks and tables are never cut for overlap. ### Tips - Start with a small **Max pages** to check the output, then raise it. - For documentation with several versions or languages, use a start URL that includes the version, such as `https://docs.example.com/en/stable/`, or add **Exclude URLs** globs. - Use **Exclude URLs** for sections you do not need, for example `https://docs.example.com/**/changelog/**`. - If your embedding model has a small context window, lower **Chunk size** (for example 256 or 512). - Turn off **Keep links in Markdown** to save tokens when URLs are not useful for your use case. - Pages that contain only a list of links (for example a generated index) are skipped automatically. ### Limitations - The Actor reads the HTML that the server returns. Pages that render their content only with JavaScript in the browser (single-page apps) return little or no text and are skipped without charge. - Pages behind a login, a paywall or bot protection are not accessible. The Actor does not try to work around them. - Only HTML pages are processed. PDFs, images and other files are skipped. ### FAQ, disclaimers, and support **Does it respect robots.txt?** Yes, always. URLs that `robots.txt` disallows for this bot are never fetched, and a `Crawl-delay` is honoured up to 30 seconds. The bot identifies itself as `LLMDatasetBuilderBot`. **What happens when a site rate-limits the crawler?** On HTTP 429 or 503 the Actor honours `Retry-After`, pauses the whole crawl, lowers concurrency and retries with exponential backoff. HTTP 5xx errors and network errors are retried up to four times. HTTP 401 and 403 are never retried. **Why was a page skipped?** The run summary (`RUN_SUMMARY` in the key-value store) counts skipped pages by reason: too little content, link-only index page, duplicate content, not HTML, disallowed by `robots.txt`, and the depth limit. **Is it legal to crawl a website?** Only crawl content you are allowed to use, and check the terms of the website. Public content may include personal data, including email addresses, names or locations. This Actor does not provide a privacy or PII-removal filter. The caller must have the necessary rights, permissions and consent or other applicable legal basis to collect and use the content. Public availability does not transfer copyright or grant training, redistribution or other usage rights. Found a bug or need a feature? Open an issue in the **Issues** tab of this Actor. You can call it from your own code with the **API** tab. #### Network and recovery boundaries Only public HTTP(S) URLs on their standard ports are accepted. Private, loopback, link-local, reserved and mapped addresses are rejected after URL canonicalisation and after DNS resolution. Every redirect is revalidated, stays within the original crawl scope and gets a fresh validated, pinned direct connection. Mixed public/private DNS results and DNS failures are refused. URL lists and robots requests use the same boundary. Robots redirects stay on their original origin; unavailable robots policies fail closed for page requests. HTTP 200 login/security templates are skipped by a conservative template detector; this is not a guarantee of detecting every restriction. No bypass is attempted. A single bounded `STATE.pendingEmission` journal is persisted before a dataset push and before charging. A failed or ambiguous push/charge stops further emission and charging. On restart, a pending entry blocks execution: inspect the dataset row, content hash and platform charged-event records, reconcile them, then explicitly resolve the entry before resuming. The Actor does not automatically retry ambiguous operations. Dataset writes, charge calls and state writes are separate operations, so crash-atomic exactly-once delivery or billing is not promised. Counters and successful content hashes are persisted before the next page is published. Only the custom `page` event is used; never enable a second dataset-item event. The local price reference is $0.0015 per `page`, matching the existing platform configuration observed in the first source gate; it does not configure platform prices. This modified source has not been deployed or accepted for release. #### Dependency release gate The pre-existing lock has five HIGH audit entries representing one root advisory, GHSA-c475-qrg2-pj4r (CVE-2026-102990), through `apify-client -> proxy-agent -> pac-proxy-agent -> get-uri -> basic-ftp 5.3.1`. The compatible dependency chain has no verified patched resolution; patched `basic-ftp >=6.2.1` requires a major change. No forced override was applied. Direct page transport does not establish that SDK environment/PAC reachability is eliminated. This residual dependency risk remains a release gate for the coordinator. # Actor input Schema ## `startUrls` (type: `array`): Pages where the crawl starts. By default the crawl stays under the path of each start URL, so https://docs.example.com/guide/ will not leave /guide/. Only public pages are crawled and robots.txt is always respected. ## `maxPages` (type: `integer`): Maximum number of pages to push to the dataset. The crawl stops as soon as this many pages have been extracted. You are charged per page pushed, so this is also your cost cap. ## `crawlScope` (type: `string`): Which links are followed. 'Under the start URL path' keeps the crawl inside the directory of each start URL (best for docs sites with several versions or languages). 'Same hostname' follows every link on the same host. 'Same domain' also follows subdomains. ## `includeGlobs` (type: `array`): Optional. Only links matching at least one of these glob patterns are followed, for example https://docs.example.com/guide/\*\*. Start URLs are always crawled. ## `excludeGlobs` (type: `array`): Optional. Links matching any of these glob patterns are never followed, for example https://docs.example.com/**/changelog/**. ## `maxCrawlDepth` (type: `integer`): How many link hops away from the start URLs the crawl may go. 0 crawls only the start URLs. ## `chunkSize` (type: `integer`): Maximum size of each chunk in tokens (OpenAI cl100k_base tokenizer, a good proxy for most embedding models). Chunks follow headings, paragraphs, lists, tables and code blocks and never cut through them unless a single block is larger than this limit. ## `chunkOverlap` (type: `integer`): Tokens of context repeated at the start of the next chunk, taken as whole sentences or lines from the end of the previous chunk. Code blocks and tables are never cut for overlap. Cannot exceed half of the chunk size. ## `minWordCount` (type: `integer`): Pages whose main content has fewer words than this are skipped (and not charged). Useful to drop empty redirect, login or listing pages. ## `includeLinks` (type: `boolean`): Keep hyperlinks as [text](absolute-url). Turn off to get plain text without URLs, which uses fewer tokens. ## `includeImages` (type: `boolean`): Keep images as ![alt](absolute-url). Off by default because image links add tokens but no text for most RAG use cases. ## `maxConcurrency` (type: `integer`): Maximum number of pages fetched in parallel. The crawler starts slowly and halves its speed when a site answers with HTTP 429 or 503. ## `requestDelaySecs` (type: `number`): Minimum pause between two requests to the same site. If robots.txt asks for a longer Crawl-delay, that value is used instead. ## `proxyConfiguration` (type: `object`): Direct public connections only. Proxy routes are rejected because their target DNS cannot be pinned by this Actor. Leave useApifyProxy false and proxyUrls empty. ## Actor input object example ```json { "startUrls": [ { "url": "https://click.palletsprojects.com/en/stable/" } ], "maxPages": 100, "crawlScope": "start-path-prefix", "includeGlobs": [], "excludeGlobs": [], "maxCrawlDepth": 20, "chunkSize": 800, "chunkOverlap": 100, "minWordCount": 10, "includeLinks": true, "includeImages": false, "maxConcurrency": 5, "requestDelaySecs": 0.5, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `dataset` (type: `string`): Dataset with one item per page: url, title, headings, markdown, chunks, wordCount, language and more ## `summary` (type: `string`): Totals, skipped URLs and settings of this run # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://click.palletsprojects.com/en/stable/" } ], "includeGlobs": [], "excludeGlobs": [], "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("atesen-software/llm-dataset-builder").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://click.palletsprojects.com/en/stable/" }], "includeGlobs": [], "excludeGlobs": [], "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("atesen-software/llm-dataset-builder").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://click.palletsprojects.com/en/stable/" } ], "includeGlobs": [], "excludeGlobs": [], "proxyConfiguration": { "useApifyProxy": false } }' | apify call atesen-software/llm-dataset-builder --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,atesen-software/llm-dataset-builder" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/h2WEOa9ySUHwe8ZQR/builds/mADK6mcoaNt26u15m/openapi.json