Website to LLM-Ready Dataset - Markdown & Chunks for RAG

What does Website to LLM-Ready Dataset do?

Website to LLM-Ready Dataset crawls a website or documentation site and turns every page into a clean, structured record that you can load straight into a vector database, a RAG pipeline or a fine-tuning set. It is a website to Markdown API with chunking built in.

For every page you get:

• Clean main content as Markdown. Navigation, sidebars, footers, cookie banners, breadcrumbs, "edit this page" links, table-of-contents boxes and other boilerplate are removed. Headings, lists, tables and code blocks (with their language) are kept.
• Semantic chunks, ready to embed. The Markdown is split into token-bounded chunks that follow the structure of the page instead of cutting through sentences, code blocks or tables. Chunk size and overlap are configurable.
• Metadata. URL, title, description, headings, language, word and token counts, canonical URL, publish and modify dates, and a content hash for deduplication and change detection.

It does not return raw HTML, and it does not bypass logins, paywalls or bot protection. It only reads public pages and always respects robots.txt.

Why use Website to LLM-Ready Dataset?

• Drop-in for vector databases. Every chunk has an index, its text, a tokens count and the section (heading path) it belongs to, so you can embed chunk.text and store url, title and section as metadata. No post-processing script needed.
• Better retrieval than raw text dumps. Boilerplate removal means your embeddings are not polluted by menus and footers, and structure-aware chunking keeps code examples and tables in one piece.
• Polite by design. Honest bot user agent, robots.txt and Crawl-delay support, low starting concurrency, automatic backoff on HTTP 429 and 5xx, and a shared cool-down when a server asks to slow down.
• Predictable cost. You are charged per page that is actually delivered. Pages that are empty, duplicates, blocked or disallowed by robots.txt are free, and Max pages is a hard cap.
• Apify platform features. Schedule runs, call it through the API, send results to your own storage with integrations and webhooks, and export the dataset as JSON, CSV, Excel or JSONL.

Typical uses: AI chatbots over documentation, internal knowledge bases, RAG for support content, keeping a search index in sync with a docs site, collecting training data from sites you are allowed to use.

What data can Website to LLM-Ready Dataset extract?

One dataset item per page:

Field	Type	Description
url	string	Final URL of the page after redirects
title	string	Page title, taken from the main heading when available
description	string	Meta description, if the page has one
language	string	Language code such as en, from the page or detected from the text
headings	array of strings	All headings in document order
wordCount	integer	Words in the extracted Markdown
tokenCount	integer	Tokens in the extracted Markdown (cl100k_base)
markdown	string	Clean Markdown of the main content
chunks	array	{ index, text, tokens, section } for each chunk
canonicalUrl	string	Canonical URL declared by the page
publishedAt, modifiedAt	string	ISO dates declared by the page, if any
contentHash	string	Fingerprint of the Markdown, for deduplication and change detection
crawledAt	string	When the page was crawled

How to turn a website into an LLM-ready dataset

1. Open the Actor and add one or more Start URLs, for example https://docs.example.com/guide/.
2. Set Max pages. This is also your cost cap.
3. Optional: choose the Chunk size (default 800 tokens) and Chunk overlap (default 100 tokens) that match your embedding model.
4. Optional: add Include URLs or Exclude URLs glob patterns to narrow the crawl.
5. Click Start. When the run finishes, open the Output tab or download the dataset as JSON, CSV or Excel. A RUN_SUMMARY record in the key-value store lists totals, skipped URLs and the settings used.

How much will it cost to turn a website into an LLM-ready dataset?

The Actor uses pay-per-event pricing: one page event is charged for every page that is pushed to the dataset. The current price is shown on the Pricing tab. Pages that are skipped (empty, duplicate, link-only index pages, HTTP errors, blocked by robots.txt) are not charged. Because Max pages limits how many pages are delivered, it also limits your spend, and the run stops by itself when the spending limit you set for the run is reached.

In local tests a default-settings crawl of a documentation site processed roughly 90 pages per minute. Speed on the platform depends on the target site and on your politeness settings.

Input

See the Input tab for all options with descriptions. The main ones:

Option	Default	What it does
Start URLs		Where the crawl starts. Only public pages are crawled.
Max pages	100	Stop after this many pages have been pushed.
Crawl scope	Under the start URL path	Keep the crawl inside the directory of each start URL, or follow the whole hostname or domain.
Include URLs / Exclude URLs	none	Glob patterns for links to follow or ignore. Start URLs are always crawled.
Max crawl depth	20	Link hops from the start URLs. 0 crawls only the start URLs.
Chunk size	800 tokens	Upper bound for the size of each chunk.
Chunk overlap	100 tokens	Context repeated at the start of the next chunk (at most half of the chunk size).
Minimum words per page	10	Skip pages with less main content than this.
Keep links / images in Markdown	links on, images off	Links add context; both add tokens.
Max concurrency / Delay between requests	5 / 0.5 s	Politeness controls. A longer Crawl-delay from robots.txt wins.
Proxy configuration	none	Direct connections only; configured proxies are rejected.

Example input:

{
    "startUrls": [{ "url": "https://click.palletsprojects.com/en/stable/" }],
    "maxPages": 30,
    "chunkSize": 800,
    "chunkOverlap": 100
}

About the crawl scope

By default the crawl stays under the path of your start URL. https://docs.example.com/guide/ will not wander into /blog/ or into other versions of the documentation. Add a trailing slash to pin the scope exactly; for https://docs.example.com/guide/intro the scope is /guide/. Choose Same hostname to follow every link on the host.

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. This is one real record (shortened) produced from the Click documentation:

{
    "url": "https://click.palletsprojects.com/en/stable/virtualenv/",
    "title": "Virtualenv",
    "description": null,
    "language": "en",
    "headings": ["Virtualenv", "Why Use Virtualenv?", "How to Use Virtualenv"],
    "wordCount": 145,
    "tokenCount": 232,
    "markdown": "# Virtualenv\n\n## Why Use Virtualenv?\n\nYou should use [Virtualenv](https://virtualenv.pypa.io/en/latest/) because:\n\n- It allows you to install multiple versions of the same dependency. ...",
    "chunks": [
        {
            "index": 0,
            "text": "# Virtualenv\n\n## Why Use Virtualenv?\n\nYou should use [Virtualenv](https://virtualenv.pypa.io/en/latest/) because: ...",
            "tokens": 232,
            "section": "Virtualenv"
        }
    ],
    "canonicalUrl": "https://click.palletsprojects.com/en/stable/virtualenv/",
    "publishedAt": null,
    "modifiedAt": null,
    "contentHash": "ca18b207c0f3cadaf2c876c17f7d0f8b",
    "crawledAt": "2026-10-23T04:09:22.355Z"
}

Long pages produce several chunks. If a code-fence wrapper itself consumes the token budget, the complete original code block is returned as bounded raw text fragments; concatenate those fragments without a separator to reconstruct it. Normal code fences retain their language and wrappers. Each chunk keeps whole paragraphs, lists, tables and code blocks, and section tells you which heading path the chunk starts in, for example Options > Multi Value Options.

Load the dataset into a vector database

Every chunk is a ready-to-embed unit. With the Apify client for Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
for page in client.dataset("<DATASET_ID>").iterate_items():
    for chunk in page["chunks"]:
        record = {
            "id": f'{page["contentHash"]}-{chunk["index"]}',
            "text": chunk["text"],
            "metadata": {
                "url": page["url"],
                "title": page["title"],
                "section": chunk["section"],
                "language": page["language"],
            },
        }
        # embed record["text"] with your model and upsert the record into your vector database

contentHash changes only when the page content changes, so you can re-run the Actor on a schedule and re-embed only the pages whose hash is new.

How the extraction and chunking work

Main-content extraction tries three strategies in order and reports the one used in the run summary:

1. A known content container (main, article, and the containers used by Docusaurus, MkDocs, Sphinx, VitePress, Starlight, Docsify, WordPress and similar), with navigation, sidebars, table-of-contents boxes, permalinks and edit links removed from inside it. This strategy has been verified on live Sphinx, MkDocs and Docusaurus documentation sites.
2. Mozilla Readability, for pages without a usable container such as blog posts and articles.
3. The whole body with navigation blocks and link farms stripped, as a last resort.

Chunking measures size in tokens with the cl100k_base tokenizer, which is a good proxy for most embedding models. A new chunk starts at a heading once the current chunk is reasonably full, so related content stays together and small sections are merged. A block larger than the chunk size is split only on natural boundaries: code by lines (each piece keeps its fence and language), tables by rows (the header row is repeated), prose by sentences. Overlap is built from whole sentences or list lines at the end of the previous chunk; code blocks and tables are never cut for overlap.

Tips

• Start with a small Max pages to check the output, then raise it.
• For documentation with several versions or languages, use a start URL that includes the version, such as https://docs.example.com/en/stable/, or add Exclude URLs globs.
• Use Exclude URLs for sections you do not need, for example https://docs.example.com/**/changelog/**.
• If your embedding model has a small context window, lower Chunk size (for example 256 or 512).
• Turn off Keep links in Markdown to save tokens when URLs are not useful for your use case.
• Pages that contain only a list of links (for example a generated index) are skipped automatically.

Limitations

• The Actor reads the HTML that the server returns. Pages that render their content only with JavaScript in the browser (single-page apps) return little or no text and are skipped without charge.
• Pages behind a login, a paywall or bot protection are not accessible. The Actor does not try to work around them.
• Only HTML pages are processed. PDFs, images and other files are skipped.

FAQ, disclaimers, and support

Does it respect robots.txt? Yes, always. URLs that robots.txt disallows for this bot are never fetched, and a Crawl-delay is honoured up to 30 seconds. The bot identifies itself as LLMDatasetBuilderBot.

What happens when a site rate-limits the crawler? On HTTP 429 or 503 the Actor honours Retry-After, pauses the whole crawl, lowers concurrency and retries with exponential backoff. HTTP 5xx errors and network errors are retried up to four times. HTTP 401 and 403 are never retried.

Why was a page skipped? The run summary (RUN_SUMMARY in the key-value store) counts skipped pages by reason: too little content, link-only index page, duplicate content, not HTML, disallowed by robots.txt, and the depth limit.

Is it legal to crawl a website? Only crawl content you are allowed to use, and check the terms of the website.

Public content may include personal data, including email addresses, names or locations. This Actor does not provide a privacy or PII-removal filter. The caller must have the necessary rights, permissions and consent or other applicable legal basis to collect and use the content. Public availability does not transfer copyright or grant training, redistribution or other usage rights.

Found a bug or need a feature? Open an issue in the Issues tab of this Actor. You can call it from your own code with the API tab.

Network and recovery boundaries

Only public HTTP(S) URLs on their standard ports are accepted. Private, loopback, link-local, reserved and mapped addresses are rejected after URL canonicalisation and after DNS resolution. Every redirect is revalidated, stays within the original crawl scope and gets a fresh validated, pinned direct connection. Mixed public/private DNS results and DNS failures are refused. URL lists and robots requests use the same boundary. Robots redirects stay on their original origin; unavailable robots policies fail closed for page requests. HTTP 200 login/security templates are skipped by a conservative template detector; this is not a guarantee of detecting every restriction. No bypass is attempted.

A single bounded STATE.pendingEmission journal is persisted before a dataset push and before charging. A failed or ambiguous push/charge stops further emission and charging. On restart, a pending entry blocks execution: inspect the dataset row, content hash and platform charged-event records, reconcile them, then explicitly resolve the entry before resuming. The Actor does not automatically retry ambiguous operations. Dataset writes, charge calls and state writes are separate operations, so crash-atomic exactly-once delivery or billing is not promised. Counters and successful content hashes are persisted before the next page is published. Only the custom page event is used; never enable a second dataset-item event.

The local price reference is $0.0015 per page, matching the existing platform configuration observed in the first source gate; it does not configure platform prices. This modified source has not been deployed or accepted for release.

Dependency release gate

The pre-existing lock has five HIGH audit entries representing one root advisory, GHSA-c475-qrg2-pj4r (CVE-2026-102990), through apify-client -> proxy-agent -> pac-proxy-agent -> get-uri -> basic-ftp 5.3.1. The compatible dependency chain has no verified patched resolution; patched basic-ftp >=6.2.1 requires a major change. No forced override was applied. Direct page transport does not establish that SDK environment/PAC reachability is eliminated. This residual dependency risk remains a release gate for the coordinator.