Website to Markdown for RAG: Clean Text, Chunks, Sitemaps

Turn static web pages into clean Markdown, plain text and RAG-ready chunks. Give it page URLs, a sitemap, or a start URL to crawl. $1.00 per 1,000 pages converted; failed, blocked and empty pages are free.

No JavaScript. This Actor fetches the HTML the server sends and does not run scripts. Pages that build their content in the browser (many single-page apps) come back with status low-content, a warning, and no charge. For JavaScript-heavy sites use a browser-based crawler such as apify/website-content-crawler.

• Three ways in: a list of page URLs, sitemap URLs (sitemap index files and .gz files work), or a start URL with a same-site crawl and a page cap (default 10, maximum 5,000). Include and exclude patterns for sitemap and crawl URLs.
• Clean output: main content only (navigation, footers and scripts removed), as Markdown with tables and code blocks, plus plain text, title, description, language, headings, word count, approximate token count and a SHA-256 content hash. Links are optional.
• RAG chunks: split at headings, then paragraphs and sentences. You set the size and overlap in tokens. Every chunk carries its heading path and index.
• Change detection: set a key and each page gets changedSinceLastRun, so you only re-embed what changed.
• Polite by default: robots.txt is obeyed (disallowed pages become a free blocked-by-robots row), 1 request per second per host (0.75 to 4), honest User-Agent, no proxies, no work-arounds for blocks. A Crawl-delay in robots.txt is honoured in full, never shortened; sites that ask for more than 1.75 seconds are skipped (free slow-host rows): 1.75 seconds is the longest delay this Actor honours.

Quick start

1. Open the Actor, keep the prefilled input (two pages of quotes.toscrape.com, a public sandbox made for scraping practice) and press Start. It takes a few seconds.
2. Open the Pages view for an overview or Markdown for the converted text.
3. Replace the input with your own urls, sitemapUrls or startUrls.

Input

Field	Type	Default	What it does
urls	list	none	Page URLs to convert.
sitemapUrls	list	none	Sitemap files; every listed page on the sitemap's own site is converted.
startUrls	list	none	Converted first, then the crawl follows links to other pages on the same site (same host, with or without www).
maxPages	integer	10	Most pages fetched in a run, all inputs together. Hard maximum 5,000.
includeGlobs / excludeGlobs	list	none	Patterns for sitemap and crawled URLs. * stays inside one path segment, ** matches anything. A pattern starting with / is matched against the path (/docs/**), otherwise against the full URL. Exclude wins. At most 20 patterns of up to 200 characters each, with at most 8 wildcards (*, **, ?) per pattern; URLs longer than 2,048 characters are not fetched. Pages you list yourself and start URLs are always converted.
crawlDepth	integer	3	Links away from a start URL (0 to 10).
chunkSizeTokens	integer	512	Chunk size in approximate tokens. 0 turns chunks off, otherwise at least 50.
chunkOverlapTokens	integer	50	Overlap when a long section has to be split. Capped at half the chunk size.
includeMarkdown / includeText / includeLinks	boolean	true / true / false	Which content fields are in each row.
removeContactDetails	boolean	true	Replaces email addresses and phone numbers with [email removed] / [phone removed]. Best effort.
changeDetectionKey	string	none	Letters, digits and dashes. Turns on changedSinceLastRun.
requestsPerSecondPerHost	number	1	0.75 to 4 (a slower rate only makes the run longer and costlier). A Crawl-delay in robots.txt is honoured in full (never shortened) when it is longer than this gap.
allowSlowHosts	boolean	false	Kept for compatibility; it has no effect. The longest Crawl-delay this Actor honours is 1.75 seconds (honoured in full, never shortened), with or without this switch: a site whose robots.txt asks for more is skipped with free slow-host rows ("over the 1.75 s maximum this Actor honours"), because at 1.75 seconds one page costs about 10% of its price in compute at $0.20 per compute unit.
maxPageSizeMb	integer	2	Larger pages are skipped (free). Hard cap 10 MB; see Limits for memory.
pageTimeoutSecs	integer	30	Per page, including redirects.
concurrency	integer	4	Pages fetched in parallel across hosts. One host is never hit faster than its rate limit.

Example 1: a few pages

{
  "urls": ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/page/2/"],
  "maxPages": 10
}

Example 2: a documentation site from its sitemap, chunked for embeddings

{
  "sitemapUrls": ["https://docs.example.org/sitemap.xml"],
  "includeGlobs": ["/guide/**"],
  "excludeGlobs": ["**/changelog/**"],
  "maxPages": 500,
  "chunkSizeTokens": 400,
  "chunkOverlapTokens": 40,
  "includeText": false
}

Example 3: crawl one site and re-run weekly to find changes

{
  "startUrls": ["https://example.org/blog"],
  "crawlDepth": 2,
  "maxPages": 200,
  "includeLinks": true,
  "changeDetectionKey": "example-blog"
}

Output

One dataset row per page. Shortened example:

{
  "url": "https://quotes.toscrape.com/",
  "source": "start",
  "depth": 0,
  "status": "ok",
  "statusCode": 200,
  "title": "Quotes to Scrape",
  "description": null,
  "lang": "en",
  "wordCount": 213,
  "tokenCountApprox": 293,
  "contentSha256": "4107e3cc3dd8b992c11647c0ac51a16308678888e82d4e42630d55a87f99b031",
  "changedSinceLastRun": null,
  "headings": [{ "level": 1, "text": "Quotes to Scrape" }],
  "markdown": "# Quotes to Scrape\n\n“The world as we have created it is a process of our thinking. ...",
  "text": "Quotes to Scrape\n\n“The world as we have created it is a process of our thinking. ...",
  "chunks": [
    { "index": 0, "text": "# Quotes to Scrape\n\n“The world as we have created it ...", "tokens": 425, "headingPath": ["Quotes to Scrape"] }
  ],
  "extraction": "readability",
  "warnings": [],
  "bytes": 1962,
  "fetchedAt": "2026-10-20T12:41:41.184Z"
}

Notes on the fields:

• status is ok (charged), low-content (fewer than 10 words found; free), blocked-by-robots (free) or error (free, with errorCode and message).
• A very large page can make a row too big for one dataset row (about 8.5 MB is the limit the Actor keeps). The Actor then drops chunks, then text, then cuts the end of the Markdown, and says so in warnings (chunks-dropped-row-too-large, text-dropped-row-too-large, markdown-truncated-row-too-large). The page is still saved and charged. If even that is not enough the row is a free row-too-large error.
• finalUrl appears when the page redirected. source is url, start, sitemap, crawl or sitemap-file (a sitemap problem).
• tokenCountApprox and chunk tokens are characters divided by 4. This is a rough rule for English; real tokenizers differ by model and language.
• contentSha256 is the SHA-256 of text (after emails and phones are removed, if that option is on). wordCount counts whitespace-separated words, so it under-counts Chinese and Japanese.
• changedSinceLastRun is true for a page not seen before with this key, false if the text is identical, and null when no key is set. It is only updated for pages that were converted and saved.
• chunks[].headingPath is the heading trail of the section where the chunk starts. Small neighbouring sections are packed into one chunk; a section longer than the chunk size is split with the overlap you set.
• extraction is readability (Mozilla Readability, the algorithm behind Firefox Reader View) or body-fallback (the body text without navigation, headers and footers) when Readability found too little.
• Images are not included. Links in the Markdown are made absolute. mailto: and tel: links keep their text and lose the target.

A RUN_SUMMARY record in the key-value store lists counts per status, pages charged, bytes downloaded, failures by error code and why the run stopped (finished, max-charge, failure-limit or timeout). failure-limit also covers a site made of free empty pages: the run stops when at least 100 low-content pages make up at least 80% of the pages fetched.

Error codes

errorCode	Meaning
http	The server answered something other than 200 (see statusCode). 401, 403, 429 and 503 are never retried or worked around; after three in a row from one host, its remaining pages get host-refused.
bot-challenge	The page is a bot-check or block page (for example a Cloudflare "Just a moment..." page or an "Access Denied" page), not the real content. Free; counts toward the three refusals that stop a host; never worked around.
host-refused	The host kept refusing requests; the rest of its pages were skipped.
not-html	The URL returned a PDF, image, JSON or other non-HTML file.
too-large	The page is above Max page size (also checked after decompression).
timeout, network, dns	The host did not answer, dropped the connection or does not exist.
redirects	More than 5 redirects.
blocked	The address is private, local or otherwise not allowed (also after a redirect).
platform	Social, video and Google sites are refused: this Actor does not read them.
invalid-url	Not an http(s) URL.
robots-unavailable	robots.txt answered with a server error or could not be read, so the host was not fetched. A robots.txt that answers 4xx (for example 404, missing) is treated as "no rules", as RFC 9309 says. A 200 answer is judged by its content, whatever its Content-Type: real User-agent: rules apply, but an HTML page (a "soft 404"), a binary file or text without any User-agent: line holds no rules. 429 and 5xx are tried once more, then the host is skipped.
slow-host	robots.txt of the host asks for a Crawl-delay over 1.75 seconds (the maximum this Actor honours; skipped whether or not allowSlowHosts is on, and the delay is never shortened). Free; nothing was fetched from the host except its robots.txt. The same code appears on a sitemap row when the sitemap's host is slow.
not-processed	The run reached its time limit (the platform timeout minus 15 seconds) before this page could be fetched or converted. Free.
extract-failed, extract-timeout, too-deeply-nested	The page could not be converted: it crashed or ran out of memory in the conversion process, took longer than the time limit (6 s plus 6 s per MB of HTML on a full CPU core, divided by the CPU share of the run: about 4 times that at the default 1,024 MB; at most 240 s), or its HTML is nested more than 1,000 levels deep. Free. The run stops after 120 s spent on such pages.
row-too-large, push-failed	The converted page does not fit in one dataset row even after cutting, or the row could not be saved. Free.
bad-sitemap, bad-type, too-many-sitemaps, sitemap-depth, filter-budget	Problems with a sitemap file (source is sitemap-file). A root sitemap that fails gets its own row; child sitemap files that fail or are skipped give one summary row per sitemap URL you gave, with a count.

Pricing

One event, page-converted: $0.0010 per page that was converted and saved (status ok). Apify platform usage is included. There is no start fee. Free rows are never charged: errors, blocked-by-robots and low-content.

Pages converted	Cost
10	$0.01
100	$0.10
1,000	$1.00
5,000 (one run, the maximum)	$5.00

Prices of other tools change, so compare current numbers on the Store: at the time of writing (2026-10-18) apify/web-fetch listed $0.0015 per fetch and 6sigmag/fast-website-content-crawler $0.003 per page plus a start fee. If you set a maximum cost per run, the Actor stops cleanly when it is reached and never fetches pages it could not charge for.

Limits

• Static HTML only. No JavaScript, no login, no cookies, no CAPTCHA solving, no proxies. Pages behind a login or a bot-protection challenge return an error row.
• Up to 5,000 pages per run; up to 5,000 listed URLs, 50 sitemap URLs and 50 start URLs; sitemap indexes up to 3 levels and 100 files (index entries are followed, never returned as pages); sitemaps up to 30 MB uncompressed.
• Rate: 1 request per second per host by default, between 0.75 and 4. At the default one site gives about 3,600 pages per hour; start several runs for several sites, or split a large site. A Crawl-delay in robots.txt is always honoured in full (also for sitemap files and redirect targets), never shortened. Above 1.75 seconds the host is skipped (free slow-host rows), because waiting between requests makes the run long and its compute cost would exceed 10% of the page price; allowSlowHosts does not change this for this Actor.
• Memory: the default is 1,024 MB (the minimum). Pages are converted one at a time in a separate process with a 300 MB heap limit (two processes from 2,048 MB of run memory); a page that exhausts it, crashes it or takes too long fails free and the run goes on. Measured on a local machine, not yet on the platform: a worst-case set of 2 MB pages peaked at about 750 MB in total. The default handles pages up to about 4.5 MB; the Actor lowers Max page size to fit the memory the run has. Typical pages are 50 to 300 KB.
• Restarts: progress is saved every 2 seconds and when the run migrates. Pages already saved are found again in the dataset after a restart and are not converted or charged twice. Links first found on pages finished in the last 2 seconds before a restart may be missed by a crawl.
• The run stops after 100 failed pages, 100 MB downloaded for pages that were not charged, 120 s spent on pages that could not be converted, or 100 free low-content pages that are at least 80% of the pages fetched, so a list of dead links or a site made of JavaScript shells cannot run up a bill.
• maxPages counts every URL fetched, including errors and blocked pages.
• Plain text, Markdown and chunks come from the main-content guess of the extraction library. Unusual layouts, listing pages and forum threads may keep or drop sections you would not.

Privacy and terms

• You are responsible for having the right to fetch and use the pages you give it, and for following each site's terms. The Actor only fetches URLs you list, URLs in sitemaps you list, and (with startUrls) same-site links from your start pages.
• It identifies itself as web-to-markdown-actor/0.1 (https://apify.com/ataism/web-to-markdown) and obeys robots.txt (including Crawl-delay, honoured in full up to 1.75 seconds; hosts that ask for more are skipped). Pages that robots.txt disallows are not fetched.
• Emails and phone numbers are removed from the text, Markdown, headings, titles, descriptions, chunks and link texts by default, and emails are removed from link URLs (links[].url; such links are not followed by a crawl). Phone numbers are recognised by their shape (international + numbers, bracketed area codes, 3-3-4 numbers, national numbers starting with 0); plain digit groups such as table rows, prices, dates and IDs are left alone. This is pattern matching and will miss unusual formats; names and addresses are not removed. Pages can contain other personal data. Turn the option off only if you have the right to collect that data.
• Page content is processed in memory and returned in the dataset. Nothing else is stored, except with changeDetectionKey: a named key-value store w2m-hashes-<key> in your account holds a map of page URL to content hash (no page text).

For AI agents (MCP)

Purpose: fetch static web pages (no JavaScript) and return the main content as Markdown, plain text and RAG chunks, one row per page. Use it for documentation sites, blogs and articles; for pages built in the browser use a browser-based crawler.

Minimal input that works (one page):

{ "urls": ["https://quotes.toscrape.com/page/2/"] }

To read a whole site, give a start page or a sitemap and raise maxPages (default is only 10): {"startUrls": ["https://docs.example.org/"], "maxPages": 50} or {"sitemapUrls": ["https://docs.example.org/sitemap.xml"], "maxPages": 50}. At least one of urls, sitemapUrls, startUrls is required.

One output row per page (trimmed): {"url": "https://quotes.toscrape.com/page/2/", "status": "ok", "title": "Quotes to Scrape", "lang": "en", "wordCount": 213, "markdown": "# Quotes to Scrape\n\n...", "text": "...", "chunks": [{"index": 0, "text": "...", "tokens": 425, "headingPath": ["Quotes to Scrape"]}], "contentSha256": "...", "fetchedAt": "..."}. Always check status: ok, low-content, blocked-by-robots or error (with errorCode and message). To get smaller rows, set "includeText": false or "chunkSizeTokens": 0.

Cost: $0.001 per page with status ok (1 page = $0.001, 100 = $0.10, 1,000 = $1.00); this Actor adds no start fee. Free: error, blocked-by-robots and low-content rows. maxPages caps the cost at about maxPages x $0.001 (the default 10 is at most $0.01). Apify's Max total charge also stops the run cleanly.

Limits: HTTP only, no JavaScript, no login, no CAPTCHA solving, no proxies. robots.txt is obeyed and blocks are never worked around. Up to 5,000 pages per run, 1 request per second per host by default, pages up to 2 MB by default (10 MB max). Emails and phone numbers in the output are replaced by default (best effort). Social, video and Google sites are refused.

How to call it via the Apify MCP server (https://mcp.apify.com; tool names from the apify-mcp-server README):

1. search-actors with a query such as website to markdown for RAG (free) may list ataism/web-to-markdown; search results are ranked, so if it isn't listed, go straight to step 2 with the Actor name.
2. fetch-actor-details with actor ataism/web-to-markdown returns the input schema, pricing and output schema (free).
3. call-actor with actor = ataism/web-to-markdown and input = {"urls": ["https://quotes.toscrape.com/page/2/"]} runs it. The response holds run metadata and a dataset ID, not the rows.
4. get-dataset-items with that dataset ID returns the rows (follow the nextStep in the call-actor response).

To expose only this Actor as a tool, connect to https://mcp.apify.com?tools=ataism/web-to-markdown.

Use it via API, n8n or Make

cURL (returns the dataset items when the run finishes):

curl -X POST "https://api.apify.com/v2/acts/ataism~web-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://quotes.toscrape.com/"], "maxPages": 5}'

JavaScript (apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('ataism/web-to-markdown').call({
    sitemapUrls: ['https://docs.example.org/sitemap.xml'],
    maxPages: 100,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const page of items) console.log(page.url, page.status, page.wordCount);

Python (apify-client):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("ataism/web-to-markdown").call(run_input={
    "startUrls": ["https://example.org/blog"],
    "maxPages": 50,
    "changeDetectionKey": "example-blog",
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page["status"] == "ok" and page["changedSinceLastRun"]:
        print(page["url"], page["title"])

n8n: add the Apify node, choose Run an Actor and get dataset, select ataism/web-to-markdown, and paste your input JSON.

Make: add the Apify module Run an Actor (Actor ataism/web-to-markdown, input JSON), then Get Dataset Items with the run's default dataset ID.