Website to Markdown Crawler (llms.txt)

Crawl a website or a docs section into clean, LLM-ready markdown, one row per page, and get llms.txt and llms-full.txt files for it. Give one or more start URLs; the Actor reads the start page, then the site's sitemap, then the links it finds (breadth-first, within the site or the section you started in), extracts each page's main content as markdown (menus, footers and ads removed, tables kept) and reports title, description, depth, word count and token estimate per page. Built for RAG pipelines, AI agents, documentation bots and anyone who needs a site's content as text. $1.50 per 1,000 pages; pages without content, duplicates and the llms.txt files are free.

• What you get: per page markdown, title, description, url, depth, discoveredVia, wordCount, tokenEstimate, contentStatus; per start URL a summary with crawl statistics and links to llms.txt and llms-full.txt in the key-value store.
• Smart scope: start at https://example.com/docs and only /docs is crawled; start at a homepage and the whole domain (subdomains included) is. Include and exclude patterns narrow it further.
• Sitemap first: the sitemap (from robots.txt or /sitemap.xml) is read before links are followed, shallow pages first, so a limited crawl covers the site's main pages.
• Polite and safe: at most 2 requests per second per host, a request cap per site, public hosts only; files and media are never downloaded.
• Price: crawled-page $0.0015 per page with content. A 200-page docs site costs $0.30.
• Works everywhere: API, MCP for AI agents, Make, Zapier, n8n, LangChain, schedules and webhooks.

Input

Field	Type	Default	Description
startUrls	string[]	–	Websites or site sections to crawl, one per line (https:// is added when the scheme is missing).
maxPagesPerSite	integer	50	Pages with content to deliver per start URL (1–10,000).
maxDepth	integer	5	Links away from the start page to follow (0–20; sitemap URLs count as depth 1; 0 reads only the start page).
crawlScope	string	auto	auto (the start URL's path when it has one, else the domain), path, host or domain (all subdomains of the registrable domain).
useSitemap	boolean	true	Queue the site's sitemap URLs in scope before the links found on pages.
includePatterns	string[]	–	Deliver only pages whose URL matches one of these: a glob (*/blog/*) or a /regular expression/. Other pages are still read for their links. A regular expression that backtracks catastrophically is refused before the crawl, and one that takes longer than 250 ms on a URL stops that site's crawl (pattern_too_slow).
excludePatterns	string[]	–	Do not deliver pages whose URL matches one of these (same syntax).
maxCharsPerPage	integer	50000	Markdown kept per page (1,000–200,000 characters), cut at a paragraph boundary.
llmsTxt	boolean	true	Write llms.txt and llms-full.txt per start URL to the run's key-value store (free).
maxSites	integer	100	Safety cap for this run; start URLs beyond it are ignored and counted as inputItemsDropped in the OUTPUT record.
concurrency	integer	2	Start URLs crawled at the same time (1–5).

{
  "startUrls": ["https://crawlee.dev/js/docs/introduction"],
  "maxPagesPerSite": 100,
  "excludePatterns": ["*/changelog*"]
}

Output

One page row per page read (pages with content are charged and numbered by position), then one summary row per start URL:

[
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "recordType": "page",
    "position": 2,
    "url": "https://crawlee.dev/js/docs/introduction/setting-up",
    "requestedUrl": "https://crawlee.dev/js/docs/introduction/setting-up",
    "title": "Setting up",
    "siteName": "Crawlee",
    "description": "To run Crawlee on your computer, you need Node.js and npm.",
    "language": "en",
    "publishedAt": null,
    "canonicalUrl": "https://crawlee.dev/js/docs/introduction/setting-up",
    "depth": 1,
    "discoveredVia": "sitemap",
    "referrer": null,
    "markdown": "# Setting up\n\nIn this guide, we will set up Crawlee and run our first crawler. …",
    "markdownTruncated": false,
    "wordCount": 423,
    "tokenEstimate": 712,
    "contentStatus": "ok",
    "status": "ok",
    "charged": true,
    "chargedEvent": "crawled-page",
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:04.000Z"
  },
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "recordType": "page",
    "url": "https://crawlee.dev/js/docs/introduction/old-page",
    "requestedUrl": "https://crawlee.dev/js/docs/introduction/old-page",
    "title": null,
    "depth": 1,
    "discoveredVia": "link",
    "referrer": "https://crawlee.dev/js/docs/introduction",
    "contentStatus": "error",
    "status": "error",
    "charged": false,
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:05.000Z",
    "error": { "code": "not_found", "message": "HTTP 404 from crawlee.dev", "status": 404 }
  },
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "startUrl": "https://crawlee.dev/js/docs/introduction",
    "finalStartUrl": "https://crawlee.dev/js/docs/introduction",
    "crawlScope": "path",
    "scopeLabel": "crawlee.dev/js/docs/introduction",
    "pagesFailed": 1,
    "duplicatePages": 0,
    "pagesFiltered": 0,
    "offsiteRedirects": 0,
    "urlsQueued": 10,
    "sitemapUrls": 9,
    "sitemap": { "discoveredVia": "robots", "sitemapsRead": 2, "sitemapsFailed": 0 },
    "requests": 11,
    "maxDepthReached": 1,
    "llmsTxt": { "key": "llms-crawlee.dev-js-docs-introduction.txt", "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-crawlee.dev-js-docs-introduction.txt", "pages": 9, "bytes": 1460 },
    "llmsFullTxt": { "key": "llms-full-crawlee.dev-js-docs-introduction.txt", "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-full-crawlee.dev-js-docs-introduction.txt", "pages": 9, "bytes": 61234, "truncated": false },
    "recordType": "summary",
    "resultsDelivered": 9,
    "stopReason": "end",
    "status": "ok",
    "charged": false,
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:09.000Z"
  }
]

Field	Description
markdown, markdownTruncated	The page's main content as GitHub-flavoured markdown, at most maxCharsPerPage characters.
title, siteName, description, language, publishedAt, canonicalUrl	Page metadata as the page states it.
depth, discoveredVia, referrer	Links away from the start page; start, sitemap or link; the page whose link led here.
wordCount, tokenEstimate	Words of the content and a rough LLM token count of the markdown.
contentStatus	ok, blocked (bot check, access denied), js_required (JavaScript app), not_html, too_large, error (failed or too little text).
llmsTxt, llmsFullTxt	summary: key, public URL, pages and size of the files in the run's key-value store.
pagesFailed, duplicatePages, pagesFiltered, requests	summary: what the crawl met on the way.

status values:

status	Meaning	Charged
ok	page row: content delivered; summary: the crawl ran	crawled-page per page row with content; summaries never
error	page row: no usable content (see contentStatus); summary: no page in scope had content (no_content) or the start page failed	no
not_found	summary: the start page answered HTTP 404 or 410	no
skipped_invalid	summary: not a public http(s) URL	no
skipped_budget / skipped_free_plan	Maximum cost per run reached / free-plan limit	no

Pricing

Store discounts: the FREE and BRONZE tiers use the list prices below. SILVER receives 5% off; GOLD, PLATINUM and DIAMOND receive 10% off billable result events. The platform start fee is unchanged. Check the Actor's Store pricing table for the prices currently in effect.

Pay-per-event:

Event	Price	When charged
crawled-page	$0.0015 per page ($1.50 / 1,000)	once per page delivered with readable content

Pages without content, duplicate pages, filtered pages, summary rows and the llms.txt files are never charged. Set Maximum cost per run to cap spend: each site stops cleanly when the next page would no longer fit (stopReason: "budget"), and its llms.txt files list the pages delivered so far.

Accounts on the Apify free plan can process up to 10 start URLs per run and up to 20 pages per site; the rest are not crawled or charged. Upgrade to any paid Apify plan for full crawls.

Use with AI agents (MCP)

Use this Actor from Claude, ChatGPT, Cursor, VS Code or any other MCP client through the Apify MCP server. Add this server URL to your client and sign in with your Apify account when asked:

https://mcp.apify.com?tools=sfoln/website-to-markdown

To run the MCP server locally instead, start npx -y @apify/actors-mcp-server --tools sfoln/website-to-markdown with your APIFY_TOKEN in the environment.

Example prompt:

Crawl https://crawlee.dev/js/docs/introduction (at most 10 pages) and give me the pages as markdown plus the llms.txt file.

Runs started by an agent are billed like any other run (see Pricing). Agents can cap what a run may spend with maxTotalChargeUsd.

API usage

Run the Actor and get its dataset items in one call (the call waits up to 5 minutes for the run to finish):

curl -X POST "https://api.apify.com/v2/acts/sfoln~website-to-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer <APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/introduction"], "maxPagesPerSite": 3}'

For large inputs, start the run asynchronously and download the dataset when it has finished:

# 1. Start a run: the response holds the run ID (data.id) and its dataset ID (data.defaultDatasetId)
curl -X POST "https://api.apify.com/v2/acts/sfoln~website-to-markdown/runs" \
  -H "Authorization: Bearer <APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/introduction"], "maxPagesPerSite": 3}'
# 2. Repeat until data.status is SUCCEEDED (each call waits up to 60 s)
curl "https://api.apify.com/v2/actor-runs/<runId>?waitForFinish=60" -H "Authorization: Bearer <APIFY_TOKEN>"
# 3. Download the results (format=json, csv or xlsx)
curl "https://api.apify.com/v2/datasets/<defaultDatasetId>/items?format=json" -H "Authorization: Bearer <APIFY_TOKEN>"

Python (apify-client):

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("sfoln/website-to-markdown").call(run_input={"startUrls": ["https://crawlee.dev/js/docs/introduction"], "maxPagesPerSite": 3})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Instead of polling, you can attach a webhook to the run (see Integrations). The API tab of this page shows the same calls for JavaScript and the Apify CLI.

Integrations

• Make, Zapier and n8n: run sfoln/website-to-markdown with the Apify app (in n8n, the Apify node) and pass the dataset items to the next step of your workflow.
• Google Sheets: add the items to a sheet with the Google Sheets step of Make, Zapier or n8n, or download them as CSV or Excel from the run's Storage tab or the API (format=csv or format=xlsx).
• Webhooks: get a POST request when a run succeeds or fails: add a webhook in the Actor's Integrations tab, or pass webhooks when you start a run through the API.
• Schedules: run it hourly, daily or weekly with the same input from Apify Schedules; every run keeps its results in its own dataset.
• More: Slack, Google Drive, GitHub, LangChain, LlamaIndex and other Apify integrations.

Related Actors

• URL to Markdown & Article Extractor
• RAG Web Search – Google + LLM-Ready Markdown
• AI Web Scraper – Any URL to Structured JSON
• Sitemap URL Extractor
• Website Change Monitor – Text Diff Alerts

See all our Actors on Apify Store.

FAQ

How much does it cost, and am I charged for failed items?

You pay per event at the prices under Pricing above, plus a run start fee of $0.00005 per GB of memory, with no subscription; platform usage is included. Errors and skipped items (skipped_… statuses) are never charged; a lookup that finds nothing is charged only where Pricing says so, and every record's charged field shows whether it was billed. Set Maximum cost per run to cap what a single run can spend.

How fresh is the data?

Each run queries its source while it runs, so you get what the source returns at that moment. Source-specific details, such as how often a provider refreshes its own database, are under Limits & notes.

What are the limits?

Per-run caps (for example the maximum number of items) are listed under Input, source limits under Limits & notes, and the free-plan limit under Pricing. For big batches, start the run asynchronously (see API usage) and read the dataset when it has finished.

Can I schedule runs or get results by webhook?

Yes. Create a schedule in Apify Console to run it hourly, daily or weekly with the same input, and add a webhook to get a POST request when a run finishes (see Integrations).

What does this Actor record about my runs?

This Actor records operational metrics about each run (item counts, duration, charges, error codes and a pseudonymous hash of your Apify user ID) to monitor reliability and costs. It never stores your input or results.

Which pages are crawled?

The start page first, then the URLs of the site's sitemap (found through robots.txt or /sitemap.xml) that are in scope, shallow paths first, then the links found on each page, breadth-first up to maxDepth. crawlScope auto stays under the start URL's path when it has one (https://example.com/docs crawls /docs and below) and otherwise covers the whole registrable domain, subdomains included. includePatterns and excludePatterns narrow it further. Files and media (PDF, images, archives) are skipped, and each page is read once even when several URLs lead to it.

What are llms.txt and llms-full.txt?

llms.txt (llmstxt.org) is a markdown index of a site for language models: the site's name and description, then one section per top-level path listing every page as a link with a short description. llms-full.txt contains the markdown of every page one after the other (up to 8 MB). Both are written to the run's key-value store (llms-<site>.txt and llms-full-<site>.txt); the summary row of each start URL links to them. They are free.

Why do some pages have contentStatus blocked or js_required?

No browser is used. blocked: the site answered with a bot check, an access-denied page or a rate limit. js_required: the page builds its content with JavaScript and the HTML has no text. Such pages are written as uncharged rows so you can see what was skipped; their links (if any) are still followed. For JavaScript-heavy sites the sitemap often still leads to readable pages.

How polite is the crawl?

At most 2 requests per second and 2 at a time per host, shared by every start URL of the run, with a 20-second timeout and a 5 MB cap per page. Each site gets at most 2 page requests per page it may deliver (maxPagesPerSite, 20 on the free plan) plus 20, so failing links cannot make a crawl run away.

How do I get help?

Open an issue in the Actor's Issues tab and include the link to the run, so we can reproduce the problem. Feature requests are welcome there too.

Limits & notes

• Pages are fetched by the Actor itself: 20-second timeout, 5 MB per page, at most 2 requests per second and 2 at a time per host, public http(s) hosts only. Each site gets at most 2 page requests per page it may deliver plus 20 (2 × maxPagesPerSite + 20; 60 on the free plan), also when few of the pages read are delivered (include/exclude patterns, pages without content); long crawls stop a minute before the time limit with stopReason: "deadline".
• No browser is used: pages that build their content with JavaScript return js_required, and pages behind bot protection or a login return blocked. Sites that answer with markdown instead of HTML are delivered as they are.
• Files and media (PDF, office documents, images, archives, feeds) are never queued; use a document Actor for PDFs. Query strings are kept (so paginated listings are separate pages) but tracking parameters (utm_*, gclid, fbclid, …) are dropped, and identical content under several URLs is delivered once.
• llms-full.txt is capped at 8 MB per start URL; the summary's llmsFullTxt.truncated says when pages were left out of it (they are still in the dataset).
• When the platform moves the run to another server (or the run is aborted and resurrected), a start URL that was still being crawled is crawled again after the restart: the pages it had delivered are skipped (not delivered or charged again, kept in the run's key-value store record DELIVERED_ROWS), positions and maxPagesPerSite continue where it stopped, and its summary counts both parts. A page delivered in the last seconds before a hard crash (not a migration or an abort) may be delivered again.