# Website to Markdown Crawler (llms.txt) (`sfoln/website-to-markdown`) Actor

Crawl a website or docs section into clean, LLM-ready markdown: sitemap first, then links, one row per page with title, depth, word and token counts, plus llms.txt and llms-full.txt files for AI agents and RAG. Polite, no browser needed. $1.50 per 1,000 pages.

- **URL**: https://apify.com/sfoln/website-to-markdown.md
- **Developed by:** [SFOLN LLC](https://apify.com/sfoln) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 crawled pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Crawl a website or a docs section into clean, LLM-ready markdown**, one row per page, and get **llms.txt** and **llms-full.txt** files for it. Give one or more start URLs; the Actor reads the start page, then the site's sitemap, then the links it finds (breadth-first, within the site or the section you started in), extracts each page's main content as markdown (menus, footers and ads removed, tables kept) and reports title, description, depth, word count and token estimate per page. Built for RAG pipelines, AI agents, documentation bots and anyone who needs a site's content as text. **$1.50 per 1,000 pages**; pages without content, duplicates and the llms.txt files are free.

- **What you get:** per page `markdown`, `title`, `description`, `url`, `depth`, `discoveredVia`, `wordCount`, `tokenEstimate`, `contentStatus`; per start URL a summary with crawl statistics and links to `llms.txt` and `llms-full.txt` in the key-value store.
- **Smart scope:** start at `https://example.com/docs` and only `/docs` is crawled; start at a homepage and the whole domain (subdomains included) is. Include and exclude patterns narrow it further.
- **Sitemap first:** the sitemap (from robots.txt or /sitemap.xml) is read before links are followed, shallow pages first, so a limited crawl covers the site's main pages.
- **Polite and safe:** at most 2 requests per second per host, a request cap per site, public hosts only; files and media are never downloaded.
- **Price:** `crawled-page` $0.0015 per page with content. A 200-page docs site costs $0.30.
- **Works everywhere:** API, MCP for AI agents, Make, Zapier, n8n, LangChain, schedules and webhooks.

### Input

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `startUrls` | string\[] | – | Websites or site sections to crawl, one per line (`https://` is added when the scheme is missing). |
| `maxPagesPerSite` | integer | 50 | Pages with content to deliver per start URL (1–10,000). |
| `maxDepth` | integer | 5 | Links away from the start page to follow (0–20; sitemap URLs count as depth 1; 0 reads only the start page). |
| `crawlScope` | string | `auto` | `auto` (the start URL's path when it has one, else the domain), `path`, `host` or `domain` (all subdomains of the registrable domain). |
| `useSitemap` | boolean | true | Queue the site's sitemap URLs in scope before the links found on pages. |
| `includePatterns` | string\[] | – | Deliver only pages whose URL matches one of these: a glob (`*/blog/*`) or a `/regular expression/`. Other pages are still read for their links. A regular expression that backtracks catastrophically is refused before the crawl, and one that takes longer than 250 ms on a URL stops that site's crawl (`pattern_too_slow`). |
| `excludePatterns` | string\[] | – | Do not deliver pages whose URL matches one of these (same syntax). |
| `maxCharsPerPage` | integer | 50000 | Markdown kept per page (1,000–200,000 characters), cut at a paragraph boundary. |
| `llmsTxt` | boolean | true | Write `llms.txt` and `llms-full.txt` per start URL to the run's key-value store (free). |
| `maxSites` | integer | 100 | Safety cap for this run; start URLs beyond it are ignored and counted as `inputItemsDropped` in the `OUTPUT` record. |
| `concurrency` | integer | 2 | Start URLs crawled at the same time (1–5). |

```json
{
  "startUrls": ["https://crawlee.dev/js/docs/introduction"],
  "maxPagesPerSite": 100,
  "excludePatterns": ["*/changelog*"]
}
```

### Output

One `page` row per page read (pages with content are charged and numbered by `position`), then one `summary` row per start URL:

```json
[
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "recordType": "page",
    "position": 2,
    "url": "https://crawlee.dev/js/docs/introduction/setting-up",
    "requestedUrl": "https://crawlee.dev/js/docs/introduction/setting-up",
    "title": "Setting up",
    "siteName": "Crawlee",
    "description": "To run Crawlee on your computer, you need Node.js and npm.",
    "language": "en",
    "publishedAt": null,
    "canonicalUrl": "https://crawlee.dev/js/docs/introduction/setting-up",
    "depth": 1,
    "discoveredVia": "sitemap",
    "referrer": null,
    "markdown": "# Setting up\n\nIn this guide, we will set up Crawlee and run our first crawler. …",
    "markdownTruncated": false,
    "wordCount": 423,
    "tokenEstimate": 712,
    "contentStatus": "ok",
    "status": "ok",
    "charged": true,
    "chargedEvent": "crawled-page",
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:04.000Z"
  },
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "recordType": "page",
    "url": "https://crawlee.dev/js/docs/introduction/old-page",
    "requestedUrl": "https://crawlee.dev/js/docs/introduction/old-page",
    "title": null,
    "depth": 1,
    "discoveredVia": "link",
    "referrer": "https://crawlee.dev/js/docs/introduction",
    "contentStatus": "error",
    "status": "error",
    "charged": false,
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:05.000Z",
    "error": { "code": "not_found", "message": "HTTP 404 from crawlee.dev", "status": 404 }
  },
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "startUrl": "https://crawlee.dev/js/docs/introduction",
    "finalStartUrl": "https://crawlee.dev/js/docs/introduction",
    "crawlScope": "path",
    "scopeLabel": "crawlee.dev/js/docs/introduction",
    "pagesFailed": 1,
    "duplicatePages": 0,
    "pagesFiltered": 0,
    "offsiteRedirects": 0,
    "urlsQueued": 10,
    "sitemapUrls": 9,
    "sitemap": { "discoveredVia": "robots", "sitemapsRead": 2, "sitemapsFailed": 0 },
    "requests": 11,
    "maxDepthReached": 1,
    "llmsTxt": { "key": "llms-crawlee.dev-js-docs-introduction.txt", "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-crawlee.dev-js-docs-introduction.txt", "pages": 9, "bytes": 1460 },
    "llmsFullTxt": { "key": "llms-full-crawlee.dev-js-docs-introduction.txt", "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-full-crawlee.dev-js-docs-introduction.txt", "pages": 9, "bytes": 61234, "truncated": false },
    "recordType": "summary",
    "resultsDelivered": 9,
    "stopReason": "end",
    "status": "ok",
    "charged": false,
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:09.000Z"
  }
]
```

| Field | Description |
| --- | --- |
| `markdown`, `markdownTruncated` | The page's main content as GitHub-flavoured markdown, at most `maxCharsPerPage` characters. |
| `title`, `siteName`, `description`, `language`, `publishedAt`, `canonicalUrl` | Page metadata as the page states it. |
| `depth`, `discoveredVia`, `referrer` | Links away from the start page; `start`, `sitemap` or `link`; the page whose link led here. |
| `wordCount`, `tokenEstimate` | Words of the content and a rough LLM token count of the markdown. |
| `contentStatus` | `ok`, `blocked` (bot check, access denied), `js_required` (JavaScript app), `not_html`, `too_large`, `error` (failed or too little text). |
| `llmsTxt`, `llmsFullTxt` | summary: key, public URL, pages and size of the files in the run's key-value store. |
| `pagesFailed`, `duplicatePages`, `pagesFiltered`, `requests` | summary: what the crawl met on the way. |

`status` values:

| `status` | Meaning | Charged |
| --- | --- | --- |
| `ok` | page row: content delivered; summary: the crawl ran | `crawled-page` per page row with content; summaries never |
| `error` | page row: no usable content (see `contentStatus`); summary: no page in scope had content (`no_content`) or the start page failed | no |
| `not_found` | summary: the start page answered HTTP 404 or 410 | no |
| `skipped_invalid` | summary: not a public http(s) URL | no |
| `skipped_budget` / `skipped_free_plan` | Maximum cost per run reached / free-plan limit | no |

### Pricing

Store discounts: the FREE and BRONZE tiers use the list prices below. SILVER receives 5% off; GOLD, PLATINUM and DIAMOND receive 10% off billable result events. The platform start fee is unchanged. Check the Actor's Store pricing table for the prices currently in effect.

Pay-per-event:

| Event | Price | When charged |
| --- | --- | --- |
| `crawled-page` | **$0.0015** per page ($1.50 / 1,000) | once per page delivered with readable content |

Pages without content, duplicate pages, filtered pages, summary rows and the llms.txt files are never charged. Set **Maximum cost per run** to cap spend: each site stops cleanly when the next page would no longer fit (`stopReason: "budget"`), and its llms.txt files list the pages delivered so far.

Accounts on the Apify free plan can process up to 10 start URLs per run and up to 20 pages per site; the rest are not crawled or charged. Upgrade to any paid Apify plan for full crawls.

### Use with AI agents (MCP)

Use this Actor from Claude, ChatGPT, Cursor, VS Code or any other MCP client through the [Apify MCP server](https://mcp.apify.com). Add this server URL to your client and sign in with your Apify account when asked:

```text
https://mcp.apify.com?tools=sfoln/website-to-markdown
```

To run the MCP server locally instead, start `npx -y @apify/actors-mcp-server --tools sfoln/website-to-markdown` with your `APIFY_TOKEN` in the environment.

Example prompt:

> Crawl https://crawlee.dev/js/docs/introduction (at most 10 pages) and give me the pages as markdown plus the llms.txt file.

Runs started by an agent are billed like any other run (see Pricing). Agents can cap what a run may spend with `maxTotalChargeUsd`.

### API usage

Run the Actor and get its dataset items in one call (the call waits up to 5 minutes for the run to finish):

```bash
curl -X POST "https://api.apify.com/v2/acts/sfoln~website-to-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer <APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/introduction"], "maxPagesPerSite": 3}'
```

For large inputs, start the run asynchronously and download the dataset when it has finished:

```bash
## 1. Start a run: the response holds the run ID (data.id) and its dataset ID (data.defaultDatasetId)
curl -X POST "https://api.apify.com/v2/acts/sfoln~website-to-markdown/runs" \
  -H "Authorization: Bearer <APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/introduction"], "maxPagesPerSite": 3}'
## 2. Repeat until data.status is SUCCEEDED (each call waits up to 60 s)
curl "https://api.apify.com/v2/actor-runs/<runId>?waitForFinish=60" -H "Authorization: Bearer <APIFY_TOKEN>"
## 3. Download the results (format=json, csv or xlsx)
curl "https://api.apify.com/v2/datasets/<defaultDatasetId>/items?format=json" -H "Authorization: Bearer <APIFY_TOKEN>"
```

Python ([apify-client](https://pypi.org/project/apify-client/)):

```python
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("sfoln/website-to-markdown").call(run_input={"startUrls": ["https://crawlee.dev/js/docs/introduction"], "maxPagesPerSite": 3})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

Instead of polling, you can attach a webhook to the run (see Integrations). The API tab of this page shows the same calls for JavaScript and the Apify CLI.

### Integrations

- **Make, Zapier and n8n:** run `sfoln/website-to-markdown` with the Apify app (in n8n, the Apify node) and pass the dataset items to the next step of your workflow.
- **Google Sheets:** add the items to a sheet with the Google Sheets step of Make, Zapier or n8n, or download them as CSV or Excel from the run's Storage tab or the API (`format=csv` or `format=xlsx`).
- **Webhooks:** get a POST request when a run succeeds or fails: add a webhook in the Actor's Integrations tab, or pass `webhooks` when you start a run through the API.
- **Schedules:** run it hourly, daily or weekly with the same input from Apify Schedules; every run keeps its results in its own dataset.
- **More:** Slack, Google Drive, GitHub, LangChain, LlamaIndex and other [Apify integrations](https://docs.apify.com/platform/integrations).

### Related Actors

- [URL to Markdown & Article Extractor](https://apify.com/sfoln/url-to-markdown-summary)
- [RAG Web Search – Google + LLM-Ready Markdown](https://apify.com/sfoln/rag-web-search)
- [AI Web Scraper – Any URL to Structured JSON](https://apify.com/sfoln/ai-web-scraper)
- [Sitemap URL Extractor](https://apify.com/sfoln/sitemap-url-extractor)
- [Website Change Monitor – Text Diff Alerts](https://apify.com/sfoln/website-change-monitor)

See [all our Actors](https://apify.com/sfoln) on Apify Store.

### FAQ

#### How much does it cost, and am I charged for failed items?

You pay per event at the prices under Pricing above, plus a run start fee of $0.00005 per GB of memory, with no subscription; platform usage is included. Errors and skipped items (`skipped_…` statuses) are never charged; a lookup that finds nothing is charged only where Pricing says so, and every record's `charged` field shows whether it was billed. Set **Maximum cost per run** to cap what a single run can spend.

#### How fresh is the data?

Each run queries its source while it runs, so you get what the source returns at that moment. Source-specific details, such as how often a provider refreshes its own database, are under Limits & notes.

#### What are the limits?

Per-run caps (for example the maximum number of items) are listed under Input, source limits under Limits & notes, and the free-plan limit under Pricing. For big batches, start the run asynchronously (see API usage) and read the dataset when it has finished.

#### Can I schedule runs or get results by webhook?

Yes. Create a schedule in Apify Console to run it hourly, daily or weekly with the same input, and add a webhook to get a POST request when a run finishes (see Integrations).

#### What does this Actor record about my runs?

This Actor records operational metrics about each run (item counts, duration, charges, error codes and a pseudonymous hash of your Apify user ID) to monitor reliability and costs. It never stores your input or results.

#### Which pages are crawled?

The start page first, then the URLs of the site's sitemap (found through robots.txt or /sitemap.xml) that are in scope, shallow paths first, then the links found on each page, breadth-first up to maxDepth. crawlScope auto stays under the start URL's path when it has one (https://example.com/docs crawls /docs and below) and otherwise covers the whole registrable domain, subdomains included. includePatterns and excludePatterns narrow it further. Files and media (PDF, images, archives) are skipped, and each page is read once even when several URLs lead to it.

#### What are llms.txt and llms-full.txt?

llms.txt (llmstxt.org) is a markdown index of a site for language models: the site's name and description, then one section per top-level path listing every page as a link with a short description. llms-full.txt contains the markdown of every page one after the other (up to 8 MB). Both are written to the run's key-value store (`llms-<site>.txt` and `llms-full-<site>.txt`); the summary row of each start URL links to them. They are free.

#### Why do some pages have contentStatus blocked or js_required?

No browser is used. blocked: the site answered with a bot check, an access-denied page or a rate limit. js_required: the page builds its content with JavaScript and the HTML has no text. Such pages are written as uncharged rows so you can see what was skipped; their links (if any) are still followed. For JavaScript-heavy sites the sitemap often still leads to readable pages.

#### How polite is the crawl?

At most 2 requests per second and 2 at a time per host, shared by every start URL of the run, with a 20-second timeout and a 5 MB cap per page. Each site gets at most 2 page requests per page it may deliver (maxPagesPerSite, 20 on the free plan) plus 20, so failing links cannot make a crawl run away.

#### How do I get help?

Open an issue in the Actor's Issues tab and include the link to the run, so we can reproduce the problem. Feature requests are welcome there too.

### Limits & notes

- Pages are fetched by the Actor itself: 20-second timeout, 5 MB per page, at most 2 requests per second and 2 at a time per host, public http(s) hosts only. Each site gets at most 2 page requests per page it may deliver plus 20 (2 × `maxPagesPerSite` + 20; 60 on the free plan), also when few of the pages read are delivered (include/exclude patterns, pages without content); long crawls stop a minute before the time limit with `stopReason: "deadline"`.
- No browser is used: pages that build their content with JavaScript return `js_required`, and pages behind bot protection or a login return `blocked`. Sites that answer with markdown instead of HTML are delivered as they are.
- Files and media (PDF, office documents, images, archives, feeds) are never queued; use a document Actor for PDFs. Query strings are kept (so paginated listings are separate pages) but tracking parameters (`utm_*`, `gclid`, `fbclid`, …) are dropped, and identical content under several URLs is delivered once.
- `llms-full.txt` is capped at 8 MB per start URL; the summary's `llmsFullTxt.truncated` says when pages were left out of it (they are still in the dataset).
- When the platform moves the run to another server (or the run is aborted and resurrected), a start URL that was still being crawled is crawled again after the restart: the pages it had delivered are skipped (not delivered or charged again, kept in the run's key-value store record `DELIVERED_ROWS`), positions and `maxPagesPerSite` continue where it stopped, and its summary counts both parts. A page delivered in the last seconds before a hard crash (not a migration or an abort) may be delivered again.

# Actor input Schema

## `startUrls` (type: `array`):

Websites or site sections to crawl, one per line, e.g. https://docs.example.com or https://example.com/blog. Accounts on the Apify free plan can process up to 10 start URLs per run (20 pages each).

## `maxPagesPerSite` (type: `integer`):

Pages with content to deliver per start URL (1–10,000). Pages without content, duplicates and filtered pages do not count.

## `maxDepth` (type: `integer`):

How many links away from the start page to follow (0–20). Sitemap URLs count as depth 1; 0 reads only the start page.

## `crawlScope` (type: `string`):

auto: under the start URL's path when it has one, else the whole domain. path: the start URL's host, under its path. host: the start URL's host only. domain: every subdomain of the start URL's registrable domain.

## `useSitemap` (type: `boolean`):

Queue the site's sitemap URLs that are in scope before the links found on pages (sitemap from robots.txt or /sitemap.xml).

## `includePatterns` (type: `array`):

Deliver only pages whose URL matches one of these, one per line: a glob where \* matches anything (e.g. */blog/*) or a regular expression in slashes (e.g. //docs/v2//). Other pages are still read for their links. A regular expression that backtracks catastrophically is refused (prefer globs or character classes).

## `excludePatterns` (type: `array`):

Do not deliver pages whose URL matches one of these (same syntax as the include patterns), e.g. */tag/* or */page/*.

## `maxCharsPerPage` (type: `integer`):

Markdown kept per page (1,000–200,000 characters), cut at a paragraph boundary. About 4 characters make one LLM token.

## `llmsTxt` (type: `boolean`):

Write llms.txt (an index of the delivered pages) and llms-full.txt (all their markdown, up to 8 MB) per start URL to the run's key-value store. Free.

## `maxSites` (type: `integer`):

Safety cap on how many start URLs are crawled in this run; the rest are ignored and counted in the OUTPUT record.

## `concurrency` (type: `integer`):

Start URLs crawled at the same time (1–5). Each site is read at most 2 requests per second.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev/js/docs/introduction"
  ],
  "maxPagesPerSite": 3,
  "maxDepth": 5,
  "crawlScope": "auto",
  "useSitemap": true,
  "maxCharsPerPage": 50000,
  "llmsTxt": true,
  "maxSites": 100,
  "concurrency": 2
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev/js/docs/introduction"
    ],
    "maxPagesPerSite": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("sfoln/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev/js/docs/introduction"],
    "maxPagesPerSite": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("sfoln/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev/js/docs/introduction"
  ],
  "maxPagesPerSite": 3
}' |
apify call sfoln/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sfoln/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ufGJKdr2cD8ZdzrE0/builds/ifmtN8Irr0UfPAFL7/openapi.json