# Website to Markdown for RAG: Clean Text, Chunks, Sitemaps (`ataism/web-to-markdown`) Actor

Convert static web pages to clean Markdown, text and RAG chunks from URLs, sitemaps or a crawl. HTTP only, no JavaScript. $1.00 per 1,000 pages.

- **URL**: https://apify.com/ataism/web-to-markdown.md
- **Developed by:** [Kaizoku Data](https://apify.com/ataism) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown for RAG: Clean Text, Chunks, Sitemaps

Turn static web pages into clean Markdown, plain text and RAG-ready chunks. Give it page URLs, a sitemap, or a start URL to crawl. **$1.00 per 1,000 pages converted**; failed, blocked and empty pages are free.

> **No JavaScript.** This Actor fetches the HTML the server sends and does not run scripts. Pages that build their content in the browser (many single-page apps) come back with status `low-content`, a warning, and **no charge**. For JavaScript-heavy sites use a browser-based crawler such as [apify/website-content-crawler](https://apify.com/apify/website-content-crawler).

- **Three ways in:** a list of page URLs, sitemap URLs (sitemap index files and `.gz` files work), or a start URL with a same-site crawl and a page cap (default 10, maximum 5,000). Include and exclude patterns for sitemap and crawl URLs.
- **Clean output:** main content only (navigation, footers and scripts removed), as Markdown with tables and code blocks, plus plain text, title, description, language, headings, word count, approximate token count and a SHA-256 content hash. Links are optional.
- **RAG chunks:** split at headings, then paragraphs and sentences. You set the size and overlap in tokens. Every chunk carries its heading path and index.
- **Change detection:** set a key and each page gets `changedSinceLastRun`, so you only re-embed what changed.
- **Polite by default:** robots.txt is obeyed (disallowed pages become a free `blocked-by-robots` row), 1 request per second per host (0.75 to 4), honest User-Agent, no proxies, no work-arounds for blocks. A `Crawl-delay` in robots.txt is honoured in full, never shortened; sites that ask for more than 1.75 seconds are skipped (free `slow-host` rows): 1.75 seconds is the longest delay this Actor honours.

### Quick start

1. Open the Actor, keep the prefilled input (two pages of [quotes.toscrape.com](https://quotes.toscrape.com/), a public sandbox made for scraping practice) and press **Start**. It takes a few seconds.
2. Open the **Pages** view for an overview or **Markdown** for the converted text.
3. Replace the input with your own `urls`, `sitemapUrls` or `startUrls`.

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `urls` | list | none | Page URLs to convert. |
| `sitemapUrls` | list | none | Sitemap files; every listed page on the sitemap's own site is converted. |
| `startUrls` | list | none | Converted first, then the crawl follows links to other pages on the same site (same host, with or without `www`). |
| `maxPages` | integer | 10 | Most pages fetched in a run, all inputs together. Hard maximum 5,000. |
| `includeGlobs` / `excludeGlobs` | list | none | Patterns for sitemap and crawled URLs. `*` stays inside one path segment, `**` matches anything. A pattern starting with `/` is matched against the path (`/docs/**`), otherwise against the full URL. Exclude wins. At most 20 patterns of up to 200 characters each, with at most 8 wildcards (`*`, `**`, `?`) per pattern; URLs longer than 2,048 characters are not fetched. Pages you list yourself and start URLs are always converted. |
| `crawlDepth` | integer | 3 | Links away from a start URL (0 to 10). |
| `chunkSizeTokens` | integer | 512 | Chunk size in approximate tokens. 0 turns chunks off, otherwise at least 50. |
| `chunkOverlapTokens` | integer | 50 | Overlap when a long section has to be split. Capped at half the chunk size. |
| `includeMarkdown` / `includeText` / `includeLinks` | boolean | true / true / false | Which content fields are in each row. |
| `removeContactDetails` | boolean | true | Replaces email addresses and phone numbers with `[email removed]` / `[phone removed]`. Best effort. |
| `changeDetectionKey` | string | none | Letters, digits and dashes. Turns on `changedSinceLastRun`. |
| `requestsPerSecondPerHost` | number | 1 | 0.75 to 4 (a slower rate only makes the run longer and costlier). A `Crawl-delay` in robots.txt is honoured in full (never shortened) when it is longer than this gap. |
| `allowSlowHosts` | boolean | false | Kept for compatibility; it has no effect. The longest `Crawl-delay` this Actor honours is **1.75 seconds** (honoured in full, never shortened), with or without this switch: a site whose robots.txt asks for more is skipped with free `slow-host` rows ("over the 1.75 s maximum this Actor honours"), because at 1.75 seconds one page costs about 10% of its price in compute at $0.20 per compute unit. |
| `maxPageSizeMb` | integer | 2 | Larger pages are skipped (free). Hard cap 10 MB; see Limits for memory. |
| `pageTimeoutSecs` | integer | 30 | Per page, including redirects. |
| `concurrency` | integer | 4 | Pages fetched in parallel across hosts. One host is never hit faster than its rate limit. |

#### Example 1: a few pages

```json
{
  "urls": ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/page/2/"],
  "maxPages": 10
}
```

#### Example 2: a documentation site from its sitemap, chunked for embeddings

```json
{
  "sitemapUrls": ["https://docs.example.org/sitemap.xml"],
  "includeGlobs": ["/guide/**"],
  "excludeGlobs": ["**/changelog/**"],
  "maxPages": 500,
  "chunkSizeTokens": 400,
  "chunkOverlapTokens": 40,
  "includeText": false
}
```

#### Example 3: crawl one site and re-run weekly to find changes

```json
{
  "startUrls": ["https://example.org/blog"],
  "crawlDepth": 2,
  "maxPages": 200,
  "includeLinks": true,
  "changeDetectionKey": "example-blog"
}
```

### Output

One dataset row per page. Shortened example:

```json
{
  "url": "https://quotes.toscrape.com/",
  "source": "start",
  "depth": 0,
  "status": "ok",
  "statusCode": 200,
  "title": "Quotes to Scrape",
  "description": null,
  "lang": "en",
  "wordCount": 213,
  "tokenCountApprox": 293,
  "contentSha256": "4107e3cc3dd8b992c11647c0ac51a16308678888e82d4e42630d55a87f99b031",
  "changedSinceLastRun": null,
  "headings": [{ "level": 1, "text": "Quotes to Scrape" }],
  "markdown": "# Quotes to Scrape\n\n“The world as we have created it is a process of our thinking. ...",
  "text": "Quotes to Scrape\n\n“The world as we have created it is a process of our thinking. ...",
  "chunks": [
    { "index": 0, "text": "# Quotes to Scrape\n\n“The world as we have created it ...", "tokens": 425, "headingPath": ["Quotes to Scrape"] }
  ],
  "extraction": "readability",
  "warnings": [],
  "bytes": 1962,
  "fetchedAt": "2026-10-20T12:41:41.184Z"
}
```

Notes on the fields:

- `status` is `ok` (charged), `low-content` (fewer than 10 words found; free), `blocked-by-robots` (free) or `error` (free, with `errorCode` and `message`).
- A very large page can make a row too big for one dataset row (about 8.5 MB is the limit the Actor keeps). The Actor then drops `chunks`, then `text`, then cuts the end of the Markdown, and says so in `warnings` (`chunks-dropped-row-too-large`, `text-dropped-row-too-large`, `markdown-truncated-row-too-large`). The page is still saved and charged. If even that is not enough the row is a free `row-too-large` error.
- `finalUrl` appears when the page redirected. `source` is `url`, `start`, `sitemap`, `crawl` or `sitemap-file` (a sitemap problem).
- `tokenCountApprox` and chunk `tokens` are **characters divided by 4**. This is a rough rule for English; real tokenizers differ by model and language.
- `contentSha256` is the SHA-256 of `text` (after emails and phones are removed, if that option is on). `wordCount` counts whitespace-separated words, so it under-counts Chinese and Japanese.
- `changedSinceLastRun` is `true` for a page not seen before with this key, `false` if the text is identical, and `null` when no key is set. It is only updated for pages that were converted and saved.
- `chunks[].headingPath` is the heading trail of the section where the chunk starts. Small neighbouring sections are packed into one chunk; a section longer than the chunk size is split with the overlap you set.
- `extraction` is `readability` (Mozilla Readability, the algorithm behind Firefox Reader View) or `body-fallback` (the body text without navigation, headers and footers) when Readability found too little.
- Images are not included. Links in the Markdown are made absolute. `mailto:` and `tel:` links keep their text and lose the target.

A `RUN_SUMMARY` record in the key-value store lists counts per status, pages charged, bytes downloaded, failures by error code and why the run stopped (`finished`, `max-charge`, `failure-limit` or `timeout`). `failure-limit` also covers a site made of free empty pages: the run stops when at least 100 `low-content` pages make up at least 80% of the pages fetched.

### Error codes

| `errorCode` | Meaning |
|---|---|
| `http` | The server answered something other than 200 (see `statusCode`). 401, 403, 429 and 503 are never retried or worked around; after three in a row from one host, its remaining pages get `host-refused`. |
| `bot-challenge` | The page is a bot-check or block page (for example a Cloudflare "Just a moment..." page or an "Access Denied" page), not the real content. Free; counts toward the three refusals that stop a host; never worked around. |
| `host-refused` | The host kept refusing requests; the rest of its pages were skipped. |
| `not-html` | The URL returned a PDF, image, JSON or other non-HTML file. |
| `too-large` | The page is above Max page size (also checked after decompression). |
| `timeout`, `network`, `dns` | The host did not answer, dropped the connection or does not exist. |
| `redirects` | More than 5 redirects. |
| `blocked` | The address is private, local or otherwise not allowed (also after a redirect). |
| `platform` | Social, video and Google sites are refused: this Actor does not read them. |
| `invalid-url` | Not an http(s) URL. |
| `robots-unavailable` | robots.txt answered with a server error or could not be read, so the host was not fetched. A robots.txt that answers 4xx (for example 404, missing) is treated as "no rules", as RFC 9309 says. A 200 answer is judged by its content, whatever its Content-Type: real `User-agent:` rules apply, but an HTML page (a "soft 404"), a binary file or text without any `User-agent:` line holds no rules. 429 and 5xx are tried once more, then the host is skipped. |
| `slow-host` | robots.txt of the host asks for a `Crawl-delay` over 1.75 seconds (the maximum this Actor honours; skipped whether or not `allowSlowHosts` is on, and the delay is never shortened). Free; nothing was fetched from the host except its robots.txt. The same code appears on a sitemap row when the sitemap's host is slow. |
| `not-processed` | The run reached its time limit (the platform timeout minus 15 seconds) before this page could be fetched or converted. Free. |
| `extract-failed`, `extract-timeout`, `too-deeply-nested` | The page could not be converted: it crashed or ran out of memory in the conversion process, took longer than the time limit (6 s plus 6 s per MB of HTML on a full CPU core, divided by the CPU share of the run: about 4 times that at the default 1,024 MB; at most 240 s), or its HTML is nested more than 1,000 levels deep. Free. The run stops after 120 s spent on such pages. |
| `row-too-large`, `push-failed` | The converted page does not fit in one dataset row even after cutting, or the row could not be saved. Free. |
| `bad-sitemap`, `bad-type`, `too-many-sitemaps`, `sitemap-depth`, `filter-budget` | Problems with a sitemap file (`source` is `sitemap-file`). A root sitemap that fails gets its own row; child sitemap files that fail or are skipped give one summary row per sitemap URL you gave, with a `count`. |

### Pricing

One event, **`page-converted`: $0.0010 per page** that was converted and saved (status `ok`). Apify platform usage is included. There is no start fee. Free rows are never charged: errors, `blocked-by-robots` and `low-content`.

| Pages converted | Cost |
|---|---|
| 10 | $0.01 |
| 100 | $0.10 |
| 1,000 | $1.00 |
| 5,000 (one run, the maximum) | $5.00 |

Prices of other tools change, so compare current numbers on the Store: at the time of writing (2026-10-18) `apify/web-fetch` listed $0.0015 per fetch and `6sigmag/fast-website-content-crawler` $0.003 per page plus a start fee. If you set a maximum cost per run, the Actor stops cleanly when it is reached and never fetches pages it could not charge for.

### Limits

- **Static HTML only.** No JavaScript, no login, no cookies, no CAPTCHA solving, no proxies. Pages behind a login or a bot-protection challenge return an error row.
- Up to 5,000 pages per run; up to 5,000 listed URLs, 50 sitemap URLs and 50 start URLs; sitemap indexes up to 3 levels and 100 files (index entries are followed, never returned as pages); sitemaps up to 30 MB uncompressed.
- Rate: 1 request per second per host by default, between 0.75 and 4. At the default one site gives about 3,600 pages per hour; start several runs for several sites, or split a large site. A `Crawl-delay` in robots.txt is always honoured in full (also for sitemap files and redirect targets), never shortened. Above 1.75 seconds the host is skipped (free `slow-host` rows), because waiting between requests makes the run long and its compute cost would exceed 10% of the page price; `allowSlowHosts` does not change this for this Actor.
- **Memory:** the default is 1,024 MB (the minimum). Pages are converted one at a time in a separate process with a 300 MB heap limit (two processes from 2,048 MB of run memory); a page that exhausts it, crashes it or takes too long fails free and the run goes on. Measured on a local machine, not yet on the platform: a worst-case set of 2 MB pages peaked at about 750 MB in total. The default handles pages up to about 4.5 MB; the Actor lowers Max page size to fit the memory the run has. Typical pages are 50 to 300 KB.
- **Restarts:** progress is saved every 2 seconds and when the run migrates. Pages already saved are found again in the dataset after a restart and are not converted or charged twice. Links first found on pages finished in the last 2 seconds before a restart may be missed by a crawl.
- The run stops after 100 failed pages, 100 MB downloaded for pages that were not charged, 120 s spent on pages that could not be converted, or 100 free `low-content` pages that are at least 80% of the pages fetched, so a list of dead links or a site made of JavaScript shells cannot run up a bill.
- `maxPages` counts every URL fetched, including errors and blocked pages.
- Plain text, Markdown and chunks come from the main-content guess of the extraction library. Unusual layouts, listing pages and forum threads may keep or drop sections you would not.

### Privacy and terms

- You are responsible for having the right to fetch and use the pages you give it, and for following each site's terms. The Actor only fetches URLs you list, URLs in sitemaps you list, and (with `startUrls`) same-site links from your start pages.
- It identifies itself as `web-to-markdown-actor/0.1 (https://apify.com/ataism/web-to-markdown)` and obeys robots.txt (including `Crawl-delay`, honoured in full up to 1.75 seconds; hosts that ask for more are skipped). Pages that robots.txt disallows are not fetched.
- Emails and phone numbers are removed from the text, Markdown, headings, titles, descriptions, chunks and link texts by default, and emails are removed from link URLs (`links[].url`; such links are not followed by a crawl). Phone numbers are recognised by their shape (international `+` numbers, bracketed area codes, 3-3-4 numbers, national numbers starting with 0); plain digit groups such as table rows, prices, dates and IDs are left alone. This is pattern matching and will miss unusual formats; names and addresses are not removed. Pages can contain other personal data. Turn the option off only if you have the right to collect that data.
- Page content is processed in memory and returned in the dataset. Nothing else is stored, except with `changeDetectionKey`: a named key-value store `w2m-hashes-<key>` in your account holds a map of page URL to content hash (no page text).

### For AI agents (MCP)

**Purpose:** fetch static web pages (no JavaScript) and return the main content as Markdown, plain text and RAG chunks, one row per page. Use it for documentation sites, blogs and articles; for pages built in the browser use a browser-based crawler.

**Minimal input that works** (one page):

```json
{ "urls": ["https://quotes.toscrape.com/page/2/"] }
```

To read a whole site, give a start page or a sitemap and raise `maxPages` (default is only 10): `{"startUrls": ["https://docs.example.org/"], "maxPages": 50}` or `{"sitemapUrls": ["https://docs.example.org/sitemap.xml"], "maxPages": 50}`. At least one of `urls`, `sitemapUrls`, `startUrls` is required.

**One output row per page** (trimmed): `{"url": "https://quotes.toscrape.com/page/2/", "status": "ok", "title": "Quotes to Scrape", "lang": "en", "wordCount": 213, "markdown": "# Quotes to Scrape\n\n...", "text": "...", "chunks": [{"index": 0, "text": "...", "tokens": 425, "headingPath": ["Quotes to Scrape"]}], "contentSha256": "...", "fetchedAt": "..."}`. Always check `status`: `ok`, `low-content`, `blocked-by-robots` or `error` (with `errorCode` and `message`). To get smaller rows, set `"includeText": false` or `"chunkSizeTokens": 0`.

**Cost:** $0.001 per page with status `ok` (1 page = $0.001, 100 = $0.10, 1,000 = $1.00); this Actor adds no start fee. Free: `error`, `blocked-by-robots` and `low-content` rows. `maxPages` caps the cost at about `maxPages` x $0.001 (the default 10 is at most $0.01). Apify's *Max total charge* also stops the run cleanly.

**Limits:** HTTP only, no JavaScript, no login, no CAPTCHA solving, no proxies. robots.txt is obeyed and blocks are never worked around. Up to 5,000 pages per run, 1 request per second per host by default, pages up to 2 MB by default (10 MB max). Emails and phone numbers in the output are replaced by default (best effort). Social, video and Google sites are refused.

**How to call it via the Apify MCP server** (`https://mcp.apify.com`; tool names from the [apify-mcp-server README](https://github.com/apify/apify-mcp-server)):

1. `search-actors` with a query such as `website to markdown for RAG` (free) may list `ataism/web-to-markdown`; search results are ranked, so if it isn't listed, go straight to step 2 with the Actor name.
2. `fetch-actor-details` with actor `ataism/web-to-markdown` returns the input schema, pricing and output schema (free).
3. `call-actor` with `actor` = `ataism/web-to-markdown` and `input` = `{"urls": ["https://quotes.toscrape.com/page/2/"]}` runs it. The response holds run metadata and a dataset ID, not the rows.
4. `get-dataset-items` with that dataset ID returns the rows (follow the `nextStep` in the `call-actor` response).

To expose only this Actor as a tool, connect to `https://mcp.apify.com?tools=ataism/web-to-markdown`.

### Use it via API, n8n or Make

**cURL** (returns the dataset items when the run finishes):

```bash
curl -X POST "https://api.apify.com/v2/acts/ataism~web-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://quotes.toscrape.com/"], "maxPages": 5}'
```

**JavaScript** (`apify-client`):

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('ataism/web-to-markdown').call({
    sitemapUrls: ['https://docs.example.org/sitemap.xml'],
    maxPages: 100,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const page of items) console.log(page.url, page.status, page.wordCount);
```

**Python** (`apify-client`):

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("ataism/web-to-markdown").call(run_input={
    "startUrls": ["https://example.org/blog"],
    "maxPages": 50,
    "changeDetectionKey": "example-blog",
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page["status"] == "ok" and page["changedSinceLastRun"]:
        print(page["url"], page["title"])
```

**n8n:** add the Apify node, choose **Run an Actor and get dataset**, select `ataism/web-to-markdown`, and paste your input JSON.

**Make:** add the Apify module **Run an Actor** (Actor `ataism/web-to-markdown`, input JSON), then **Get Dataset Items** with the run's default dataset ID.

# Actor input Schema

## `urls` (type: `array`):

Page URLs to convert, one per entry, e.g. \["https://quotes.toscrape.com/page/2/"]. Each page becomes one row. Costs $0.001 per page converted (status ok); errors, blocked, robots-disallowed and empty (low-content) pages are free. Private, local and internal addresses are blocked; social, video and Google sites are refused. Give at least one of urls, sitemapUrls or startUrls.

## `sitemapUrls` (type: `array`):

Sitemap files to expand into pages, e.g. \["https://docs.example.org/sitemap.xml"] (sitemap index files and .gz files work). Every page listed is converted, up to Max pages. Only pages on the sitemap's own site are used. Each converted page costs $0.001. Narrow it with Include/Exclude patterns.

## `startUrls` (type: `array`):

Start pages for a same-site crawl, e.g. \["https://quotes.toscrape.com/"]. Converts each start page, then follows links to other pages on the same site (the same host, with or without www) up to Crawl depth and Max pages. Each converted page costs $0.001.

## `maxPages` (type: `integer`):

Hard cap on pages fetched in one run, counting all inputs together (start URLs first, then page URLs, then sitemaps, then crawled links), e.g. 50. Default 10, minimum 1, maximum 5,000. This also caps the cost: at most about maxPages x $0.001.

## `includeGlobs` (type: `array`):

Only sitemap and crawled URLs matching one of these are converted, e.g. \["/docs/**"]. "\*" matches within one path segment, "**" matches anything. A pattern starting with "/" is matched against the path; otherwise against the full URL. Max 20 patterns. Pages you list yourself and start URLs are always converted. Empty = no filter.

## `excludeGlobs` (type: `array`):

Sitemap and crawled URLs matching any of these are skipped, e.g. \["**/changelog/**", "/tag/\*\*"]. Same pattern rules as Include patterns. Exclude wins over include. Empty = nothing excluded.

## `crawlDepth` (type: `integer`):

How many links away from a start URL the crawl may go, e.g. 2. 0 converts the start pages only. Default 3, maximum 10. Only affects Start URLs; the Max pages cap still applies.

## `chunkSizeTokens` (type: `integer`):

Size of each RAG chunk in approximate tokens (characters divided by 4), e.g. 512. Chunks are split at headings, then paragraphs and sentences, and returned in the "chunks" field. 0 turns chunks off; otherwise at least 50, maximum 8,000. Default 512. No extra charge.

## `chunkOverlapTokens` (type: `integer`):

Tokens repeated at the start of the next chunk when a long section has to be split, e.g. 50. Default 50, maximum 4,000; capped at half the chunk size. Only used when chunk size is above 0.

## `includeMarkdown` (type: `boolean`):

true (default): add the "markdown" field (main content as GFM Markdown). false: omit it to shrink rows. No change to price.

## `includeText` (type: `boolean`):

true (default): add the "text" field (main content as plain text). false: omit it to shrink rows. No change to price.

## `includeLinks` (type: `boolean`):

true: add a "links" list with every http(s) link on the page (objects with url and text, up to 2,000). false (default): no links. No extra charge.

## `removeContactDetails` (type: `boolean`):

true (default): replace email addresses and phone numbers in the output with \[email removed] and \[phone removed]. Best effort: unusual formats can slip through. Set false only if you have the right to collect them.

## `changeDetectionKey` (type: `string`):

Optional label for change detection, e.g. "docs-weekly" (letters, digits and dashes, up to 40 characters). When set, the Actor remembers a hash of each page's text in a named storage and sets "changedSinceLastRun" (true or false) on later runs that use the same key. Leave empty to turn it off.

## `requestsPerSecondPerHost` (type: `number`):

Politeness limit per website, in requests per second, e.g. 1. Default 1, minimum 0.75, maximum 4 (a slower rate only makes the run longer and costlier). A Crawl-delay in robots.txt is also honoured in full, never shortened; see Allow slow hosts.

## `allowSlowHosts` (type: `boolean`):

Kept for compatibility; it has no effect. The longest Crawl-delay this Actor honours is 1.75 seconds (in full, never shortened), with or without this switch. A site whose robots.txt asks for more is skipped with free "slow-host" rows, because waiting between requests makes the run slow and at 1.75 seconds one page costs about 10% of its price in compute at $0.20 per compute unit.

## `maxPageSizeMb` (type: `integer`):

Largest page to download, in MB, e.g. 2. Larger pages are skipped with an error row (not charged). Default 2, hard cap 10. The limit also depends on run memory (about 4.5 MB at the default 1,024 MB); the Actor lowers it when the run has less memory.

## `pageTimeoutSecs` (type: `integer`):

Time allowed per page in seconds, including redirects, e.g. 30. Default 30, minimum 5, maximum 120. Pages that time out are free error rows.

## `concurrency` (type: `integer`):

Pages fetched at once across different hosts, e.g. 4. Default 4, maximum 8. One host is never hit faster than Requests per second per host.

## `mode` (type: `string`):

Internal: "selfTest" runs the maintainer health check and saves no items. Leave as "run" (the default).

## Actor input object example

```json
{
  "urls": [
    "https://quotes.toscrape.com/page/2/"
  ],
  "startUrls": [
    "https://quotes.toscrape.com/"
  ],
  "maxPages": 2,
  "crawlDepth": 3,
  "chunkSizeTokens": 512,
  "chunkOverlapTokens": 50,
  "includeMarkdown": true,
  "includeText": true,
  "includeLinks": false,
  "removeContactDetails": true,
  "requestsPerSecondPerHost": 1,
  "allowSlowHosts": false,
  "maxPageSizeMb": 2,
  "pageTimeoutSecs": 30,
  "concurrency": 4,
  "mode": "run"
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page with status, Markdown, plain text, title, headings, RAG chunks and a content hash (JSON, CSV, Excel or XML).

## `runSummary` (type: `string`):

Counts per status, pages charged, bytes downloaded, failures by code and why the run stopped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://quotes.toscrape.com/page/2/"
    ],
    "startUrls": [
        "https://quotes.toscrape.com/"
    ],
    "maxPages": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("ataism/web-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://quotes.toscrape.com/page/2/"],
    "startUrls": ["https://quotes.toscrape.com/"],
    "maxPages": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("ataism/web-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://quotes.toscrape.com/page/2/"
  ],
  "startUrls": [
    "https://quotes.toscrape.com/"
  ],
  "maxPages": 2
}' |
apify call ataism/web-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ataism/web-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JsEDDFuB5Syay61qt/builds/ygtpVV2f4tIRyXzYA/openapi.json