# Web Page to Markdown for LLMs (robots.txt-safe) (`datafeed/web-page-to-markdown`) Actor

Clean Markdown and RAG chunks from public pages or whole doc sites. Sitemap-first polite crawl: robots.txt, Crawl-delay, named bot, hard page/time budgets. Flat per-page price; cheap unchanged-page recrawls. No browser, no logins.

- **URL**: https://apify.com/datafeed/web-page-to-markdown.md
- **Developed by:** [Benjamin Cojocaru](https://apify.com/datafeed) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Page to Markdown for LLMs (polite crawler, robots.txt-respecting)

Turn public web pages, or a whole documentation site, into clean Markdown for LLMs and RAG: headings, lists, links, code blocks and tables kept; navigation, cookie banners, share bars, newsletter boxes, related-post lists and comments removed. Optional token-budgeted chunks with the source URL, heading path and character offsets, ready for citations.

It is built to be a **well-behaved bot**: it reads each site's `robots.txt` before fetching anything (sitemaps included), re-checks on every redirect, sends an identifiable User-Agent, waits between requests to the same site and honours `Crawl-delay`, and skips pages that opt out of AI use with `noai`. You pay a flat price per page, never for compute, and nothing for pages that fail.

### Why this one

- **Cleaner main content, biased towards keeping text.** When in doubt it keeps a block: losing content is worse than leaking a menu. On the 25 hand-written test pages in [BENCHMARK.md](BENCHMARK.md) (benchmark pages only, not a guarantee for every site) it found every expected content block. It leaked 21% of the boilerplate strings on the 8 pages written after the rules were frozen; v0.1 leaked 39% and lost 10% of the content. Every rule is listed below.
- **Never loops, never times out silently.** Hard `maxPages` and `maxRunMinutes` budgets, a stop before the platform run timeout, and each request's deadline shrunk to fit the time left. URLs are de-duplicated after normalisation, and crawler traps (repeating paths, endless query strings, binary files) are refused. Every run ends with a `SUMMARY` that says why it stopped.
- **Predictable price.** Pay per page (`page-converted`). In incremental mode an unchanged page costs only `page-unchanged`. Failed and skipped pages are free. The charge is taken in the same call that stores the item.
- **Chunks that keep order.** Chunks follow the page in order. Every heading starts a new chunk, and long sections are split at paragraphs, then sentences. Code fences stay whole while they fit. Overlap never reaches back across the chunk's heading, so `headingPath` is right for the whole chunk. Each chunk is an exact slice `markdown[charStart:charEnd]`.

### What it does and does not do

- Fetches plain HTTP(S) pages with a normal GET request. **No browser, no JavaScript, no logins, no cookies, no proxies, no CAPTCHA solving, no search-engine results.** Pages that need JavaScript to show their text come back as `empty_content` errors (free).
- Respects `robots.txt` per RFC 9309 for the user-agent token `DataFeedBot` (falls back to the `*` group). A site whose robots.txt answers with a server error is treated as "disallow everything", as the RFC says. Honours `<meta name="robots" content="nofollow">` and `rel="nofollow"` when crawling.
- Skips pages marked `<meta name="robots" content="noai">` or `X-Robots-Tag: noai`.
- Refuses URLs pointing at private or internal network addresses (DNS is resolved once and the connection is pinned to the checked address).
- You are responsible for having the right to use the content you convert.

### Modes

**`list`** (default) converts exactly the URLs you give (up to 500).

**`crawl`** starts from up to 20 start URLs and works in this order:

1. It fetches the start URL.
2. It reads the site's sitemaps: the ones `robots.txt` names, otherwise `/sitemap.xml`. It reads at most 20 sitemap files, and only from the same site.
3. It follows links on the crawled pages, breadth-first, up to `maxDepth` clicks.

The crawl stays on the start URL's host and inside its directory (`https://x.com/docs/intro` stays under `/docs/`), unless you set `includePaths`. Links that robots.txt forbids are never fetched and never appear in the dataset. They are counted in `SUMMARY.rejectedUrls`. If a site's `Crawl-delay` is longer than 10 s, the Actor fetches one page from it and drops the rest of that site, with a note in the warnings.

### Extraction rules (main-content mode)

1. **Root:** `<main>`, or the single top-level `<article>` (widgets rendered as `<article>` are ignored), accepted only if it holds the page's `<h1>` or at least 30% of the page's prose. Several articles: the one holding 60%+ of article text, otherwise all of them. Then `role=main`. Otherwise a readability-style density score, widened over neighbouring prose (not menus) and to include the page's `<h1>`.
2. **Always removed:** scripts, styles, forms with little text, buttons, `hidden` / `aria-hidden` / `display:none` / `visibility:hidden` elements. Exception: hidden tab panels (`role=tabpanel`, e.g. code in other languages) are kept and labelled with their tab name. A `<form>` that wraps the whole page (ASP.NET) is kept.
3. **Page chrome:** `<nav>`, `<footer>`, `<header>`, `<aside>`, and ARIA roles `navigation`, `banner`, `contentinfo`, `complementary`, `search`, `dialog`, `menu`, `toolbar`. Kept anyway: an article's own `<header>` with its title, an `<aside>` that is a note/warning or holds real prose, and a role-marked section with its own heading and prose.
4. **Boilerplate class and id names:** a curated list matched against each *whole* class name, for example `cookie-banner`, `share-buttons`, `related-posts`, `newsletter`, `ad-slot`, `sidebar`, `comments`, `breadcrumb`, `toc`, `sr-only`. A word inside a longer name never counts (`social-security`, `related-work`, `menu-planning` and `ad-hoc` stay). A block is never dropped by this rule if it holds the `<h1>`, more than half of the content, or 200+ characters of prose.
5. **Cookie and consent banners by text:** a short block mentioning cookies plus "accept all", "we use cookies", "consent", "manage preferences" and the like.
6. **Link farms:** a `div`/`section` with 4+ links that is 70%+ link text and small compared with the page.
7. **"Related posts", "Read next", "More from ..." headings**, only when followed by a list of real links (not in-page anchors). The heading and that list are dropped; a heading followed by prose is kept.
8. **Heading permalink anchors** (`¶`, `#`, `hash-link`) and **"Edit this page" links.**
9. **Kept intact:** tables (GFM tables plus structured `tables`; `colspan` and `rowspan` expanded; code in a cell becomes inline code), code blocks (language read from `language-*`, `highlight-*` or `data-lang`; a real line-number gutter table becomes one code block), lists (nested and ordered), definition lists (code inside `<dd>` stays fenced), blockquotes, images with alt text.
10. **Crawl mode only (`removeRepeatedBlocks`):** a short paragraph that already appeared on 3 earlier pages of the same site is dropped. Headings, code and tables are never dropped by this rule.

### Input

```json
{
  "urls": ["https://docs.example.com/guide/"],
  "mode": "crawl",
  "maxPages": 200,
  "maxDepth": 3,
  "maxRunMinutes": 30,
  "incremental": true,
    "chunk": true,
  "chunkTokens": 512
}
```

All fields except `urls` are optional. The full list, with defaults, is in the input schema: `mainContentOnly`, `includeTables`, `maxChars`, `maxPageMb`, `useSitemaps`, `followLinks`, `includePaths`, `excludePaths`, `removeRepeatedBlocks`, `chunkOverlapTokens`.

### Output (one dataset item per page)

```json
{
  "url": "https://docs.example.com/guide/limits",
  "finalUrl": "https://docs.example.com/guide/limits",
  "canonicalUrl": "https://docs.example.com/guide/limits",
  "status": "ok",
  "change": "changed",
  "depth": 1,
  "source": "sitemap",
  "title": "Rate limits | Example Docs",
  "description": "How many requests you may send",
  "language": "en",
  "markdown": "# Rate limits\n\nEvery API key may send 600 requests per minute...",
  "truncated": false,
  "wordCount": 412,
  "approxTokens": 690,
  "contentType": "text/html",
  "boilerplateRemoved": 7,
  "contentHash": "sha256:5f1c...",
  "tables": [{"index": 0, "caption": null, "headers": ["Header", "Description"], "headerRows": [["Header", "Description"]], "rows": [["Retry-After", "Seconds to wait after a 429"]]}],
  "chunks": [{"index": 0, "chunkId": "a1b2c3d4e5f60718", "url": "https://docs.example.com/guide/limits", "headingPath": ["Rate limits"], "charStart": 0, "charEnd": 1630, "approxTokens": 408, "text": "# Rate limits\n\n..."}],
  "fetchedAt": "2026-10-14T00:00:00+00:00",
  "error": null
}
```

- `status` can be:
  - `ok`: converted.
  - `unchanged`: incremental mode only. The item has no Markdown and costs only `page-unchanged`.
  - `skipped`: robots.txt, a `noai` opt-out, or a Crawl-delay that is too long.
  - `error`: an HTTP error, a page that is too large, not HTML, or empty.
- Skipped and error items explain why in `error.code` / `error.message` and are **not charged**.
- `change` appears in incremental mode only: `new`, `changed` or `unchanged`.
- `depth` and `source` (`seed` / `sitemap` / `link` / `state`) appear in crawl mode only.
- `approxTokens` is characters / 4, an estimate rather than a real tokenizer count. Chunk budgets use the same estimate.
- `contentHash` is taken from the extracted Markdown, before the repeated-block step. A rotating ad or a timestamp in the page chrome therefore does not count as a change.

The `SUMMARY` record in the run's key-value store contains:

- the counts per status
- `stopReason`: `null`, `page_budget`, `time_budget`, `charge_limit` or `request_budget`
- `notFetched`: pages discovered but not fetched
- `discovered` and `rejectedUrls`, with the reason for each rejected URL
- the incremental state
- any warnings

**Run-summary row (free).** When a run produces no other row (for example a monitor run where nothing changed, or a run stopped at once by your maximum charge or the run timeout), the Actor adds one row `{"type": "run_summary", "status": "run_summary", "message": "...", "stopReason": ..., "warnings": [...], "summary": {...}}` (`summary` is the same as the `SUMMARY` record), so the dataset is never empty. It is never charged. Skip it with `type != "run_summary"`.

### Incremental recrawls

With `incremental: true`, each page's ETag, Last-Modified and content hash are saved in a named key-value store. By default the name is derived from the mode, start URLs and paths (`wpm-state-...`), so different jobs never share state; set `stateStoreName` to choose it. On the next run, the Actor sends `If-None-Match` / `If-Modified-Since`. A page is reported `unchanged` when the server answers 304, or when it sends the page again and the extracted Markdown (hashed in full, before `maxChars` truncation) is the same. In crawl mode, newly discovered pages are fetched first; pages known from earlier runs (`source: "state"`) are re-checked with the budget that is left, so re-checks can never stop new pages from being found. Pages that now answer 404/410 or are disallowed by robots.txt are dropped from the state (counted in `SUMMARY.incremental.removedPages` for 404/410). A 304 page that now sends `X-Robots-Tag: noai` is skipped and not charged. If the extraction settings (`mainContentOnly`, `includeTables`, `maxChars`, Actor version) changed since the state was written, all pages are converted again and a warning says so. If the store cannot be read or written, the run does not fail: every page is treated as new and a warning says so.

**Restarts.** The run records which pages it has already stored in its own key-value store (`PROGRESS`). If the platform migrates or restarts the run, those pages are not fetched or charged again (`SUMMARY.resumedSkipped`).

### Pricing

Pay per event:

| Event | When | Suggested price |
|---|---|---|
| `page-converted` | A page with `status: ok` was stored (chunks included, free) | $0.0015 |
| `page-unchanged` | Incremental mode: a page checked and found unchanged was stored | $0.0002 |

Nothing else is charged by this Actor: no compute surcharge, no start fee of its own, no charge for skipped or failed pages, robots.txt or sitemaps. Each charge is taken in the same call that stores the item, so you are never charged for an item that was not stored. Before each page, the Actor checks that a `page-converted` still fits your maximum charge per run. If it does not, the Actor stops and SUMMARY says so.

### Using it from an AI agent

The input is a plain URL list and the output is one JSON object per page, so the Actor works as a tool via the Apify API or Apify's MCP server. Turn on `chunk` to get embedding-ready chunks with citation offsets directly. You can also feed `markdown` into **RAG Text Chunker** for other chunking options.

### Limits

- List mode: 500 URLs per run. Crawl mode: 20 start URLs and 1,000 pages per run.
- 10 MB per page.
- 2,000 HTTP requests per run (robots.txt and sitemaps included). Sitemaps: 20 files per site.
- 5 redirects.
- At least 1 s between two requests to one site, and at most 10 s of waiting for `Crawl-delay`.
- Tables: `colspan` and `rowspan` are expanded. Several header rows are merged per column (`"Price / Min"`) and also returned raw in `headerRows`.
- No JavaScript rendering. Single-page apps whose HTML has no text come back as `empty_content` (free).

# Actor input Schema

## `urls` (type: `array`):

List mode: the pages to convert (up to 500, duplicates removed). Crawl mode: up to 20 start URLs; the crawl stays on their host and inside their path (https://x.com/docs/intro stays under /docs/). Pages that robots.txt disallows are never fetched and never charged.

## `mode` (type: `string`):

list: convert the URLs given. crawl: read the sites' sitemaps, then follow links inside the start path, up to maxPages and maxRunMinutes.

## `maxPages` (type: `integer`):

Hard cap on pages fetched in crawl mode (sitemaps and robots.txt not counted). Discovered pages beyond it are counted as notFetched in SUMMARY.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL the crawl may go (sitemap URLs count as depth 1). 0 = start URLs only.

## `useSitemaps` (type: `boolean`):

Read the sitemaps listed in robots.txt (or /sitemap.xml) before following links.

## `followLinks` (type: `boolean`):

Follow links found on crawled pages (same host, inside the start path).

## `includePaths` (type: `array`):

Path prefixes such as /docs/ that replace the start-URL path scope. Empty = the start URLs' own directories.

## `excludePaths` (type: `array`):

Path prefixes never fetched, e.g. /blog/tag/.

## `removeRepeatedBlocks` (type: `boolean`):

Crawl mode: drop paragraphs that already appeared on 3 earlier pages of the same site (leftover banners, footers). Headings, code and tables are always kept.

## `maxRunMinutes` (type: `integer`):

The run stops starting new pages after this time and always finishes cleanly with a SUMMARY (it also stops before the platform run timeout).

## `mainContentOnly` (type: `boolean`):

Keep the main content only: drop navigation, headers, footers, sidebars, cookie and consent banners, share bars, newsletter boxes, related-post lists and comments. Tables, code blocks, headings and lists are kept. Turn off to convert the whole <body>.

## `includeTables` (type: `boolean`):

Also return every HTML table as {caption, headers, rows} next to the Markdown.

## `maxChars` (type: `integer`):

Longer Markdown is cut and marked truncated: true.

## `maxPageMb` (type: `integer`):

Pages larger than this are reported as errors (not charged).

## `incremental` (type: `boolean`):

Remember each page's ETag, Last-Modified and content hash in a named key-value store. Next time, unchanged pages are reported as status 'unchanged' and charged the much cheaper page-unchanged event.

## `stateStoreName` (type: `string`):

Named key-value store that keeps the recrawl state (letters, digits, '-'). Empty = one store per job, named from the mode, start URLs and paths. Set a name to share state between differently configured runs.

## `chunk` (type: `boolean`):

Also return token-budgeted chunks of each page's Markdown, in order, each with the URL, heading path and character offsets. Free.

## `chunkTokens` (type: `integer`):

Approximate tokens per chunk (1 token ~ 4 characters).

## `chunkOverlapTokens` (type: `integer`):

Repeated from the previous chunk; at most half the chunk size.

## Actor input object example

```json
{
  "urls": [
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ],
  "mode": "list",
  "maxPages": 100,
  "maxDepth": 3,
  "useSitemaps": true,
  "followLinks": true,
  "includePaths": [],
  "excludePaths": [],
  "removeRepeatedBlocks": true,
  "maxRunMinutes": 30,
  "mainContentOnly": true,
  "includeTables": true,
  "maxChars": 200000,
  "maxPageMb": 5,
  "incremental": false,
  "stateStoreName": "",
  "chunk": false,
  "chunkTokens": 512,
  "chunkOverlapTokens": 0
}
```

# Actor output Schema

## `results` (type: `string`):

One row per result (see the dataset schema for fields).

## `summary` (type: `string`):

SUMMARY record: counts, warnings, stop reason and what was charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.rfc-editor.org/rfc/rfc9309.html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("datafeed/web-page-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://www.rfc-editor.org/rfc/rfc9309.html"] }

# Run the Actor and wait for it to finish
run = client.actor("datafeed/web-page-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ]
}' |
apify call datafeed/web-page-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datafeed/web-page-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TFN3HvbG5zmMU7pJA/builds/U9XBvEsl8ZxUFeBVh/openapi.json