Web Page to Markdown for LLMs (polite crawler, robots.txt-respecting)

Turn public web pages, or a whole documentation site, into clean Markdown for LLMs and RAG: headings, lists, links, code blocks and tables kept; navigation, cookie banners, share bars, newsletter boxes, related-post lists and comments removed. Optional token-budgeted chunks with the source URL, heading path and character offsets, ready for citations.

It is built to be a well-behaved bot: it reads each site's robots.txt before fetching anything (sitemaps included), re-checks on every redirect, sends an identifiable User-Agent, waits between requests to the same site and honours Crawl-delay, and skips pages that opt out of AI use with noai. You pay a flat price per page, never for compute, and nothing for pages that fail.

Why this one

• Cleaner main content, biased towards keeping text. When in doubt it keeps a block: losing content is worse than leaking a menu. On the 25 hand-written test pages in BENCHMARK.md (benchmark pages only, not a guarantee for every site) it found every expected content block. It leaked 21% of the boilerplate strings on the 8 pages written after the rules were frozen; v0.1 leaked 39% and lost 10% of the content. Every rule is listed below.
• Never loops, never times out silently. Hard maxPages and maxRunMinutes budgets, a stop before the platform run timeout, and each request's deadline shrunk to fit the time left. URLs are de-duplicated after normalisation, and crawler traps (repeating paths, endless query strings, binary files) are refused. Every run ends with a SUMMARY that says why it stopped.
• Predictable price. Pay per page (page-converted). In incremental mode an unchanged page costs only page-unchanged. Failed and skipped pages are free. The charge is taken in the same call that stores the item.
• Chunks that keep order. Chunks follow the page in order. Every heading starts a new chunk, and long sections are split at paragraphs, then sentences. Code fences stay whole while they fit. Overlap never reaches back across the chunk's heading, so headingPath is right for the whole chunk. Each chunk is an exact slice markdown[charStart:charEnd].

What it does and does not do

• Fetches plain HTTP(S) pages with a normal GET request. No browser, no JavaScript, no logins, no cookies, no proxies, no CAPTCHA solving, no search-engine results. Pages that need JavaScript to show their text come back as empty_content errors (free).
• Respects robots.txt per RFC 9309 for the user-agent token DataFeedBot (falls back to the * group). A site whose robots.txt answers with a server error is treated as "disallow everything", as the RFC says. Honours <meta name="robots" content="nofollow"> and rel="nofollow" when crawling.
• Skips pages marked <meta name="robots" content="noai"> or X-Robots-Tag: noai.
• Refuses URLs pointing at private or internal network addresses (DNS is resolved once and the connection is pinned to the checked address).
• You are responsible for having the right to use the content you convert.

Modes

list (default) converts exactly the URLs you give (up to 500).

crawl starts from up to 20 start URLs and works in this order:

1. It fetches the start URL.
2. It reads the site's sitemaps: the ones robots.txt names, otherwise /sitemap.xml. It reads at most 20 sitemap files, and only from the same site.
3. It follows links on the crawled pages, breadth-first, up to maxDepth clicks.

The crawl stays on the start URL's host and inside its directory (https://x.com/docs/intro stays under /docs/), unless you set includePaths. Links that robots.txt forbids are never fetched and never appear in the dataset. They are counted in SUMMARY.rejectedUrls. If a site's Crawl-delay is longer than 10 s, the Actor fetches one page from it and drops the rest of that site, with a note in the warnings.

Extraction rules (main-content mode)

1. Root: <main>, or the single top-level <article> (widgets rendered as <article> are ignored), accepted only if it holds the page's <h1> or at least 30% of the page's prose. Several articles: the one holding 60%+ of article text, otherwise all of them. Then role=main. Otherwise a readability-style density score, widened over neighbouring prose (not menus) and to include the page's <h1>.
2. Always removed: scripts, styles, forms with little text, buttons, hidden / aria-hidden / display:none / visibility:hidden elements. Exception: hidden tab panels (role=tabpanel, e.g. code in other languages) are kept and labelled with their tab name. A <form> that wraps the whole page (ASP.NET) is kept.
3. Page chrome: <nav>, <footer>, <header>, <aside>, and ARIA roles navigation, banner, contentinfo, complementary, search, dialog, menu, toolbar. Kept anyway: an article's own <header> with its title, an <aside> that is a note/warning or holds real prose, and a role-marked section with its own heading and prose.
4. Boilerplate class and id names: a curated list matched against each whole class name, for example cookie-banner, share-buttons, related-posts, newsletter, ad-slot, sidebar, comments, breadcrumb, toc, sr-only. A word inside a longer name never counts (social-security, related-work, menu-planning and ad-hoc stay). A block is never dropped by this rule if it holds the <h1>, more than half of the content, or 200+ characters of prose.
5. Cookie and consent banners by text: a short block mentioning cookies plus "accept all", "we use cookies", "consent", "manage preferences" and the like.
6. Link farms: a div/section with 4+ links that is 70%+ link text and small compared with the page.
7. "Related posts", "Read next", "More from ..." headings, only when followed by a list of real links (not in-page anchors). The heading and that list are dropped; a heading followed by prose is kept.
8. Heading permalink anchors (¶, #, hash-link) and "Edit this page" links.
9. Kept intact: tables (GFM tables plus structured tables; colspan and rowspan expanded; code in a cell becomes inline code), code blocks (language read from language-*, highlight-* or data-lang; a real line-number gutter table becomes one code block), lists (nested and ordered), definition lists (code inside <dd> stays fenced), blockquotes, images with alt text.
10. Crawl mode only (removeRepeatedBlocks): a short paragraph that already appeared on 3 earlier pages of the same site is dropped. Headings, code and tables are never dropped by this rule.

Input

{
  "urls": ["https://docs.example.com/guide/"],
  "mode": "crawl",
  "maxPages": 200,
  "maxDepth": 3,
  "maxRunMinutes": 30,
  "incremental": true,
    "chunk": true,
  "chunkTokens": 512
}

All fields except urls are optional. The full list, with defaults, is in the input schema: mainContentOnly, includeTables, maxChars, maxPageMb, useSitemaps, followLinks, includePaths, excludePaths, removeRepeatedBlocks, chunkOverlapTokens.

Output (one dataset item per page)

{
  "url": "https://docs.example.com/guide/limits",
  "finalUrl": "https://docs.example.com/guide/limits",
  "canonicalUrl": "https://docs.example.com/guide/limits",
  "status": "ok",
  "change": "changed",
  "depth": 1,
  "source": "sitemap",
  "title": "Rate limits | Example Docs",
  "description": "How many requests you may send",
  "language": "en",
  "markdown": "# Rate limits\n\nEvery API key may send 600 requests per minute...",
  "truncated": false,
  "wordCount": 412,
  "approxTokens": 690,
  "contentType": "text/html",
  "boilerplateRemoved": 7,
  "contentHash": "sha256:5f1c...",
  "tables": [{"index": 0, "caption": null, "headers": ["Header", "Description"], "headerRows": [["Header", "Description"]], "rows": [["Retry-After", "Seconds to wait after a 429"]]}],
  "chunks": [{"index": 0, "chunkId": "a1b2c3d4e5f60718", "url": "https://docs.example.com/guide/limits", "headingPath": ["Rate limits"], "charStart": 0, "charEnd": 1630, "approxTokens": 408, "text": "# Rate limits\n\n..."}],
  "fetchedAt": "2026-10-14T00:00:00+00:00",
  "error": null
}

• status can be:
  • ok: converted.
  • unchanged: incremental mode only. The item has no Markdown and costs only page-unchanged.
  • skipped: robots.txt, a noai opt-out, or a Crawl-delay that is too long.
  • error: an HTTP error, a page that is too large, not HTML, or empty.
• Skipped and error items explain why in error.code / error.message and are not charged.
• change appears in incremental mode only: new, changed or unchanged.
• depth and source (seed / sitemap / link / state) appear in crawl mode only.
• approxTokens is characters / 4, an estimate rather than a real tokenizer count. Chunk budgets use the same estimate.
• contentHash is taken from the extracted Markdown, before the repeated-block step. A rotating ad or a timestamp in the page chrome therefore does not count as a change.

The SUMMARY record in the run's key-value store contains:

• the counts per status
• stopReason: null, page_budget, time_budget, charge_limit or request_budget
• notFetched: pages discovered but not fetched
• discovered and rejectedUrls, with the reason for each rejected URL
• the incremental state
• any warnings

Run-summary row (free). When a run produces no other row (for example a monitor run where nothing changed, or a run stopped at once by your maximum charge or the run timeout), the Actor adds one row {"type": "run_summary", "status": "run_summary", "message": "...", "stopReason": ..., "warnings": [...], "summary": {...}} (summary is the same as the SUMMARY record), so the dataset is never empty. It is never charged. Skip it with type != "run_summary".

Incremental recrawls

With incremental: true, each page's ETag, Last-Modified and content hash are saved in a named key-value store. By default the name is derived from the mode, start URLs and paths (wpm-state-...), so different jobs never share state; set stateStoreName to choose it. On the next run, the Actor sends If-None-Match / If-Modified-Since. A page is reported unchanged when the server answers 304, or when it sends the page again and the extracted Markdown (hashed in full, before maxChars truncation) is the same. In crawl mode, newly discovered pages are fetched first; pages known from earlier runs (source: "state") are re-checked with the budget that is left, so re-checks can never stop new pages from being found. Pages that now answer 404/410 or are disallowed by robots.txt are dropped from the state (counted in SUMMARY.incremental.removedPages for 404/410). A 304 page that now sends X-Robots-Tag: noai is skipped and not charged. If the extraction settings (mainContentOnly, includeTables, maxChars, Actor version) changed since the state was written, all pages are converted again and a warning says so. If the store cannot be read or written, the run does not fail: every page is treated as new and a warning says so.

Restarts. The run records which pages it has already stored in its own key-value store (PROGRESS). If the platform migrates or restarts the run, those pages are not fetched or charged again (SUMMARY.resumedSkipped).

Pricing

Pay per event:

Event	When	Suggested price
page-converted	A page with status: ok was stored (chunks included, free)	$0.0015
page-unchanged	Incremental mode: a page checked and found unchanged was stored	$0.0002

Nothing else is charged by this Actor: no compute surcharge, no start fee of its own, no charge for skipped or failed pages, robots.txt or sitemaps. Each charge is taken in the same call that stores the item, so you are never charged for an item that was not stored. Before each page, the Actor checks that a page-converted still fits your maximum charge per run. If it does not, the Actor stops and SUMMARY says so.

Using it from an AI agent

The input is a plain URL list and the output is one JSON object per page, so the Actor works as a tool via the Apify API or Apify's MCP server. Turn on chunk to get embedding-ready chunks with citation offsets directly. You can also feed markdown into RAG Text Chunker for other chunking options.

Limits

• List mode: 500 URLs per run. Crawl mode: 20 start URLs and 1,000 pages per run.
• 10 MB per page.
• 2,000 HTTP requests per run (robots.txt and sitemaps included). Sitemaps: 20 files per site.
• 5 redirects.
• At least 1 s between two requests to one site, and at most 10 s of waiting for Crawl-delay.
• Tables: colspan and rowspan are expanded. Several header rows are merged per column ("Price / Min") and also returned raw in headerRows.
• No JavaScript rendering. Single-page apps whose HTML has no text come back as empty_content (free).