Website & Page to Markdown for RAG and LLMs

What does Website & Page to Markdown for RAG and LLMs do?

Page to Markdown converts web pages to clean Markdown for RAG, LLM prompts and AI agents. Give it page URLs or a whole docs site; you get each page's main content as Markdown, without menus, footers or cookie banners, plus its title, language and links.

Headings, lists, tables, code blocks (with their language) and links are kept. It can follow same-site links (up to 2 clicks deep), convert a whole site or section, sitemap first, return only the pages that changed since your last run, and split each page into chunks by heading for your vector store.

Try it in one click: the input comes pre-filled with two pages of Apify's own documentation (the Storage overview and the Key-value store page). That's 2 pages: $0.002, plus the $0.00005 start fee, so about $0.0021 in all. Then replace them with the pages you actually want.

Monitor a docs site: get only new and changed pages, in Slack, email or a webhook

With Only changed pages on, each run returns (and charges for) only the pages that are new or changed since the last run of the same input, so your RAG index stays current without re-embedding everything. A run where nothing changed returns 0 pages and costs only the $0.00005 start fee.

1. Put the docs root in Web page URLs (for example https://docs.example.com), turn on Whole site (sitemap first) ("crawlWholeSite": true) and Only changed pages ("onlyChangedPages": true), set Max pages per site (maxPagesPerSite) to cover the site, and click Start. This first run returns every page and is the baseline.
2. Click Save as a new task (top right of the actor page). The memory is kept separately for each URL and set of options, so the task compares with its own last run; changing the options starts a new baseline.
3. In Apify Console, open Schedules, click Create new, set how often in Schedule setup (for example weekly, Monday 06:00), then Add your task.
4. On the task, open the Integrations tab and pick where the changes go:
   • HTTP webhook (for a vector store): event ACTOR.RUN.SUCCEEDED, your URL. Apify POSTs {"eventType": ..., "resource": {...}}; resource.defaultDatasetId is the run's dataset, and GET https://api.apify.com/v2/datasets/<defaultDatasetId>/items?format=json (with your API token) returns the changed pages. Upsert them by id, which stays the same for a page from run to run.
   • Slack: click Configure, sign in, pick the workspace and channel, and the "run succeeded" event. A useful message: {{resource.statusMessage}} (the example below's repeat run said "0 pages converted from 2/2 URLs; 2 unchanged since the last run") and a link, <https://console.apify.com/storage/datasets/{{resource.defaultDatasetId}}|changed pages>.
   • Gmail: click Connect with Google, set the subject and body, and attach the dataset. It sends after each successful run.

Apify's integrations fire after every successful run, including quiet ones: a quiet run's dataset is empty, and its status message says how many pages were unchanged.

What data does Page to Markdown return?

Field	Example	Notes
id	96dc7de0c0bcc49ea35fac19	From the page URL; the same on every run. Use it as your vector-store key.
url	https://docs.apify.com/storage/key-value-store	After redirects.
title	json — JSON encoder and decoder
description	Source code: Lib/json/__init__.py ...	Meta description; null if none.
language	en	From the page; null if not declared.
markdown	# Key-value store\n\nThe key-value store is ...	The main content, up to 500,000 characters.
wordCount	1899
links	[{"url": "https://json.org", "text": "JSON ..."}]	Links inside the content, absolute, each once.
canonicalUrl	https://docs.python.org/3/library/json.html	null if none.
discoveredVia, lastmod	input, null	How the page was found (input, link or sitemap); the sitemap's date, if any.
contentHash	7a6b543f5ef8fa31	Same text, same hash on every run.
chunks, chunkCount	[{"headingPath": ["Storage", "Datasets"], ...}]	When chunking is on.

One row per converted page. The full list is under Output.

How much does it cost to convert web pages to Markdown?

You pay per page converted: $1.00 per 1,000 pages, plus $0.00005 each time a run starts.

• The example below: 2 pages × $0.001 = $0.002, plus the $0.00005 start fee; the repeat run $0.00005.
• A month, for example: a 500-page docs site in whole-site mode (Max pages per site 500), weekly, only changed pages: the first run is 500 × $0.001 = $0.50; if 25 pages change a week, the next 3 runs are 75 × $0.001 = $0.075; about $0.58 in all.
• Caps: Max pages per run in the input, and Maximum cost per run in the run options. The run stops cleanly at whichever comes first. Each page is counted against the limit before it's fetched (and the count given back if the page fails), so a capped run never fetches pages it can't return.

Never charged: pages that fail, need JavaScript, are disallowed by robots.txt or opted out of AI use, a page returned twice (each page is converted once per run, even when several of your URLs link to it), and, with "Only changed pages", pages that haven't changed since your last run.

Chunks don't cost extra. Splitting into chunks is included in the page price, whichever chunk output you pick: with one row per chunk, the chunk rows go to a separate dataset of the run (chunks), which isn't charged; the default dataset still has one (charged) row per page.

How to convert a web page or a whole site to Markdown

1. Open Page to Markdown and click Try for free (or Start if you're signed in).
2. Put your pages in Web page URLs, one per line (a missing https:// is added for you).
3. Optional: set Follow links (crawl depth), or turn on Whole site (sitemap first) for a site or section; add Only pages matching / Skip pages matching patterns; turn on Split into chunks for RAG.
4. Click Start, then open the Output tab and export as JSON, CSV or Excel.

Example: two documentation pages, then a re-check

The pre-filled input, with Only changed pages turned on:

{"urls": ["https://docs.apify.com/storage", "https://docs.apify.com/storage/key-value-store"],
 "onlyChangedPages": true}

One of the two pages (real output from a local run on 2026-09-25; markdown shortened, links left out):

{
  "id": "96dc7de0c0bcc49ea35fac19",
  "url": "https://docs.apify.com/storage/key-value-store",
  "title": "Key-value store | Platform | Apify Documentation",
  "language": "en",
  "markdown": "# Key-value store\n\nThe key-value store is simple storage that can be used for storing any kind of data. ...",
  "wordCount": 1899,
  "discoveredVia": "input",
  "contentHash": "7a6b543f5ef8fa31"
}

Run again straight after, the same input returned 0 pages: neither page had changed, so only the start fee was charged. For a whole site, use Whole site (sitemap first) with the site's docs root instead of single pages.

Input

Field	What it does
Web page URLs (urls)	The pages to convert, one per line. A missing https:// is added for you.
Follow links (crawl depth) (crawlDepth)	0 (default): only the pages you list. 1: also the pages they link to on the same site. 2: one more click further. Other sites and file links (PDF, images, ZIP, ...) are never followed. www. counts as the same site.
Max pages per URL when following links (maxPagesPerUrl)	At most this many pages are tried for each URL you list, including itself (default 10).
Whole site (sitemap first) (crawlWholeSite)	Convert every page of each site you list (see below). Crawl depth is ignored then.
Max pages per site (maxPagesPerSite)	In whole-site mode, at most this many pages are returned per URL you list (default 100, up to 10,000).
Only pages matching / Skip pages matching (includeUrlPatterns / excludeUrlPatterns)	URL patterns for pages found by crawling (either mode). A pattern without * matches anywhere in the URL (/docs/); with * it must match the whole URL (https://example.com/blog/*). Case-insensitive. Pages that don't pass are never fetched; the URLs you list are always converted.
Only changed pages (onlyChangedPages)	Return only pages that are new or changed since the last run of the same input (see below).
Include images (includeImages)	Keep images as . Off by default.
Split into chunks for RAG (chunkMarkdown)	Also split each page's Markdown into chunks by heading (see below).
Max chunk size (characters) (maxChunkChars)	No chunk is longer than this (default 2,000).
Chunk output (chunkOutput)	One row per page with a chunks array (default), or one row per chunk in the run's chunks dataset.
Max pages per run (maxResults)	Cap the total number of pages across all URLs.

{
  "urls": ["https://docs.python.org/3/tutorial/index.html"],
  "crawlDepth": 1,
  "maxPagesPerUrl": 25
}

Whole-site mode

Each line is a site (https://docs.example.com), a section of one (https://docs.example.com/guide: only pages under /guide), or a sitemap URL (read as given). Pages are found sitemap first: the Sitemap: lines of the site's robots.txt, else /sitemap.xml or /sitemap_index.xml, with sitemap indexes followed. Pages on other sites and links to files are skipped. If the site has no usable sitemap, or it lists nothing in your section, the actor follows the site's own links instead, starting from your URL, with no depth limit (if your URL redirects, the section is taken from where it lands, e.g. /platform/storage → /storage). RUN_STATS says which way each site was read (discoveredVia) and why.

The page limit counts pages returned. To keep a run bounded when most pages turn out unchanged or broken, at most 5 pages are fetched per page returned (plus 100).

{
  "urls": ["https://docs.apify.com/platform/storage"],
  "crawlWholeSite": true,
  "maxPagesPerSite": 500,
  "excludeUrlPatterns": ["/changelog"],
  "onlyChangedPages": true
}

Only changed pages

The first run returns every page and remembers each one's sitemap lastmod date and a hash of its Markdown. Later runs of the same input skip a page when its sitemap lastmod is unchanged (without fetching it), or, for pages without a date, when its Markdown is identical to last time (it's fetched to compare, but not returned or charged). The memory is kept in a key-value store named page-to-markdown-changes in your own Apify account, separately for each URL and set of options, so two different searches never hide each other's pages. Only pages that were actually delivered are remembered: a page cut by your max cost per run comes back next time. Delete that store to start over.

Chunks for RAG

Each page's Markdown is split at its headings: one chunk per section, and a section longer than the max chunk size is split at paragraph breaks, then line breaks (code blocks keep their lines). A heading with no text of its own (a page title right before its first subsection) doesn't become a chunk; it appears in the heading path of the sections under it. Each chunk has:

• headingPath: the headings it sits under, outermost first, e.g. ["Storage", "Datasets", "Retention"]
• chunkIndex (from 0) and charCount, plus, in one-row-per-chunk output, id, pageId, url, startUrl, title, language, chunkCount and scrapedAt.

With one row per page (the default), the chunks are a chunks array on the page's row. With one row per chunk, the page rows stay in the default dataset and each chunk is its own row in the run's second dataset, chunks (the run's status message and RUN_STATS.chunksDataset give its id). A chunk's id is its page's id plus its index, so it's stable while the page doesn't change.

Output

One item per converted page. Fields a page doesn't have are null.

{
  "id": "c9c50a504fb81ab27af5a4d0",
  "startUrl": "https://docs.python.org/3/library/json.html",
  "url": "https://docs.python.org/3/library/json.html",
  "canonicalUrl": "https://docs.python.org/3/library/json.html",
  "depth": 0,
  "title": "json — JSON encoder and decoder",
  "description": "Source code: Lib/json/__init__.py JSON (JavaScript Object Notation), specified by RFC 7159 ...",
  "language": "en",
  "markdown": "# `json` — JSON encoder and decoder\n\n---\n\n[JSON (JavaScript Object Notation)](https://json.org), specified by ...",
  "markdownTruncated": false,
  "wordCount": 3757,
  "links": [{"url": "https://json.org", "text": "JSON (JavaScript Object Notation)"}],
  "discoveredVia": "input",
  "lastmod": null,
  "contentHash": "5e0f3c1d2a9b8e7f",
  "chunkCount": null,
  "chunks": null,
  "scrapedAt": "2026-09-24T12:00:00Z"
}

• id is derived from the page URL and is the same on every run: use it to deduplicate or as your vector-store key.
• url is the address after redirects; startUrl is the URL you listed that led to it; depth is how many links away from it the page is (null for pages found in a sitemap); discoveredVia is input, link or sitemap, and lastmod the date the sitemap gives, if any.
• contentHash is a hash of the Markdown: the same text gives the same hash on every run.
• chunkCount and chunks are filled when chunking is on (see above).
• links lists the links inside the extracted content, made absolute, each once.
• markdown is capped at 500,000 characters (markdownTruncated says when that happened). Pages larger than 5 MB of HTML are skipped.
• When the content has no heading of its own, the page title is added as the first line (# Title), so every chunk knows what document it came from.

Run it on a schedule, or from your own code

1. Save your input as a task and add it to a schedule (Console → Schedules), as in Monitor a docs site above.
2. Collect results: download the dataset as JSON, CSV or Excel; fetch the latest run's results from the API (GET https://api.apify.com/v2/actor-tasks/<task id>/runs/last/dataset/items?status=SUCCEEDED&format=csv, with your API token); let a webhook tell your system when a run succeeds; or connect it to Make, Zapier or n8n through Apify's integrations.

Each page's id stays the same from run to run: use it as the key in your vector store, so a changed page replaces its old version instead of being added twice.

Can I use Page to Markdown from an AI agent (MCP)?

Yes, through Apify's MCP server: add https://mcp.apify.com?tools=humble-echidna/page-to-markdown to your MCP client (or let the agent find it with the server's actor search). The agent passes the pages it wants to read, e.g. {"urls": ["https://docs.example.com/guide/install"]}, and gets each page's Markdown back; set maxResults to keep the answer small.

Who it's for

Teams that keep an LLM, RAG index or AI assistant supplied with documentation they're allowed to use (their own product docs, a vendor's public docs, an internal knowledge base on a public site). The recurring job: convert the docs site once, then re-run on a schedule and get only the pages that changed.

Why this one?

• Clean output, not the whole page. A readability-style extractor keeps the article and drops the site chrome around it, so you don't pay tokens (or embedding space) for menus and footers.
• Honest about what it can't do. Pages that only render with JavaScript (single-page apps) are reported as "needs JavaScript" and not charged, instead of being returned as an empty or boilerplate-only result. The same goes for PDFs and other files, missing pages, and sites that refuse the request.
• Polite by design. It checks each site's robots.txt before every request and honours its Crawl-delay (see below).
• Fast and cheap. Plain HTTP, no browser, so there's no page rendering to wait for and no browser to pay for. Most of a run's time is the sites' own response time and the politeness limits below.
• Whole sites, sitemap first. In whole-site mode it reads the site's sitemap (the same way search engines find it), so it gets every listed page without guessing, and falls back to following links when there is no sitemap.
• Only what changed. Re-run it on a schedule with "Only changed pages" and you get (and pay for) only new and changed pages: sitemap lastmod dates skip unchanged pages without even fetching them, and a content hash catches the rest.
• Chunks ready for your vector store, split by heading with a size cap, each with its heading path. Chunks are free: you pay per page.
• Reliable. One failing URL never affects the others in your run. The run log and the RUN_STATS record say exactly which URL had a problem and why.

robots.txt and politeness

• Before fetching any page, including every redirect hop and every page found by following links, it checks that site's robots.txt and skips pages it disallows. robots.txt is read once per origin (https:// + host) per run, not once per page. Skipped pages are listed in the log and in RUN_STATS (blockedByRobots), and are never charged.
• It identifies itself honestly, with the user agent HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna). Site owners can allow or block it by name: robots.txt rules written for User-agent: HumbleEchidnaApify apply to it (otherwise the * rules do).
• It also honours opt-outs from AI use. Its output is meant for LLMs, so when a site's robots.txt disallows a page for any of the well-known AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, CCBot, ClaudeBot, Claude-User, anthropic-ai, Claude-Web, Google-Extended, PerplexityBot, Applebot-Extended, Bytespider, cohere-ai, Meta-ExternalAgent, Amazonbot, Diffbot, Omgilibot), the page isn't fetched. It's reported as "site opts out of AI use" (optedOutOfAI in RUN_STATS) and not charged.
• If a site's robots.txt can't be read because the server errors, the site is treated as disallowing everything, as the robots.txt standard (RFC 9309) requires. A site without a robots.txt allows everything.
• It honours Crawl-delay (retries included) and Retry-After, and it never makes more than 2 requests at a time to one site. A site asking for more than 30 seconds between requests, or before a retry, is reported as such rather than fetched slowly.
• It doesn't log in, and doesn't try to get around paywalls, logins or bot protection: a site that answers "403 Forbidden" is reported as refusing the request.

Limits

• No browser: pages whose content is built by JavaScript are reported as "needs JavaScript" and not charged.
• HTML pages only: PDFs and other files are reported as "not an HTML page" and not charged.
• markdown is capped at 500,000 characters per page; pages larger than 5 MB of HTML are skipped.
• Only public web pages on the standard ports (80 and 443); no logins, no proxies.
• Main-content detection is heuristic: it works best on articles, documentation and blog posts.

FAQ

Why did a page come back as "needs JavaScript"?

Its content is built in the browser by JavaScript, and this version doesn't run a browser. The page is not charged. Many such sites also publish the same content somewhere static (a docs site, a blog, an RSS feed) that works.

Why did a page come back as "blocked by robots.txt"?

The site's owner has asked crawlers not to fetch it. This actor respects that, and the page is not charged.

Why did a page come back as "site opts out of AI use"?

The site's robots.txt disallows that page for AI crawlers (see the list above). Converting it for an LLM is exactly that use, so this actor skips it, and it's not charged.

Does whole-site mode respect AI opt-outs too?

Yes. The sitemaps are fetched with the same checks as the pages: robots.txt, and its opt-outs for AI crawlers. A page the site opts out of AI use for is never fetched, even when its sitemap lists it.

Can I get only new and changed pages since my last run?

Yes: turn on Only changed pages. See Monitor a docs site and Only changed pages above.

Why does it refuse localhost, 10.x.x.x, an internal hostname or a URL with a port?

It only fetches public web pages, on the standard web ports (80 for http://, 443 for https://); a URL with any other port, such as :8080, is refused. Every hostname (and every redirect) is resolved first, and a page is refused if any address it resolves to is private, loopback, link-local (cloud metadata) or otherwise not on the public internet; the connection then goes to the address that was checked. These refusals are reported as "not a public web address", not as robots.txt blocks, and aren't charged. A domain that doesn't exist is reported as such.

Can it convert PDFs to Markdown?

Not in this version: PDFs and other files are reported as "not an HTML page" and not charged. Use PDF, DOCX & XLSX to Text for those.

The output contains a bit of menu or misses a section. What now?

Main-content detection is heuristic and works best on articles, documentation and blog posts. Please open an issue with the URL.

Something that used to work now fails. Why?

Sites change their markup without notice. The run log names the URL and what went wrong, and every other URL in the run is unaffected. Please open an issue with the input you used.

Is it legal to convert web pages to Markdown?

It fetches only the pages you give it (and, if you ask, same-site pages they link to or the site's sitemap lists), as a normal logged-out visitor, and follows each site's robots.txt, including its opt-outs for AI crawlers. It doesn't log in or get around any protection. What you do with the content (copyright, the site's terms) is your responsibility, as with any web page you save.

Related actors

Actor	Use it when
Article Extractor: Text, Author, Date	You want news and blog articles: it finds the newest posts through a site's feed or sitemap and returns author and publication date. Pick Page to Markdown for documentation and other pages, whole sites, and heading-based chunks.
PDF, DOCX & XLSX to Text	The docs site links to PDFs, Word or Excel files (this actor reports those as "not an HTML page" and doesn't charge for them).
Sitemap URL Extractor	You want to see a site's full URL list, with lastmod dates, before choosing URL patterns.

Feedback and support

Found a bug, or need a field or option that isn't here? Open an issue on the Issues tab with the input you used.

Versions

Current version: 1.1. See the Changelog tab for what changed in each version.