Website & Page to Markdown for RAG and LLMs

What does Website & Page to Markdown for RAG and LLMs do?

Page to Markdown converts web pages to clean Markdown for RAG, LLM prompts and AI agents. Give it page URLs, a whole docs site, or another actor's results (chained); you get each page's main content as Markdown, without menus, footers or cookie banners, plus its title, language and links.

Headings, lists, tables, code blocks (with their language) and links are kept. It can follow same-site links (up to 2 clicks deep), convert a whole site or section, sitemap first, return only the pages that changed since your last run, and split each page into chunks by heading for your vector store. The usual job: turn a documentation site into RAG-ready chunks and keep them current (how).

Input fields · API · Use it from Claude, ChatGPT or Cursor (MCP)

Jump to: Monitor a site · Fields · Price · How to use · RAG chunks · Input · Output · Chain after another actor · AI agents (MCP) · Limits · FAQ

Try it in one click: the input comes pre-filled with two pages of Apify's own documentation (the Storage overview and the Key-value store page). That's 2 pages: $0.002, plus the $0.00005 start fee, so about $0.0021 in all. Then replace them with the pages you actually want.

Monitor a docs site: get only new and changed pages, in Slack, email or a webhook

With Only changed pages on, each run returns (and charges for) only the pages that are new or changed since the last run of the same input, so your RAG index stays current without re-embedding everything. A run where nothing changed returns 0 pages and costs only the $0.00005 start fee.

1. Put the docs root in Web page URLs (for example https://docs.example.com), turn on Whole site (sitemap first) ("crawlWholeSite": true) and Only changed pages ("onlyChangedPages": true), set Max pages per site (maxPagesPerSite) to cover the site, and click Start. This first run returns every page and is the baseline.
2. Click Save as a new task (top right of the actor page). The memory is kept separately for each URL and set of options, so the task compares with its own last run; changing the options starts a new baseline.
3. In Apify Console, open Schedules, click Create new, set how often in Schedule setup (for example weekly, Monday 06:00), then Add your task.
4. On the task, open the Integrations tab and pick where the changes go:
   • HTTP webhook (for a vector store): event ACTOR.RUN.SUCCEEDED, your URL. Apify POSTs {"eventType": ..., "resource": {...}}; resource.defaultDatasetId is the run's dataset, and GET https://api.apify.com/v2/datasets/<defaultDatasetId>/items?format=json (with your API token) returns the changed pages. Upsert them by id, which stays the same for a page from run to run.
   • Slack: click Configure, sign in, pick the workspace and channel, and the "run succeeded" event. A useful message: {{resource.statusMessage}} (the example below's repeat run said "0 pages converted from 2/2 URLs; 2 unchanged since the last run") and a link, <https://console.apify.com/storage/datasets/{{resource.defaultDatasetId}}|changed pages>.
   • Gmail: click Connect with Google, set the subject and body, and attach the dataset. It sends after each successful run.

Apify's integrations fire after every successful run, including quiet ones: a quiet run's dataset is empty, and its status message says how many pages were unchanged.

What data does Page to Markdown return?

One row per converted page. The full list is under Output.

How much does it cost to convert web pages to Markdown?

You pay per page converted: $1.00 per 1,000 pages, plus $0.00005 each time a run starts.

It's cheaper on paid Apify plans: $0.90 per 1,000 pages on Starter, $0.80 on Scale and $0.70 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show.

• The example below: 2 pages × $0.001 = $0.002, plus the $0.00005 start fee; the repeat run $0.00005.
• A month, for example: a 500-page docs site in whole-site mode (Max pages per site 500), weekly, only changed pages: the first run is 500 × $0.001 = $0.50; if 25 pages change a week, the next 3 runs are 75 × $0.001 = $0.075; about $0.58 in all.
• Caps: Max pages per run in the input, and Maximum cost per run in the run options. The run stops cleanly at whichever comes first. Each page is counted against the limit before it's fetched (and the count given back if the page fails), so a capped run never fetches pages it can't return.

Never charged: pages that fail, need JavaScript, are disallowed by robots.txt or opted out of AI use, a page returned twice (each page is converted once per run, even when several of your URLs link to it), and, with "Only changed pages", pages that haven't changed since your last run.

Chunks don't cost extra. Splitting into chunks is included in the page price, whichever chunk output you pick: with one row per chunk, the chunk rows go to a separate dataset of the run (chunks), which isn't charged; the default dataset still has one (charged) row per page.

How to convert a web page or a whole site to Markdown

1. Open Page to Markdown and click Try for free (or Start if you're signed in).
2. Put your pages in Web page URLs, one per line (a missing https:// is added for you).
3. Optional: set Follow links (crawl depth), or turn on Whole site (sitemap first) for a site or section; add Only pages matching / Skip pages matching patterns; turn on Split into chunks for RAG.
4. Click Start, then open the Output tab and export as JSON, CSV or Excel.

Turn a documentation site into RAG chunks

Documentation sites are what Page to Markdown handles best: they almost always have a sitemap, render their text without JavaScript, and are organised by headings, which become each chunk's headingPath.

1. Put the docs root (or one section, e.g. https://docs.example.com/guides) in Web page URLs and turn on Whole site (sitemap first). Set Max pages per site above the site's page count.
2. Turn on Split into chunks for RAG, and pick One row per chunk if your loader wants one record per embedding. Skip changelogs, tag pages or old versions with Skip pages matching (e.g. /changelog, /v1/).
3. Load the chunks: embed markdown, store url, title and headingPath as metadata for citations, and use the chunk id (the page id plus the chunk index) as the vector key.
4. Keep it current: save the input as a task with Only changed pages on and schedule it (details); re-embed only the pages that come back.

{
  "urls": ["https://docs.apify.com/platform"],
  "crawlWholeSite": true,
  "maxPagesPerSite": 1000,
  "excludeUrlPatterns": ["/changelog"],
  "chunkMarkdown": true,
  "chunkOutput": "chunk",
  "onlyChangedPages": true
}

What it costs: you pay per page, chunks included. A 300-page docs site is 300 × $0.001 = $0.30 on the Free plan (less on paid plans); in our test run on Apify's own docs, those 300 pages took under a minute. How many chunks a page makes depends on the max chunk size (default 2,000 characters; bigger chunks make fewer).

Example: two documentation pages, then a re-check

The pre-filled input, with Only changed pages turned on:

{"urls": ["https://docs.apify.com/storage", "https://docs.apify.com/storage/key-value-store"],
 "onlyChangedPages": true}

One of the two pages (real output from a local run on 2026-09-25; markdown shortened, links left out):

{
  "id": "96dc7de0c0bcc49ea35fac19",
  "url": "https://docs.apify.com/storage/key-value-store",
  "title": "Key-value store | Platform | Apify Documentation",
  "language": "en",
  "markdown": "# Key-value store\n\nThe key-value store is simple storage that can be used for storing any kind of data. ...",
  "wordCount": 1899,
  "discoveredVia": "input",
  "contentHash": "7a6b543f5ef8fa31"
}

Run again straight after, the same input returned 0 pages: neither page had changed, so only the start fee was charged. For a whole site, use Whole site (sitemap first) with the site's docs root instead of single pages.

Ready-to-run examples

Each example opens with the input already filled in. Run it as it is, or change the input first.

• Convert a docs site to Markdown chunks for RAG: Turn a whole documentation site into clean Markdown split into heading-based chunks, ready for a vector store.
• Monitor a website for new and changed pages as Markdown: Run on a schedule and get only the pages that are new or changed since the last run, as clean Markdown.
• Convert Docusaurus docs to Markdown chunks for RAG: Turn a Docusaurus documentation site into Markdown chunks of about 512 tokens, skipping old doc versions.
• Convert GitBook docs to Markdown for LLMs and RAG: Convert a documentation site hosted on GitBook into clean Markdown, one row per page.

Input

{
  "urls": ["https://docs.python.org/3/tutorial/index.html"],
  "crawlDepth": 1,
  "maxPagesPerUrl": 25
}

Whole-site mode

Each line is a site (https://docs.example.com), a section of one (https://docs.example.com/guide: only pages under /guide), or a sitemap URL (read as given). Pages are found sitemap first: the Sitemap: lines of the site's robots.txt, else /sitemap.xml or /sitemap_index.xml, with sitemap indexes followed. Pages on other sites and links to files are skipped. If the site has no usable sitemap, or it lists nothing in your section, the actor follows the site's own links instead, starting from your URL, with no depth limit (if your URL redirects, the section is taken from where it lands, e.g. /platform/storage → /storage). RUN_STATS says which way each site was read (discoveredVia) and why.

The page limit counts pages returned. To keep a run bounded when most pages turn out unchanged or broken, at most 5 pages are fetched per page returned (plus 100).

{
  "urls": ["https://docs.apify.com/platform/storage"],
  "crawlWholeSite": true,
  "maxPagesPerSite": 500,
  "excludeUrlPatterns": ["/changelog"],
  "onlyChangedPages": true
}

Only changed pages

The first run returns every page and remembers each one's sitemap lastmod date and a hash of its Markdown. Later runs of the same input skip a page when its sitemap lastmod is unchanged (without fetching it), or, for pages without a date, when its Markdown is identical to last time (it's fetched to compare, but not returned or charged). The memory is kept in a key-value store named page-to-markdown-changes in your own Apify account, separately for each URL and set of options, so two different searches never hide each other's pages. Only pages that were actually delivered are remembered: a page cut by your max cost per run comes back next time. Delete that store to start over.

Chunks for RAG

Each page's Markdown is split at its headings: one chunk per section, and a section longer than the max chunk size is split at paragraph breaks, then line breaks (code blocks keep their lines). A heading with no text of its own (a page title right before its first subsection) doesn't become a chunk; it appears in the heading path of the sections under it. Each chunk has:

• headingPath: the headings it sits under, outermost first, e.g. ["Storage", "Datasets", "Retention"]
• chunkIndex (from 0), charCount and approxTokens, plus, in one-row-per-chunk output, id, pageId, url, startUrl, title, language, chunkCount and scrapedAt.

Sizing chunks in tokens. Embedding models limit their input in tokens, not characters. Set Max chunk size (approx. tokens) (maxChunkTokens, e.g. 512) and chunks are sized at about 4 characters per token, the usual rule for English prose; each chunk's approxTokens is its length divided by 4, rounded up. It's an estimate, not your model's tokenizer: code, tables and non-Latin scripts use more tokens per character, so set it 10-20% below your model's limit.

{
  "urls": ["https://docs.apify.com/platform"],
  "crawlWholeSite": true,
  "maxPagesPerSite": 1000,
  "chunkMarkdown": true,
  "maxChunkTokens": 512,
  "chunkOutput": "chunk"
}

With one row per page (the default), the chunks are a chunks array on the page's row. With one row per chunk, the page rows stay in the default dataset and each chunk is its own row in the run's second dataset, chunks (the run's status message and RUN_STATS.chunksDataset give its id). A chunk's id is its page's id plus its index, so it's stable while the page doesn't change.

Output

One item per converted page. Fields a page doesn't have are null.

{
  "id": "c9c50a504fb81ab27af5a4d0",
  "startUrl": "https://docs.python.org/3/library/json.html",
  "url": "https://docs.python.org/3/library/json.html",
  "canonicalUrl": "https://docs.python.org/3/library/json.html",
  "depth": 0,
  "title": "json — JSON encoder and decoder",
  "description": "Source code: Lib/json/__init__.py JSON (JavaScript Object Notation), specified by RFC 7159 ...",
  "language": "en",
  "ogImage": "https://docs.python.org/3.14/_images/social_previews/summary_library_json_0fabf18f.png",
  "ogSiteName": "Python documentation",
  "markdown": "# `json` — JSON encoder and decoder\n\n---\n\n[JSON (JavaScript Object Notation)](https://json.org), specified by ...",
  "markdownTruncated": false,
  "wordCount": 3757,
  "links": [{"url": "https://json.org", "text": "JSON (JavaScript Object Notation)"}],
  "discoveredVia": "input",
  "lastmod": null,
  "contentHash": "5e0f3c1d2a9b8e7f",
  "chunkCount": null,
  "chunks": null,
  "scrapedAt": "2026-09-24T12:00:00Z",
  "sourceTitle": null,
  "sourcePlaceId": null,
  "sourceIndex": null
}

• id is derived from the page URL and is the same on every run: use it to deduplicate or as your vector-store key.
• url is the address after redirects; startUrl is the URL you listed that led to it; depth is how many links away from it the page is (null for pages found in a sitemap); discoveredVia is input, link or sitemap, and lastmod the date the sitemap gives, if any.
• contentHash is a hash of the Markdown: the same text gives the same hash on every run.
• chunkCount and chunks are filled when chunking is on (see above).
• ogImage and ogSiteName come from the page's Open Graph tags (og:image, made absolute, and og:site_name); null when the page doesn't have them.
• sourceTitle, sourcePlaceId and sourceIndex say which item of your dataset the page came from (its title or name, its placeId, its position from 0), on every page found from it; null for URLs you type in.
• links lists the links inside the extracted content, made absolute, each once.
• markdown is capped at 500,000 characters (markdownTruncated says when that happened). Pages larger than 5 MB of HTML are skipped.
• When the content has no heading of its own, the page title is added as the first line (# Title), so every chunk knows what document it came from.

Run it on a schedule, or from your own code

1. Save your input as a task and add it to a schedule (Console → Schedules), as in Monitor a docs site above.
2. Collect results: download the dataset as JSON, CSV or Excel; fetch the latest run's results from the API (GET https://api.apify.com/v2/actor-tasks/<task id>/runs/last/dataset/items?status=SUCCEEDED&format=csv, with your API token); let a webhook tell your system when a run succeeds; or connect it to Make, Zapier or n8n through Apify's integrations.

Each page's id stays the same from run to run: use it as the key in your vector store, so a changed page replaces its old version instead of being added twice.

Chain it after another actor

To convert the pages another actor found as soon as it finishes (a crawler, a sitemap or RSS reader, any list of links; any actor, ours or not), chain it with an integration:

1. Open the other actor or its saved task, go to the Integrations tab and add Apify actor → Page to Markdown, triggered when a run succeeds.
2. Set "datasetId": "{{resource.defaultDatasetId}}" in the input (Apify fills in the finished run's dataset) and the rest as usual, e.g. "chunkMarkdown": true. Web page URLs is ignored then.

Every successful run of the other actor then gives you its pages as Markdown, with no code. From the API it's the same input: pass a run's defaultDatasetId as datasetId. In the form, pick the dataset in Or: page URLs from a dataset.

• Which field: the page address is found automatically in url, pageUrl, link, website or loadedUrl (the first that holds one in the first 100 items), or one level down (metadata.url). Otherwise set Field with the page URL (datasetUrlField). A Google Maps scraper's own url (the Maps link) is never used; its website is.
• Options apply as usual. Each URL is treated as if you had typed it in: with Follow links or Whole site it's where a crawl starts (so a list of company home pages can become their whole sites), and Max pages per run and your maximum cost per run cap the total. For a list of pages, leave both off.
• Only changed pages remembers each URL, not the dataset: a chained run reads a new dataset every time and still skips what the last run returned, and a URL shares its memory with the same URL typed in.
• Joining back: each page carries sourceTitle, sourcePlaceId and sourceIndex (0 = the dataset's first item) from the item it came from.
• Skipped, never fetched or charged: items without a URL, values that aren't a web address, and the same page twice (tracking parameters such as utm_source are dropped first); they're counted in RUN_STATS.dataset. A dataset with no page URL at all fails the run with the reason.
• Limits: the first 20,000 items and 10,000 distinct URLs of a dataset per run, read with your own account's access, read-only. A page from a dataset costs the same as one typed in; the other actor's run is billed by it.

Use it from n8n

With n8n's Apify node (Apify's n8n guide):

1. Add an Apify node with the operation Run an Actor, pick humble-echidna/page-to-markdown, paste your input JSON (for example {"urls": ["https://docs.example.com/guide/install"]}), and turn on Wait for finish.
2. Add a second Apify node with the operation Get dataset items, and set Dataset ID to the defaultDatasetId from the first node's output. Each item is one page, with its Markdown in markdown.
3. Send the pages on: an embeddings step, a vector store, a Google Doc. With One row per chunk, the chunks are in the run's second dataset, whose id the run's status message gives.

Run an Actor and get dataset does steps 1 and 2 in one node. Don't replace step 1 with a fixed Wait node, and don't read the actor's "last run" instead of the run's own defaultDatasetId: a fixed wait breaks when a run takes longer than usual (a whole site takes longer than a page), and a "last run" read returns another run's pages when two runs overlap (a schedule and a manual run, or two workflows). For one page at a time, an HTTP Request node calling the real-time API needs no run at all.

Can I use Page to Markdown from an AI agent (MCP)?

Yes, through Apify's MCP server, from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=humble-echidna/page-to-markdown"
    }
  }
}

To use an Apify API token instead of signing in, add "headers": {"Authorization": "Bearer <APIFY_TOKEN>"} next to url.

The agent passes the pages it wants to read, e.g. {"urls": ["https://docs.example.com/guide/install"]}, and gets each page's Markdown back; set maxResults to keep the answer small. An agent that has just run a crawler or another scraper can pass its dataset instead: {"datasetId": "<dataset id>", "maxResults": 20}.

Use it as a real-time API: one page per request (Standby)

For an agent's web-fetch tool, or any app that reads pages one at a time, call the actor's Standby URL instead of starting runs. There's no run to wait for: each request returns one page's Markdown.

curl -H "Authorization: Bearer <APIFY_TOKEN>" \
  "https://humble-echidna-1--page-to-markdown.apify.actor/?url=https://docs.apify.com/platform/actors"

• Answer: the page's Markdown (text/markdown). Add &format=json for the full record (title, description, language, links, word count, content hash), or &includeImages=true to keep images. POST / with a JSON body {"url": "...", "format": "json"} works too. The Endpoints tab lists every parameter and lets you try it.
• Price: the same as a run, one page event per page returned ($0.001 on the Free plan, less on paid plans). Failed pages are free and answer with a status code and the reason: 403 robots.txt or AI opt-out, 404 not found, 415 not an HTML page, 422 needs JavaScript, 502 the site failed.
• Same rules as a run: robots.txt and AI opt-outs respected, main content only, no private or local addresses.
• Speed: the first request after a quiet spell starts the server (about 8 seconds in our test); requests that follow within 30 seconds go straight to it (0.1-0.4 seconds plus the site's own response time).
• Every page is also saved in the Standby run's dataset (Console → Runs, origin Standby).

Who it's for

Teams that keep an LLM, RAG index or AI assistant supplied with documentation they're allowed to use (their own product docs, a vendor's public docs, an internal knowledge base on a public site). The recurring job: convert the docs site into RAG chunks once, then re-run on a schedule and get only the pages that changed.

Why this one?

• One flat price, no usage bill. You pay per result at the price above and nothing else: no compute, proxy or storage charges on top, apart from the $0.00005 run-start fee, so you know what a run costs before it starts. Results you don't get aren't charged.
• Clean output, not the whole page. A readability-style extractor keeps the article and drops the site chrome around it, so you don't pay tokens (or embedding space) for menus and footers.
• Honest about what it can't do. Pages that only render with JavaScript (single-page apps) are reported as "needs JavaScript" and not charged, instead of being returned as an empty or boilerplate-only result. The same goes for PDFs and other files, missing pages, and sites that refuse the request.
• Polite by design. It checks each site's robots.txt before every request and honours its Crawl-delay (see below).
• Fast and cheap. Plain HTTP, no browser, so there's no page rendering to wait for and no browser to pay for. Most of a run's time is the sites' own response time and the politeness limits below.
• Whole sites, sitemap first. In whole-site mode it reads the site's sitemap (the same way search engines find it), so it gets every listed page without guessing, and falls back to following links when there is no sitemap.
• Only what changed. Re-run it on a schedule with "Only changed pages" and you get (and pay for) only new and changed pages: sitemap lastmod dates skip unchanged pages without even fetching them, and a content hash catches the rest.
• Chunks ready for your vector store, split by heading with a size cap, each with its heading path. Chunks are free: you pay per page.
• Reliable. One failing URL never affects the others in your run. The run log and the RUN_STATS record say exactly which URL had a problem and why.

robots.txt and politeness

• Before fetching any page, including every redirect hop and every page found by following links, it checks that site's robots.txt and skips pages it disallows. robots.txt is read once per origin (https:// + host) per run, not once per page. Skipped pages are listed in the log and in RUN_STATS (blockedByRobots), and are never charged.
• It identifies itself honestly, with the user agent HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna). Site owners can allow or block it by name: robots.txt rules written for User-agent: HumbleEchidnaApify apply to it (otherwise the * rules do).
• It also honours opt-outs from AI use. Its output is meant for LLMs, so when a site's robots.txt disallows a page for any of the well-known AI crawlers (Google-Extended, Applebot-Extended, CCBot, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User), the page isn't fetched. It's reported as "site opts out of AI use" (optedOutOfAI in RUN_STATS) and not charged. A site that blocks only one company's crawler (Bytespider, Meta-ExternalAgent, Amazonbot and the like) while allowing these hasn't opted out of AI use, and is read.
• If a site's robots.txt can't be read because the server errors, the site is treated as disallowing everything, as the robots.txt standard (RFC 9309) requires. A site without a robots.txt allows everything.
• It honours Crawl-delay (retries included) and Retry-After, and it never makes more than 2 requests at a time to one site. A site asking for more than 30 seconds between requests, or more than 90 seconds before a retry (one such long wait per site, at most 3 minutes of them per run), is reported as such rather than fetched slowly.
• It doesn't log in, and doesn't try to get around paywalls, logins or bot protection: a site that answers "403 Forbidden" is reported as refusing the request.

Limits

• No browser: pages whose content is built by JavaScript are reported as "needs JavaScript" and not charged.
• HTML pages only: PDFs and other files are reported as "not an HTML page" and not charged.
• markdown is capped at 500,000 characters per page; pages larger than 5 MB of HTML are skipped.
• Only public web pages on the standard ports (80 and 443); no logins, no proxies.
• Main-content detection is heuristic: it works best on articles, documentation and blog posts.
• Big sites and the run timeout. Pages are fetched at most 2 at a time per site, and more slowly when the site asks for a Crawl-delay (10 seconds between requests is about 360 pages an hour). Our test crawl of 300 pages of Apify's docs took under a minute, but a site of several thousand pages can take longer than the default 1-hour run timeout. The run then stops cleanly just before the timeout: it succeeds with every page converted so far (all delivered and charged as usual), and its status message says it stopped early. Raise Timeout in the run options to get the rest, or, with Only changed pages on, run it again: pages already returned are skipped, so the next run carries on with the ones not reached.
• Whole-site limits: at most 10,000 pages per site per run, and at most 20,000 URLs read from one site's sitemaps; for a bigger site, list its sections as separate URLs. A site that's a single JavaScript app with no sitemap gives nothing to follow: its start page comes back as "needs JavaScript".
• Only changed pages: a site without sitemap lastmod dates is fetched in full on every run to compare (you're only charged for changed pages). The memory holds the 20,000 most recently returned pages per search.
• Chunks: split by headings and size only; no overlap between chunks and no semantic splitting. Token counts are estimates (see above).

FAQ

Why did a page come back as "needs JavaScript"?

Its content is built in the browser by JavaScript, and this version doesn't run a browser. The page is not charged. Many such sites also publish the same content somewhere static (a docs site, a blog, an RSS feed) that works.

Why did a page come back as "blocked by robots.txt"?

The site's owner has asked crawlers not to fetch it. This actor respects that, and the page is not charged.

Why did a page come back as "site opts out of AI use"?

The site's robots.txt disallows that page for AI crawlers (see the list above). Converting it for an LLM is exactly that use, so this actor skips it, and it's not charged.

Does whole-site mode respect AI opt-outs too?

Yes. The sitemaps are fetched with the same checks as the pages: robots.txt, and its opt-outs for AI crawlers. A page the site opts out of AI use for is never fetched, even when its sitemap lists it.

Can I get only new and changed pages since my last run?

Yes: turn on Only changed pages. See Monitor a docs site and Only changed pages above.

Does it work on Docusaurus, MkDocs, Sphinx or Read the Docs sites?

Docs generators like these publish plain HTML pages, and usually a sitemap, which is what Page to Markdown reads. A local run on 2026-10-09 converted pages of Apify's documentation (built with Docusaurus) and Python's (built with Sphinx). Put the docs root in Web page URLs and turn on Whole site (sitemap first). A docs site that builds its text in the browser comes back as "needs JavaScript" and isn't charged, so try one page first.

Why Markdown rather than HTML or plain text for an LLM?

Markdown keeps the structure a model uses (headings, lists, tables, code blocks with their language, links) without the HTML tags, scripts and site menus, so the same content takes far fewer tokens than the page's HTML and keeps more meaning than plain text. Headings are also where the chunks are split, so each chunk is one section.

What chunk size should I use, and does it add overlap between chunks?

Size chunks for your embedding model with Max chunk size (approx. tokens), 10-20% below the model's input limit (the count is an estimate; see Chunks for RAG). Chunks are split at headings, then at paragraphs, with no overlap. Each chunk carries its headingPath, which you can put in front of its text before embedding for more context; if your pipeline needs overlap, add it in your loader.

How do I load the output into LangChain or a vector database?

Each row is JSON with the text in markdown and the metadata beside it. In LangChain, ApifyDatasetLoader (package langchain-apify, Apify's guide) reads a run's dataset; map each item with

dataset_mapping_function=lambda item: Document(page_content=item["markdown"] or "", metadata={"source": item["url"], "title": item["title"]})

Can it convert pages behind a login or a paywall?

No. It reads pages as a logged-out visitor, with no cookies or credentials, and doesn't get around paywalls, logins or bot protection (see Limits).

Why does it refuse localhost, 10.x.x.x, an internal hostname or a URL with a port?

It only fetches public web pages, on the standard web ports (80 for http://, 443 for https://); a URL with any other port, such as :8080, is refused. Every hostname (and every redirect) is resolved first, and a page is refused if any address it resolves to is private, loopback, link-local (cloud metadata) or otherwise not on the public internet; the connection then goes to the address that was checked. These refusals are reported as "not a public web address", not as robots.txt blocks, and aren't charged. A domain that doesn't exist is reported as such.

Can it convert PDFs to Markdown?

Not in this version: PDFs and other files are reported as "not an HTML page" and not charged. Use PDF to Text Converter for those.

The output contains a bit of menu or misses a section. What now?

Main-content detection is heuristic and works best on articles, documentation and blog posts. Please open an issue with the URL.

Something that used to work now fails. Why?

Sites change their markup without notice. The run log names the URL and what went wrong, and every other URL in the run is unaffected. Please open an issue with the input you used.

Is it legal to convert web pages to Markdown?

It fetches only the pages you give it (and, if you ask, same-site pages they link to or the site's sitemap lists), as a normal logged-out visitor, and follows each site's robots.txt, including its opt-outs for AI crawlers. It doesn't log in or get around any protection. What you do with the content (copyright, the site's terms) is your responsibility, as with any web page you save.

Related actors

Feedback and support

Found a bug, or need a field or option that isn't here? Open an issue on the Issues tab with the input you used.

Versions

Current version: 1.4. See the Changelog tab for what changed in each version.