Website to Markdown Crawler - Content Crawler for LLM & RAG

What does Website to Markdown Crawler do?

Website to Markdown Crawler converts a website to Markdown for RAG: give it a start URL (a docs site, help center or blog) and get clean, LLM-ready Markdown for every page, RAG chunks with heading paths and one combined OUTPUT.md, for $1 per 1,000 pages ($2.50 when a real browser is needed), compute included. Failed, blocked and duplicate pages are free.

How to convert a website to Markdown for RAG

1. Enter the site in Start URLs, one per line (for example https://docs.example.com/ or just docs.example.com), and set Max pages. A small first run of 10-20 pages shows you the output for a few cents.
2. Optional: set RAG chunk size (characters), for example 2000.
3. Click Start. Each page becomes one row with markdown and chunks; OUTPUT.md holds the whole site in one file. A page that cannot be read gets a free row with an error and a hint saying what to try (see FAQ).
4. Load the chunks into your vector store (LangChain and LlamaIndex examples below) or fetch them via API. A 1,000-page docs site costs $1 in fast mode.

Why this one

• A price you know before the run. Apify's free Website Content Crawler (WCC) bills compute time plus Apify Proxy instead: its README estimates about $0.20 per 1,000 pages over plain HTTP and $0.50–$5 with a browser (checked September 2026), and its default settings cost $1.55–$3.65 per 1,000 pages in our benchmark. A tuned WCC run can still cost less on large plain-HTTP sites (comparison).
• Keeps the body text. In our 5-site test (25 pages each, default settings) this Actor kept at least 97% of the sampled body paragraphs on every site and took 3–34 seconds per site; WCC took 56–200 seconds and kept 0% on a Next.js docs site and 34% on an Intercom help center (benchmark).
• Linked PDF, Word, Excel and CSV files are converted to Markdown (WCC can save the files, but does not convert them).
• RAG chunks with heading paths and one combined OUTPUT.md file (llms-full.txt style) in the same run, at no extra charge.
• No proxy or API key to set up: pages are fetched from our servers first, and a real browser is used only when a page needs it.

Input (copy and paste)

{
    "urls": ["https://docs.apify.com/academy"],
    "maxPages": 20,
    "chunkSize": 2000
}

Output (real result, shortened)

One page from a test run on docs.apify.com/academy, read in fast mode and charged $0.001. A 20-page run like the input above costs $0.02 in fast mode, and at most $0.05 if every page needed the browser:

{
    "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",
    "startUrl": "https://docs.apify.com/academy",
    "title": "Actor description & SEO description | Academy | Apify Documentation",
    "mode": "fast",
    "success": true,
    "charged": true,
    "wordCount": 1205,
    "markdown": "# Actor description & SEO description\n\nCopy for LLM\n\nLearn about Actor description and meta description. …\n\n## What is an Actor description?\n\n…",
    "contentDigestKind": "markdown:v1",
    "contentDigest": "sha256:…",
    "chunks": [
        { "index": 1, "text": "… ### Is there any benefit in the description and meta description being different?\n\n…", "headingPath": ["Actor description & SEO description", "Regular description vs. SEO description", "Is there any benefit in the description and meta description being different?"], "charCount": 1850 }
    ],
    "crawledAt": "2026-09-29T03:56:19.278Z"
}

All fields, the failed-page row and the SUMMARY record: Output example.

How much does it cost?

Pay per event, only for pages that were converted successfully:

Event	Price
Page (fast mode, plain HTTP)	$1.00 / 1,000 pages
Page (browser mode, JavaScript rendering)	$2.50 / 1,000 pages

Example: a 2,000-page documentation site costs $2.00 when every page is read in fast mode, and at most $5.00 if every page needed the browser.

No start fee. Compute time is included: the table is the whole price. In Auto mode (default) most pages use the fast mode. Duplicates, blocked pages (403, bot checks), pages disallowed by robots.txt, pages with no text even after the browser fallback, and errors are not charged. Your maximum charge limit is always respected. On Apify's Scale plan prices are 10% lower, on Business and higher 20% lower ($0.80 / 1,000 fast pages).

Features

• 📄 Documents too: linked PDF, Word (DOCX), Excel (XLSX) and CSV files are converted to Markdown, not just downloaded
• ✂️ RAG chunks built in: optional chunks array split at paragraph boundaries with overlap, each with its headingPath (h1 > h2 > h3) and charCount, ready for embeddings
• 📚 One-file export: every converted page also goes into one OUTPUT.md file in crawl order (llms-full.txt style), each page under a # Title header with a Source: line, ready to paste into an LLM. Its link is combinedFileUrl in the SUMMARY record. Free, on by default
• 💵 Fixed price per page: $1 per 1,000 pages over plain HTTP, $2.50 when a real browser is needed. Compute is included, and failed, blocked and duplicate pages are free
• 🧹 Clean Markdown by default: menus, footers, sidebars, related-post and newsletter blocks are removed; code fences, tables and absolute links are kept. Add your own CSS selectors to remove more, or keep only one part of the page (details under What gets removed in Tips)
• 🌐 Many sites in one run: paste start URLs one per line (bare domains are fine). Every page has startUrl, so you can group results by site, Max pages per start URL gives each site its share, and the SUMMARY record has one entry per site
• 🕷️ Crawls for you: start from one URL and follow links within the same folder, the whole site, or only the pages you list; the site's sitemap.xml is read too, so pages behind JavaScript menus are not missed. Links inside page content are crawled first; other languages, old versions and author/tag listing pages come last, so a small Max pages goes to real content
• 🤖 Respects robots.txt by default (can be turned off for your own sites)
• 🏷️ Page metadata: description, language, canonicalUrl and wordCount for every page
• 🔤 Markdown or plain text: choose one or both
• ⚡ Real browser only when needed: JavaScript-only pages and sites that block simple requests are rendered in headless Chrome automatically

Who is it for?

• Building a RAG chatbot or AI assistant over your docs, help center or website
• Feeding LLM agents with up-to-date documentation
• Creating a knowledge base export or content audit
• Migrating a website's content to another CMS

Price comparison with other crawlers

Compared with Apify's Website Content Crawler

Apify's free Website Content Crawler (WCC) is the most used crawler on the Store and has more options than this Actor (several HTML-to-Markdown converters, logins, expanding hidden content, HTML output). You pay for its compute time instead of per page. Its own README estimates about $0.20 per 1,000 pages with the plain HTTP crawler and $0.50–$5 per 1,000 pages with a browser, depending on the site, plus Apify Proxy (checked September 2026).

Our benchmark (5 sites, 25 pages each, both tools on the settings a new user gets, September 2026): this Actor took 3–34 seconds per site and WCC 56–200 seconds; WCC's default run cost about $1.55–$3.65 per 1,000 pages in platform usage on our account. WCC's defaults lost most of the body text on a Next.js docs site (0 of 82 sampled paragraphs) and an Intercom help center (34%); this Actor kept at least 97% on all five sites. WCC was still cleaner on blog pages (fewer leftover footer and related-post lines). Method and per-run numbers: benchmark page.

So for large sites that work over plain HTTP, a tuned WCC run can cost less than this Actor. Choose this Actor when you want:

• a price you know before the run: $1 or $2.50 per 1,000 pages, and nothing for failed, blocked or duplicate pages;
• linked PDF, Word and Excel files converted to Markdown (WCC can save the files, but does not convert them);
• RAG chunks with heading paths in the same run.

Already crawled a site with WCC? Website to Vector Embeddings can embed its dataset directly.

Other paid crawlers on the Apify Store, per 1,000 pages (checked September 2026): worshipful_knife Website Content Crawler $1.00 (Business plan $0.70) plus platform usage and a start fee; joyouscam35875 Website Content Crawler free today, $2.00 from October 5, 2026; datascoutapi Website Content Crawler Pro $2.99; easyapi Website Content to Markdown $2.99 plus a $0.09 start fee and platform usage; 6sigmag Fast Website Content Crawler $3.00 plus a start fee; parseforge Website Content Crawler $20–$25.

	This Actor
Fixed price per page, compute included	✅
PDF, Word and Excel links converted	✅
RAG chunks with heading paths	✅
Navigation and footer removal	✅
JavaScript pages (real browser when needed)	✅
Sitemap seeding and robots.txt	✅
Duplicates and failed pages free	✅

Control your cost

• What is charged: each page converted successfully, at the fast or the browser price.
• What is free: failed and blocked pages, pages with no text (errorType: "no_text"), duplicate pages, pages disallowed by robots.txt and invalid input lines. Every row has charged: true or false.
• Max pages caps the whole run; with several start URLs, Max pages per start URL stops one big site from using it all.
• At the start, the run shows an estimate in the log and the run's status message, e.g. Estimate: up to 50 page(s) x $0.001 (fast) to $0.0025 (browser) per page = at most $0.125 (about $0.05 if pages read in fast mode). You pay only for pages actually returned; failed, empty and duplicate pages are free.
• It also logs its worst case, e.g. Plan: 50 pages × $0.0025 = at most $0.125 (every page at the browser price), and warns when that is more than your maximum charge per run.
• The log also shows the expected cost, and the SUMMARY record has both: maxCostUsd (the most this run can cost: every planned page at the browser price, or the fast price in Fast mode, capped at your maximum charge per run) and expectedCostUsd (every planned page in fast mode, the usual case).
• When the maximum charge per run is reached, the run stops and keeps all pages converted so far. The status message says so, and the SUMMARY record has status: "LIMIT_REACHED" and notProcessed (how many queued pages were not crawled, and up to 100 of them).

From a test run to the whole site

Start with a small run (10-20 pages). If it converts at least one page and stops only because it reached Max pages or Max pages per start URL while more pages were found, the SUMMARY record in the run's key-value store has a nextStep object. For a 20-page run with a $5 maximum charge per run in Auto mode it looks like this:

{
    "nextStep": {
        "message": "Small crawl succeeded (20 pages converted). To scale up, set Crawl scope to \"Whole website\", raise Max pages to 2000; estimated cost at most $5 for up to 2000 pages.",
        "suggestedInput": { "crawlScope": "domain", "maxPages": 2000 },
        "estimatedCostUsd": 5
    }
}

• suggestedInput lists only the fields to change: crawlScope set to domain (Whole website), maxPagesPerStartUrl set back to 0 (no per-start-URL limit) if you had set one, and maxPages set to the number of pages your budget covers. Copy these values into a new run.
• estimatedCostUsd is the most the new run can cost: Auto and Browser mode count every page at the browser price ($2.50 per 1,000), Fast mode at the fast price ($1 per 1,000). If you set a maximum charge per run, maxPages is kept within it; without one, maxPages is 10,000 (the highest allowed), so the estimate is $25 in Auto mode. You pay only for pages actually converted; failed, blocked and duplicate pages are free.
• There is no nextStep when nothing was left to crawl, when no page was converted, when the run stopped because of the memory or maximum charge limit, or when the suggested values would not change your input.

How to use it

1. Enter one or more Start URLs, one per line, e.g. https://docs.example.com/ or just docs.example.com.
2. Choose the Crawl scope (same folder is best for documentation).
3. Set Max pages.
4. Optional: keep Remove navigation, headers and footers on (the default), set a Content selector such as main or article, URL include/exclude patterns, Output formats, and a RAG chunk size (e.g. 2000).
5. Optional: Use the site's sitemap is on (Seed) by default; choose Only to convert just the sitemap's pages, or Off to follow links only.
6. Click Start and download the results as JSON, CSV or Excel, or fetch them via API.

Input example

{
    "urls": ["https://docs.apify.com/academy"],
    "crawlScope": "path",
    "maxPages": 200,
    "maxPagesPerStartUrl": 0,
    "mode": "auto",
    "removeBoilerplate": true,
    "removeSelectors": ".related-posts, #comments",
    "excludeUrlPatterns": ["**/changelog/**"],
    "useSitemap": "seed",
    "respectRobotsTxt": true,
    "outputFormats": ["markdown", "text"],
    "chunkSize": 2000,
    "chunkOverlap": 200
}

Output example

Real output from a test run on https://docs.apify.com/academy with Remove navigation, headers and footers on, plain text and 2,000-character chunks (long text shortened with …):

{
    "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",
    "finalUrl": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",
    "startUrl": "https://docs.apify.com/academy",
    "title": "Actor description & SEO description | Academy | Apify Documentation",
    "depth": 0,
    "httpStatus": 200,
    "contentType": "text/html; charset=utf-8",
    "mode": "fast",
    "via": "backend",
    "success": true,
    "charged": true,
    "description": "Learn about Actor description and meta description. Where to set them and best practices for both content and length.",
    "language": "en",
    "canonicalUrl": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",
    "wordCount": 1205,
    "foundIn": "sitemap",
    "markdown": "# Actor description & SEO description\n\nCopy for LLM\n\nLearn about Actor description and meta description. …\n\n## What is an Actor description?\n\n…",
    "text": "Actor description & SEO description\n\nCopy for LLM\n\nLearn about Actor description and meta description. …",
    "contentDigestKind": "markdown:v1",
    "contentDigest": "sha256:…",
    "chunks": [
        { "index": 1, "text": "… ### Is there any benefit in the description and meta description being different?\n\n…", "headingPath": ["Actor description & SEO description", "Regular description vs. SEO description", "Is there any benefit in the description and meta description being different?"], "charCount": 1850 }
    ],
    "crawledAt": "2026-09-29T03:56:19.278Z"
}

Without the clean-up option, pages on this site start with [Skip to main content](#…) and On this page, and end with the page's table-of-contents links. On docs.apify.com/academy/web-scraping-for-beginners the option cut the Markdown from 7,612 to 7,108 characters by removing exactly those parts.

• via says where the page was fetched from: backend (our servers), direct (Apify's network or your proxy) or browser.
• foundIn is start, sitemap or link.
• startUrl is the start URL the page was reached from. url is the address requested, kept as you entered it for start pages (a trailing / is not removed); start pages also have inputUrl, the line exactly as you typed it.
• description, language and canonicalUrl are null when the page does not declare them (and may be null for pages rendered in the browser).
• text appears only when Output formats includes Plain text; markdown is null if you leave Markdown out.
• contentDigestKind and contentDigest are a SHA-256 digest of this row's own output text, so you can tell whether two rows carry the same content, or diff two crawls of the same site, without re-reading the pages. contentDigest is sha256: plus the hex digest of the UTF-8 bytes of <contentDigestKind>\0<field value>, with no other normalisation: the same text always gives the same digest, and one changed character gives a different one. contentDigestKind is markdown:v1 (the digest is of the markdown field) when Markdown output is on, and text:v1 (the text field) when you leave Markdown out; when the row has both, the digest is of markdown.
• The digest is of this row's own output text only — not of OUTPUT.md, the page's original HTML, or the fetch time — so it changes only when the row's content changes. Failed pages and pages with no text have no digest fields (see the failed-page row below).
• Tip: turn on Remove navigation, headers and footers, or set Content selector to main or article, to drop "Skip to main content" and menu text.

A failed page is not charged and looks like this:

{ "url": "https://example.com/missing", "startUrl": "https://example.com/", "depth": 1, "success": false, "error": "target returned HTTP 404 (not charged)", "errorType": "not_found", "hint": "The page does not exist (HTTP 404/410). Check the start URL for typos, …", "routesTried": ["backend"], "charged": false }

The run's key-value store has a SUMMARY record with status (SUCCESS, PARTIAL_RESULTS, FAILED, NO_RESULTS, LIMIT_REACHED or INVALID_INPUT), page counts, disallowedByRobotsTxt, sitemapUrlsQueued, emptyPagesNotCharged, sites (one entry per start URL: pages, failed, duplicates, notCrawled), the cost estimate maxCostUsd and expectedCostUsd, and combinedFileUrl: the link to OUTPUT.md, all pages in one Markdown file. Very large crawls split that file into 8 MB parts (OUTPUT-2.md, …) listed in combinedFileUrls. Turn it off with Also save one combined Markdown file (combinedFile: false).

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/website-markdown-crawler) to Claude, Cursor or any MCP client, then ask e.g. "Crawl the first 20 pages of docs.apify.com/academy and summarize what the course covers."

Minimal input:

{ "urls": ["https://docs.apify.com/academy"], "maxPages": 20 }

Failed items are not charged and carry an errorType.

Use it from code

Example with the Apify API:

curl -X POST "https://api.apify.com/v2/acts/tidytools~website-markdown-crawler/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://docs.example.com"],"maxPages":20}'

LangChain (Python)

from langchain_apify import ApifyWrapper
from langchain_core.documents import Document

loader = ApifyWrapper().call_actor(
    actor_id="tidytools/website-markdown-crawler",
    run_input={"urls": ["https://docs.apify.com/academy"], "maxPages": 20},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("markdown") or "", metadata={"source": item["url"], "title": item.get("title")}
    ),
)
docs = loader.load()

LlamaIndex (Python)

from apify_client import ApifyClient
from llama_index.core import Document

client = ApifyClient("YOUR_TOKEN")
run = client.actor("tidytools/website-markdown-crawler").call(
    run_input={"urls": ["https://docs.apify.com/academy"], "maxPages": 20})
docs = [Document(text=i["markdown"], metadata={"url": i["url"], "title": i.get("title")})
        for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i.get("success")]

Schedules and integrations: run it daily or weekly with Apify Schedules, get a webhook when a run finishes, or send the results to Zapier, Make, n8n, Google Sheets, Slack and other apps with Apify integrations. Results can be exported as JSON, CSV, Excel or XML.

Tips

• Documentation sites: keep the scope on Same folder and keep Remove navigation, headers and footers on. If the start page links to nothing inside its own folder (sites that give every page its own folder, such as /en/getting-started/), the crawl uses the folder one level up for it (/en/) and lists it in SUMMARY scopeWidened. If the start URL redirects to another host or folder (a docs site that moved, e.g. docs.example.io/en/stable/ to docs.example.com/platform/stable/), links under the redirect target are crawled as well and it is listed in SUMMARY scopeRedirected. If some menu text is left, add its selector to Also remove these elements or set the content selector to main or article.
• Calling the API? Clean-up and tidying are on unless you send "removeBoilerplate": false or "tidyMarkdown": false. Use the plain urls list: the older startUrls field still works, but Apify rejects the whole run (HTTP 400) when it holds a bare domain or a blank line.
• Several sites at once: set Max pages per start URL (e.g. 20) so each site gets pages, and group the results by startUrl.
• JavaScript-heavy apps (React, Vue, Angular): Auto mode switches to the browser automatically; choose Browser to force it.
• Only part of a site: use include patterns like https://example.com/blog/**. Start URLs are always converted, even when they do not match the patterns.
• Big sites: raise Max pages and Parallel pages; you only pay for pages converted. Memory use is bounded: in our test a 2,700-page crawl peaked at about 83 MB, so the default 256 MB covers roughly 5,000 pages; for 10,000 pages set the run's memory to 512 MB. If a crawl would not fit, the log warns at the start, and if memory still runs short the run stops cleanly with the pages done so far (SUMMARY.status is LIMIT_REACHED, stoppedEarly.reason is memory); run again with more memory for the rest. In browser mode, pages that find no free browser ("no browser became available") are retried twice after a short wait before they count as failed (free).
• Only the pages in the sitemap: set Use the site's sitemap to Only.
• A site blocks our servers: under Advanced settings, Auto already retries from Apify's network; add an Apify proxy if it still fails (proxy usage is billed to your Apify account).

What gets removed

With removeBoilerplate on, this also removes hidden elements (hidden or inline display:none) before Markdown conversion.

Remove navigation, headers and footers drops menus, footers, sidebars, breadcrumbs and "Skip to main content" links, and Tidy the Markdown removes skip links left at the top ("Skip to content"), leftover <!doctype> lines, empty headings, tracking pixels, schema.org JSON-LD blocks and sections repeated word for word (e.g. a page rendered twice for desktop and mobile). When a page has one clear main content element (<article>, role="main" or <main>), only that part is converted, so "Was this helpful?" widgets, theme pickers and related-post blocks around it stay out. Code blocks keep their fences (also on Sphinx/Pygments sites), links are absolute, heading permalinks ("¶", "#" or an icon) are removed, and footnote markers such as [1] or [a] that only jump to a note on the same page are removed. Blog posts lose their tag, category and author link lines (and "By … " bylines with their avatars), and page-end "Related tags", "Related posts", share, follow, newsletter and comment-count sections; code samples lose their tab labels ("JavaScriptTypeScript", "npm Yarn pnpm"). A block repeated on many pages of the same site (a footer, a banner, a "this page is also available as…" note) is removed from the fourth page on; headings, tables, code, dates and file names above code always stay..

Limitations

• Only public http/https pages; logins, private networks and local addresses are not supported. The one exception is Custom request headers, meant for your own docs or staging site (for example an Authorization token): they are sent only to the host(s) of your start URLs, never logged, and pages read with them are never rendered in a browser. This Actor does not log in to third-party sites and must not be used to access accounts or content you are not authorised to use.
• Sites that block automated access are reported as failed and not charged.
• With Respect robots.txt on, disallowed pages are skipped (counted in the SUMMARY record, not in the dataset).
• Linked documents up to 15 MB are converted; images are not described.
• title is the page's <title>; when a page has none (e.g. a bare HTML document), the first heading of the converted content is used instead, and it is null only if the page has no heading either.
• Remove navigation, headers and footers works from common HTML tags, roles and class names (nav, footer, aside, [role=banner], .site-header, .breadcrumb, .sidebar…). Unusual layouts can keep some menu text; use Also remove these elements for those.

Is it legal?

By default the crawler reads each site's robots.txt and skips the pages it disallows (they are free and counted as disallowedByRobotsTxt in the SUMMARY record); turn Respect robots.txt off only for sites you own or may crawl. It fetches only public pages: it does not log in or get past paywalls. Crawling publicly available pages is generally allowed, but you are responsible for how you use the content: respect the site's terms of service, copyright and privacy laws. This is not legal advice.

FAQ

My run shows failed pages (HTTP 5xx, 403 or "no text"). What now? Failed pages are never charged. Each failed row has an errorType and a hint with the next step. HTTP 5xx means the site's own server errored: the Actor already read the page twice, so try again later. 403 or a bot check means the site refuses automated requests: set Plain HTTP requests from to Apify's network and add an Apify proxy (Advanced settings), or set Page fetching to Browser. A 404 means the address is wrong: start from the site's home page or documentation index. routesTried shows which routes (backend, direct, vm, browser) were attempted.

What if my input is wrong (no URL, a typo, bad headers)? The run ends normally without crawling or charging anything. The dataset has one row with errorType: "invalid_input" and a hint, and the SUMMARY record has status: "INVALID_INPUT".

Can I convert a list of URLs to Markdown without crawling? Yes. Set Crawl scope to Only the start URLs (no crawling) and paste your URLs one per line.

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

How is it different from Apify's Website Content Crawler? This Actor has a fixed price per page with compute included ($1 per 1,000 pages over plain HTTP, $2.50 with a browser). Website Content Crawler has more options and bills compute time, so a tuned run can cost less on large sites that work over plain HTTP. In our 5-site test with default settings, this Actor was faster on every site and kept more of the body text on docs and help-center pages, while WCC left less clutter on blog pages (see the comparison above).

My start URL redirects to a new address (a docs site that moved). Is the crawl cut short? No. After the start page is read, the folder or host it redirected to is added to the crawl scope (the original stays), so its links are followed; the redirect is listed in SUMMARY scopeRedirected. Include and exclude patterns still apply to the pages found.

How do I go from a test crawl to the whole site? Run 10-20 pages first, then open the run's SUMMARY record. If it has a nextStep, copy its suggestedInput (crawlScope: "domain" and a maxPages your budget covers) into a new run. estimatedCostUsd is the maximum, not a quote: you pay only for pages actually converted. See From a test run to the whole site.

Does it work on JavaScript websites? Yes. In Auto mode pages that need JavaScript, or sites that block simple requests, are rendered in headless Chrome automatically and charged at the browser price.

Is the output ready for RAG and vector databases? Yes. Set RAG chunk size (characters) to get a chunks array split at paragraph boundaries with overlap, each with its headingPath and charCount. The LangChain and LlamaIndex examples above load the dataset directly.

Can I crawl my own docs site that needs a token? Yes, if it is your own site (or one you are authorised to access). Put the header in Custom request headers (your own site only), e.g. {"Authorization": "Bearer <token>"}. It goes only to your start URLs' host over a direct connection (a redirect to another host drops it), values are stored encrypted and never logged, and such pages are read in fast mode without the browser fallback. Sitemaps and robots.txt are fetched without the headers, so pages that are only listed in a protected sitemap are not discovered; use links or list the URLs yourself.

How much will a run cost before I start it? At the start the log and status message show the estimate (Max pages x price per page, worst case and expected). You pay only for pages actually returned; failed, empty and duplicate pages are free.

A page has several <article> blocks (a blog index, a feed). Is everything converted? Yes. The converter narrows a page to one main element only when there is exactly one <article> (or one role=main / <main>); with several articles it keeps the whole page body, so none of them is cut off.

What are the limits? Up to 10,000 pages per run (Max pages; Max pages per start URL splits them across sites), public pages only, and linked documents up to 15 MB. The default 256 MB of memory covers roughly 5,000 pages; for 10,000 pages set 512 MB. The run never charges more than your maximum charge per run: it stops there and keeps the pages converted so far.

How fast is it? In our 5-site benchmark, each 25-page site took 3–34 seconds with default settings. Pages are processed in parallel (Parallel pages: 10 by default, up to 30); pages that need the browser take longer.

Does it work with the API, schedules, webhooks, Make, n8n or Zapier? Yes. Start runs from the Apify API, a schedule or a webhook, use the Apify apps for Make, n8n and Zapier, load the results into LangChain or LlamaIndex (examples), or call it from AI agents through MCP.

Support

Open an issue in the Issues tab with the URL and your input. Issues are checked regularly.

Found this useful? A short review on the Store helps other users find it.