Web Page to Markdown for LLM & RAG: Fast URL Fetcher

This Actor is a URL to Markdown and HTML to Markdown converter for LLMs, RAG pipelines, vector databases and AI agents. Give it a list of URLs and get back clean Markdown (or plain text / cleaned HTML) plus metadata, links and a token estimate for each page, ready to paste into a prompt or an index. It is a cheaper alternative to a RAG web browser when you already have the URLs: it does not search Google, it only fetches the pages you give it.

• Fast and cheap by default. Pages are fetched with plain HTTP. A headless Chromium starts only when a page is client-rendered (React/Vue/Angular shells), and only that page is billed at the browser price.
• Main content only. Mozilla Readability (Firefox Reader View) plus removal of navigation, footers, scripts, dialogs and cookie banners. Tables, lists, code blocks, links and images are kept as GitHub-flavored Markdown with absolute URLs.
• Rich metadata. Title, description, author, language, canonical URL, published/modified dates, Open Graph, JSON-LD, word count and a token estimate.
• Your URLs only. No search engine step: it fetches exactly the URLs you give it.
• Polite. Respects robots.txt and Crawl-delay, per-host rate limits, no login or captcha bypass.

Use cases

• Feed documentation, articles or product pages into a RAG index or a custom GPT.
• Give an AI agent a "read this URL" tool that returns compact Markdown instead of raw HTML.
• Convert a list of pages to Markdown for summarization, translation or archiving.
• Extract titles, descriptions, canonical URLs and links from many pages at once.

Input

Field	Default	What it does
startUrls	–	URLs to fetch (one result each).
query	–	Optional: URLs as text (new line or comma separated). Compatible with RAG Web Browser's field name. Search terms are rejected.
outputFormats	["markdown"]	Any of markdown, text, html.
scrapingTool	auto	auto (HTTP, browser only when needed), raw-http, or browser-playwright (always render).
htmlTransformer	readableTextIfPossible	readableText, readableTextIfPossible or none (whole cleaned page).
removeElementsCssSelector / keepElementsCssSelector	nav, footer, … / –	Fine-tune what's removed or kept.
removeCookieWarnings	true	Strip cookie-consent banners from the output (nothing is clicked).
includeLinks, maxLinksPerPage	true, 500	Links with anchor text and internal/external flag.
requestTimeoutSecs, maxRequestRetries	40, 1	Timeouts and retries on network errors / 5xx.
dynamicContentWaitSecs, waitUntil, waitForSelector, blockMedia	10, load, –, true	Browser-only rendering controls.
desiredConcurrency, minDelayPerHostMs	5, 500	Parallelism and per-host politeness.
proxyConfiguration	off	Optional datacenter proxy.
debugMode	false	Adds the render decision and page signals.

{
  "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }, { "url": "https://crawlee.dev" }],
  "outputFormats": ["markdown", "text"]
}

Output

One dataset item per URL:

{
  "url": "https://crawlee.dev/",
  "crawl": { "loadedUrl": "https://crawlee.dev/", "loadedAt": "2026-11-11T20:30:21.341Z", "httpStatusCode": 200, "httpStatusMessage": "OK", "requestStatus": "handled", "renderedWith": "raw-http", "durationMs": 412, "retries": 0 },
  "metadata": { "title": "Crawlee · Build reliable crawlers. Fast.", "description": "…", "author": null, "languageCode": "en", "canonicalUrl": "https://crawlee.dev/", "publishedAt": null, "modifiedAt": null, "imageUrl": "https://crawlee.dev/img/…", "siteName": null, "openGraph": { "title": "…" }, "jsonLd": [] },
  "markdown": "## Build reliable web scrapers. Fast.\n\nCrawlee is a web scraping library…",
  "links": [{ "url": "https://crawlee.dev/js/docs/quick-start", "text": "Get started", "internal": true }],
  "wordCount": 233,
  "tokenEstimate": 1650,
  "error": null
}

requestStatus is handled, failed (with error, e.g. HTTP 404) or blockedByRobots. A RUN_SUMMARY record has the counts.

Pricing

Pay per event: you pay only for pages you actually get, and Apify platform usage is included.

Event	Price	When
Page (HTTP)	$0.50 per 1,000	Each page fetched with plain HTTP and converted
Page (browser)	$1.00 per 1,000	Each page that needed a headless browser for JavaScript

Failed pages, robots.txt-blocked pages and invalid inputs are saved with an error and not charged. Apify adds a tiny per-run start fee. Set Maximum cost per run to cap spending.

Example: 10,000 documentation pages ≈ $5.00. The same pages through RAG Web Browser's fetch event cost $22.80 on the Free tier ($13.30 on Gold).

FAQ

Does it search Google? No. It fetches the URLs you give it. Free text in query is rejected with a clear error.

Will it render JavaScript sites? Yes. In auto mode it detects client-rendered pages (empty app root, "enable JavaScript" notices, script-only shells) and renders just those with Chromium. Use browser-playwright to render every page.

How do I get only the article text? Keep htmlTransformer on readableTextIfPossible (default) or readableText. Use keepElementsCssSelector (e.g. article) for full control.

Does it handle PDFs? No, HTML and text pages only. Other content types are returned with an error and not charged.

Can it get past logins, paywalls or captchas? No, and it never tries. Pages that block automated access come back as errors.

Why is a page blockedByRobots? The site's robots.txt disallows it. The Actor always follows robots.txt.

Legal & data use

• The Actor fetches only public pages you list. It obeys robots.txt and Crawl-delay, rate-limits per host and never logs in or bypasses captchas, paywalls or other access controls.
• It doesn't target personal data and has no email or phone extraction. Output is the page's content and metadata.
• You're responsible for having the right to fetch and use the content (copyright, site terms, applicable law), especially before reusing it to train or ground AI models.

Running on Apify

You can run this Actor on the Apify platform from the Console, the API, a schedule, integrations (webhooks, Slack, Google Sheets, Make, Zapier) or MCP.

Want code? There's a walkthrough with JS and Python examples, plus chunking and embedding: docs site to Markdown for RAG on dev.to.

To use it from Claude, Cursor or another MCP client, add Apify's hosted MCP server and pin this actor: https://mcp.apify.com/?tools=fetch-actor-details,datlab/web-page-to-markdown

Other DatLab actors

• Website to Markdown Crawler — crawl a site into Markdown, with sitemaps and llms.txt
• Sitemap Change Monitor — extract sitemap URLs and watch new, removed and updated pages
• ATS Job Listings — Personio, Teamtailor, Workable and Recruitee, one schema
• SAM.gov Contract Opportunities — US federal opportunities from the official daily extract, no API key
• Bulk Website Screenshot — full page or mobile screenshots of a URL list, PNG, JPEG or PDF