# Web Page to Markdown for LLM & RAG: Fast URL Fetcher (`datlab/web-page-to-markdown`) Actor

Turn any list of URLs into clean Markdown, text and metadata for LLMs, RAG and AI agents. Plain HTTP first, a headless browser only when a page needs JavaScript. Respects robots.txt.

- **URL**: https://apify.com/datlab/web-page-to-markdown.md
- **Developed by:** [DatLab](https://apify.com/datlab) (community)
- **Features:** Limited permissions
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 page converted (http)s

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Page to Markdown for LLM & RAG: fast URL fetcher

This Actor is a **URL to Markdown** and **HTML to Markdown** converter for LLMs, RAG pipelines, vector databases and AI agents. Give it a list of URLs and get back **clean Markdown** (or plain text / cleaned HTML) plus **metadata, links and a token estimate** for each page, ready to paste into a prompt or an index. It is a cheaper alternative to a RAG web browser when you already have the URLs: it does not search Google, it only fetches the pages you give it.

- **Fast and cheap by default.** Pages are fetched with plain HTTP. A headless Chromium starts **only** when a page is client-rendered (React/Vue/Angular shells), and only that page is billed at the browser price.
- **Main content only.** Mozilla Readability (Firefox Reader View) plus removal of navigation, footers, scripts, dialogs and cookie banners. Tables, lists, code blocks, links and images are kept as GitHub-flavored Markdown with absolute URLs.
- **Rich metadata.** Title, description, author, language, canonical URL, published/modified dates, Open Graph, JSON-LD, word count and a token estimate.
- **Your URLs only.** No search engine step: it fetches exactly the URLs you give it.
- **Polite.** Respects robots.txt and Crawl-delay, per-host rate limits, no login or captcha bypass.

### Use cases

- Feed documentation, articles or product pages into a RAG index or a custom GPT.
- Give an AI agent a "read this URL" tool that returns compact Markdown instead of raw HTML.
- Convert a list of pages to Markdown for summarization, translation or archiving.
- Extract titles, descriptions, canonical URLs and links from many pages at once.

### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | – | URLs to fetch (one result each). |
| `query` | – | Optional: URLs as text (new line or comma separated). Compatible with RAG Web Browser's field name. Search terms are rejected. |
| `outputFormats` | `["markdown"]` | Any of `markdown`, `text`, `html`. |
| `scrapingTool` | `auto` | `auto` (HTTP, browser only when needed), `raw-http`, or `browser-playwright` (always render). |
| `htmlTransformer` | `readableTextIfPossible` | `readableText`, `readableTextIfPossible` or `none` (whole cleaned page). |
| `removeElementsCssSelector` / `keepElementsCssSelector` | nav, footer, … / – | Fine-tune what's removed or kept. |
| `removeCookieWarnings` | `true` | Strip cookie-consent banners from the output (nothing is clicked). |
| `includeLinks`, `maxLinksPerPage` | `true`, 500 | Links with anchor text and internal/external flag. |
| `requestTimeoutSecs`, `maxRequestRetries` | 40, 1 | Timeouts and retries on network errors / 5xx. |
| `dynamicContentWaitSecs`, `waitUntil`, `waitForSelector`, `blockMedia` | 10, `load`, –, `true` | Browser-only rendering controls. |
| `desiredConcurrency`, `minDelayPerHostMs` | 5, 500 | Parallelism and per-host politeness. |
| `proxyConfiguration` | off | Optional datacenter proxy. |
| `debugMode` | `false` | Adds the render decision and page signals. |

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }, { "url": "https://crawlee.dev" }],
  "outputFormats": ["markdown", "text"]
}
```

### Output

One dataset item per URL:

```json
{
  "url": "https://crawlee.dev/",
  "crawl": { "loadedUrl": "https://crawlee.dev/", "loadedAt": "2026-11-11T20:30:21.341Z", "httpStatusCode": 200, "httpStatusMessage": "OK", "requestStatus": "handled", "renderedWith": "raw-http", "durationMs": 412, "retries": 0 },
  "metadata": { "title": "Crawlee · Build reliable crawlers. Fast.", "description": "…", "author": null, "languageCode": "en", "canonicalUrl": "https://crawlee.dev/", "publishedAt": null, "modifiedAt": null, "imageUrl": "https://crawlee.dev/img/…", "siteName": null, "openGraph": { "title": "…" }, "jsonLd": [] },
  "markdown": "## Build reliable web scrapers. Fast.\n\nCrawlee is a web scraping library…",
  "links": [{ "url": "https://crawlee.dev/js/docs/quick-start", "text": "Get started", "internal": true }],
  "wordCount": 233,
  "tokenEstimate": 1650,
  "error": null
}
```

`requestStatus` is `handled`, `failed` (with `error`, e.g. `HTTP 404`) or `blockedByRobots`. A `RUN_SUMMARY` record has the counts.

### Pricing

Pay per event: you pay only for pages you actually get, and Apify platform usage is included.

| Event | Price | When |
|---|---|---|
| Page (HTTP) | **$0.50 per 1,000** | Each page fetched with plain HTTP and converted |
| Page (browser) | **$1.00 per 1,000** | Each page that needed a headless browser for JavaScript |

Failed pages, robots.txt-blocked pages and invalid inputs are saved with an error and **not charged**. Apify adds a tiny per-run start fee. Set **Maximum cost per run** to cap spending.

Example: 10,000 documentation pages ≈ $5.00. The same pages through RAG Web Browser's fetch event cost $22.80 on the Free tier ($13.30 on Gold).

### FAQ

**Does it search Google?** No. It fetches the URLs you give it. Free text in `query` is rejected with a clear error.

**Will it render JavaScript sites?** Yes. In `auto` mode it detects client-rendered pages (empty app root, "enable JavaScript" notices, script-only shells) and renders just those with Chromium. Use `browser-playwright` to render every page.

**How do I get only the article text?** Keep `htmlTransformer` on `readableTextIfPossible` (default) or `readableText`. Use `keepElementsCssSelector` (e.g. `article`) for full control.

**Does it handle PDFs?** No, HTML and text pages only. Other content types are returned with an error and not charged.

**Can it get past logins, paywalls or captchas?** No, and it never tries. Pages that block automated access come back as errors.

**Why is a page `blockedByRobots`?** The site's robots.txt disallows it. The Actor always follows robots.txt.

### Legal & data use

- The Actor fetches only public pages you list. It obeys robots.txt and `Crawl-delay`, rate-limits per host and never logs in or bypasses captchas, paywalls or other access controls.
- It doesn't target personal data and has no email or phone extraction. Output is the page's content and metadata.
- You're responsible for having the right to fetch and use the content (copyright, site terms, applicable law), especially before reusing it to train or ground AI models.

### Running on Apify

You can run this Actor on the Apify platform from the Console, the API, a schedule, integrations (webhooks, Slack, Google Sheets, Make, Zapier) or MCP.

Want code? There's a walkthrough with JS and Python examples, plus chunking and embedding: [docs site to Markdown for RAG on dev.to](https://dev.to/datlab/turning-a-docs-site-into-clean-markdown-for-rag-for-about-10-cents-per-1000-pages-1nfp).

To use it from Claude, Cursor or another MCP client, add Apify's hosted MCP server and pin this actor: `https://mcp.apify.com/?tools=fetch-actor-details,datlab/web-page-to-markdown`

### Other DatLab actors

- [Website to Markdown Crawler](https://apify.com/datlab/site-to-markdown-crawler) — crawl a site into Markdown, with sitemaps and llms.txt
- [Sitemap Change Monitor](https://apify.com/datlab/sitemap-change-monitor) — extract sitemap URLs and watch new, removed and updated pages
- [ATS Job Listings](https://apify.com/datlab/ats-job-listings) — Personio, Teamtailor, Workable and Recruitee, one schema
- [SAM.gov Contract Opportunities](https://apify.com/datlab/sam-gov-contract-opportunities) — US federal opportunities from the official daily extract, no API key
- [Bulk Website Screenshot](https://apify.com/datlab/bulk-website-screenshot) — full page or mobile screenshots of a URL list, PNG, JPEG or PDF

# Actor input Schema

## `startUrls` (type: `array`):

Pages to fetch. Each URL is fetched once and returned as one dataset item. This Actor fetches only the URLs you give it; it does not run web searches.

## `query` (type: `string`):

Alternative to the list above, compatible with RAG Web Browser's `query` field: one or more URLs separated by new lines or commas. Free-text search terms are rejected, since this Actor doesn't search the web.

## `outputFormats` (type: `array`):

Which content fields to include in each item.

## `scrapingTool` (type: `string`):

Auto fetches each page with plain HTTP and switches to a headless Chromium only when the HTML looks client-rendered. You're charged the browser price only for pages that actually used the browser.

## `htmlTransformer` (type: `string`):

How to pick the main content. Readable text uses Mozilla Readability (as in Firefox Reader View). 'If possible' falls back to the whole cleaned page when Readability finds too little.

## `readableTextCharThreshold` (type: `integer`):

Readability results shorter than this are treated as a failed extraction.

## `removeElementsCssSelector` (type: `string`):

Elements removed before conversion (navigation, footers, scripts, dialogs, buttons by default). Leave empty to use the default.

## `keepElementsCssSelector` (type: `string`):

Optional. If set, only matching elements are converted (e.g. `article, .docs-content`).

## `removeCookieWarnings` (type: `boolean`):

Strip common cookie-consent banners (OneTrust, Cookiebot, Didomi, Usercentrics and others) from the output. Banners are removed from the content only; nothing is clicked or accepted.

## `includeLinks` (type: `boolean`):

Add the list of links on each page (absolute URL, anchor text, internal/external).

## `maxLinksPerPage` (type: `integer`):

Caps the links list per page.

## `requestTimeoutSecs` (type: `integer`):

Maximum time to load one page.

## `dynamicContentWaitSecs` (type: `integer`):

For browser-rendered pages: how long to wait for the network to go idle after load.

## `waitUntil` (type: `string`):

Which event counts as 'page loaded' before the dynamic-content wait.

## `waitForSelector` (type: `string`):

Optional. Wait until this element appears before reading the page.

## `blockMedia` (type: `boolean`):

Faster and cheaper rendering. Doesn't affect image links in the Markdown.

## `maxRequestRetries` (type: `integer`):

Retries on network errors and 5xx responses.

## `desiredConcurrency` (type: `integer`):

How many pages to fetch in parallel.

## `minDelayPerHostMs` (type: `integer`):

Politeness: minimum gap between requests to the same host. robots.txt Crawl-delay is honored when larger.

## `proxyConfiguration` (type: `object`):

Optional. Datacenter proxies only; residential and SERP groups are not supported.

## `debugMode` (type: `boolean`):

Add a `debug` object with content type, render decision and page signals.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    },
    {
      "url": "https://crawlee.dev"
    }
  ],
  "outputFormats": [
    "markdown"
  ],
  "scrapingTool": "auto",
  "htmlTransformer": "readableTextIfPossible",
  "readableTextCharThreshold": 100,
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], form[role=\"search\"], button",
  "removeCookieWarnings": true,
  "includeLinks": true,
  "maxLinksPerPage": 500,
  "requestTimeoutSecs": 40,
  "dynamicContentWaitSecs": 10,
  "waitUntil": "load",
  "blockMedia": true,
  "maxRequestRetries": 1,
  "desiredConcurrency": 5,
  "minDelayPerHostMs": 500,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "debugMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

One item per URL with Markdown/text, metadata and links.

## `runSummary` (type: `string`):

RUN_SUMMARY record with counts of handled, failed, robots-blocked and browser-rendered pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        },
        {
            "url": "https://crawlee.dev"
        }
    ],
    "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], form[role=\"search\"], button"
};

// Run the Actor and wait for it to finish
const run = await client.actor("datlab/web-page-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" },
        { "url": "https://crawlee.dev" },
    ],
    "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], form[role=\"search\"], button",
}

# Run the Actor and wait for it to finish
run = client.actor("datlab/web-page-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    },
    {
      "url": "https://crawlee.dev"
    }
  ],
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='\''data:'\''], [role=\\"alert\\"], [role=\\"banner\\"], [role=\\"dialog\\"], [role=\\"alertdialog\\"], [aria-modal=\\"true\\"], form[role=\\"search\\"], button"
}' |
apify call datlab/web-page-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datlab/web-page-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Kv9PbSntHPUhkyMDk/builds/ic4dT14HQrOuqY5D6/openapi.json