# Web to Markdown for AI agents - fast page fetcher, LLM-ready (`datagleaner/web-to-markdown`) Actor

Fetch any web page or URL list and return clean, LLM-ready Markdown (main content, headings, lists, tables, code, links), or search a query and fetch the top results. Fast HTTP first, headless browser only when needed. $2 per 1,000 pages.

- **URL**: https://apify.com/datagleaner/web-to-markdown.md
- **Developed by:** [Data Gleaner](https://apify.com/datagleaner) (community)
- **Categories:** AI, Developer tools
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 pages

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web to Markdown: URL to Markdown for LLMs, RAG and AI agents

**A URL to Markdown converter and RAG web browser alternative for LLMs, RAG and AI agents, at $2 per 1,000 pages.** Give it a list of URLs, or a search query, and it returns the main content of each page as Markdown with headings, lists, tables, code blocks and links kept, and the navigation, footer and cookie banners removed. It fetches over plain HTTP first and starts a headless browser only for pages that come back empty or need JavaScript, so most pages return in a few seconds.

**It costs $0.002 per page that returns content.** Failed, blocked and error pages return an error item and are free. A blocked page's error carries a hint to retry with `proxyConfiguration` set to residential proxies.

### Built for AI agents

Agents call Apify Actors as tools through Apify's MCP server, and page fetchers are among the tools they use most. This one is priced and shaped for that job: one cheap call, a predictable output, and a failure that costs nothing.

- **Fast.** Plain HTTP first and 10 pages in parallel by default; the run's status message reports the median time per page.
- **Reliable.** Retries with backoff, and a browser fallback instead of an empty answer.
- **Low flat price.** $0.002 per page, a flat price with no start fee.
- **Also a Standby API.** A low-latency HTTP endpoint for agents, same price (see Standby mode below).
- **Safe in a loop.** A bad URL never fails the run, a `maxCharsPerPage` cap protects your context window, and the run stops at your maximum charge.

Tip for agents: set `maxCharsPerPage` to the budget you can afford in context, and leave `outputFormats` on `["markdown"]` so the response stays small.

### Input

Nothing is required. With empty input the Actor fetches three well-known pages (Wikipedia on Markdown, the Python tutorial, a Paul Graham essay), which is also what Apify's daily auto-test runs. You pay the normal $0.002 per page for them.

Empty input example: `{}`

| Field | Type | What it does |
|---|---|---|
| `urls` | list of text | Pages to fetch. One item per URL. A URL without `https://` gets it added. |
| `query` | text | Optional search phrase. The Actor searches DuckDuckGo (Google if that fails), takes the top results and fetches them. No separate search proxy or API key needed. A query that is a full URL (starts with `http://` or `https://`) is fetched directly, not searched. |
| `maxResults` | number | Search results to fetch for the query. Default 3, maximum 20. |
| `outputFormats` | list | `markdown` (default), `text`, `html`. |
| `includeLinks` | true / false | Keep links as `[text](url)`. Default on. |
| `maxCharsPerPage` | number | Cut each content field at this length. 0 means unlimited (default). Cut items get `truncated: true` and `originalLength`. |
| `timeoutSecs` | number | Per page, for HTTP and browser each. Default 30. |
| `concurrency` | number | Pages in parallel. Default 10. |
| `renderJs` | select | `auto` (default): plain HTTP first, a headless browser only when the page comes back empty or looks like a JavaScript app. `http`: never use a browser; cheapest and fastest, but a JavaScript-only page returns a "no readable content" error. `browser`: load every page in a browser first, for single-page apps that auto-detection misses; slower than HTTP. The price per page is the same in all three modes. |
| `removeElementsCssSelector` | text | CSS selectors of elements to delete before conversion, comma separated, e.g. `nav, footer, .sidebar`. Leave empty to use the built-in list (nav, footer, aside and common cookie banners). Your list replaces the built-in one; enter a single space to remove nothing. An invalid selector is skipped with a warning in the log. |
| `proxyConfiguration` | object | Optional Apify Proxy. Default is no proxy. Residential proxies help on sites that block datacenter IPs. |

```json
{
  "urls": ["https://docs.python.org/3/tutorial/introduction.html", "https://en.wikipedia.org/wiki/Web_scraping"],
  "query": "model context protocol servers",
  "maxResults": 3,
  "outputFormats": ["markdown"],
  "maxCharsPerPage": 20000
}
```

### Output

One item per URL, in the order they finish. Export as JSON, CSV, Excel or via the API. Every item has the same 22 keys; a key that does not apply is `null`, so you can rely on one fixed shape.

| Field | Meaning |
|---|---|
| `url`, `finalUrl` | The URL you gave and where it ended up after redirects. |
| `statusCode` | HTTP status of the page. |
| `title`, `description`, `language` | From the page metadata. |
| `author`, `publishedAt`, `siteName` | `author` and `publishedAt` are what the page declares in its meta tags or JSON-LD, as written (a PDF's `author` comes from its metadata). `siteName` is the site name from the page metadata. `null` when absent. |
| `markdown` | Main content as Markdown. |
| `text`, `html` | Plain text and cleaned HTML, only if requested in `outputFormats`. |
| `wordCount` | Words in the extracted content. |
| `truncated`, `originalLength` | `truncated` is true when `maxCharsPerPage` cut a field; `originalLength` is the uncut length in characters. |
| `fetchedVia` | `http` or `browser`. |
| `searchRank`, `searchQuery`, `searchTitle`, `searchSnippet` | Only for pages found through `query`. |
| `scrapedAt` | ISO 8601 UTC time. |
| `error` | Set only on failed URLs, whose content fields are `null` and which are not charged. |

```json
{
  "url": "https://docs.python.org/3/tutorial/introduction.html",
  "finalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "statusCode": 200,
  "title": "3. An Informal Introduction to Python",
  "description": "In the following examples, input and output are distinguished by the presence or absence of prompts...",
  "language": "en",
  "author": null,
  "publishedAt": null,
  "siteName": null,
  "markdown": "# 3. An Informal Introduction to Python\n\nIn the following examples, input and output are distinguished by ...",
  "text": null,
  "html": null,
  "wordCount": 3050,
  "truncated": false,
  "originalLength": 17342,
  "fetchedVia": "http",
  "searchQuery": null,
  "searchRank": null,
  "searchTitle": null,
  "searchSnippet": null,
  "error": null,
  "scrapedAt": "2026-10-09T09:12:44+00:00"
}
```

### Standby mode (HTTP API for agents)

Enable Standby for the Actor in the Apify Console, then call it as a plain HTTP endpoint with no run to start or poll:

```bash
curl -H "Authorization: Bearer <APIFY_TOKEN>" \
  "https://datagleaner--web-to-markdown.apify.actor/?url=https://example.com&format=markdown"
```

You can send the token as `?token=` instead of the header. Parameters:

- `url`: page to fetch; repeat it for up to 10 pages.
- `query` (or `q`): search phrase, as in the input. Use `maxResults` (default 3, max 20).
- `format`: `json` (default), `markdown` or `text`. `markdown` and `text` return the raw body; `Accept: text/markdown` does the same.
- `outputFormats` (`markdown,text,html`), `maxChars`, `includeLinks` (default true), `timeoutSecs` (default 30).
- `renderJs` (`auto`, `http` or `browser`, default `auto`) and `removeElementsCssSelector`, as in the input.

`?url=...` returns one JSON item (a list for several URLs or a query). Standby uses no proxy, because the request cannot set one yet. A blocked page comes back as an item with an `error` that carries the residential-proxy hint, and is free; when every page in the request fails, the response is HTTP 502. Use the normal run mode with `proxyConfiguration` for those sites. Same $2 per 1,000 pages, same `page` event, and no start fee. It is a low-latency alternative to `apify/rag-web-browser` Standby, at a flat price.

### Use with Python

```python
## pip install apify-client
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("datagleaner/web-to-markdown").call(run_input={
    "urls": ["https://en.wikipedia.org/wiki/Markdown", "https://docs.python.org/3/tutorial/introduction.html"],
    "maxCharsPerPage": 5000,
})
if run is None:
    raise SystemExit("The Actor run did not start.")

for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["url"], item.get("title"), item.get("wordCount"), item.get("error"))
```

Two pages cost at most $0.004.

### Use with JavaScript / Node.js

```js
// npm install apify-client
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('datagleaner/web-to-markdown').call({
    urls: ['https://en.wikipedia.org/wiki/Markdown', 'https://docs.python.org/3/tutorial/introduction.html'],
    maxCharsPerPage: 5000,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) {
    console.log(item.url, item.title, item.wordCount, item.error);
}
```

Or call the Apify API directly and get the items back in the same request:

```bash
curl -X POST "https://api.apify.com/v2/acts/datagleaner~web-to-markdown/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://en.wikipedia.org/wiki/Markdown"], "maxCharsPerPage": 20000}'
```

### Use it from n8n, Make, Zapier or an AI agent

Actor ID: `datagleaner/web-to-markdown`

Minimal input:

```
{"urls": ["https://en.wikipedia.org/wiki/Markdown"]}
```

Each tool below runs this Actor with your own Apify API token.

- **n8n:** add the **Apify** node (`@apify/n8n-nodes-apify`). On n8n Cloud you install it from the community node registry. Choose **Run an Actor and get dataset**, set Actor to `datagleaner/web-to-markdown` and paste the input above.
- **Make:** use the Apify app's **Run an Actor** module, then **Get Dataset Items** to read the results. **Watch Actor Runs** can trigger a scenario when a run finishes.
- **Zapier:** use the Apify action **Run Actor**, then the search **Fetch dataset items**. The trigger **Finished Actor run** starts a Zap when a run ends.
- **AI agents (MCP):** connect to `https://mcp.apify.com?tools=datagleaner/web-to-markdown`. In Claude Code:

  ```
  claude mcp add --transport http apify "https://mcp.apify.com?tools=datagleaner/web-to-markdown"
  ```

  Then run `/mcp` to sign in to Apify in your browser. Clients that take a header instead can send `Authorization: Bearer <APIFY_TOKEN>`. Claude Desktop and other clients that run local servers can use Apify's stdio server (`npx -y @apify/actors-mcp-server` with `APIFY_TOKEN` set). It does not preload this Actor, so the agent finds and runs `datagleaner/web-to-markdown` by name with its `search-actors` and `call-actor` tools. Then ask your agent in plain words, for example: "Use datagleaner/web-to-markdown to read https://docs.apify.com/platform/actors and https://modelcontextprotocol.io/introduction as Markdown, capped at 15,000 characters each, and summarise how they differ." The tool input is the same JSON as in the Input section.
- **LangChain (Python):**

```python
## pip install langchain-apify, then set APIFY_TOKEN in your environment
import json
from langchain_apify import ApifyActorsTool
tool = ApifyActorsTool("datagleaner/web-to-markdown")
result = tool.invoke({"run_input": json.loads('{"urls": ["https://en.wikipedia.org/wiki/Markdown"]}')})
```

### Pricing

Pay per event: **$0.002 per page that returns content** ($2 per 1,000), no start fee. This price applies from 21 October 2026. Failed URLs, 404s and blocked pages are free. Platform usage is included, and the run stops by itself when your maximum total charge is reached.

| Pages | Cost |
|---|---|
| 1,000 | $2.00 |
| 10,000 | $20.00 |
| 100,000 | $200.00 |

Worked example: a research agent reads 5 pages to answer each question and answers 2,000 questions a month, which is 10,000 pages, or $20.00 here. A failed fetch costs nothing.

### Limits

- **Bot protection.** Some sites (large news publishers, Cloudflare or DataDome challenge pages, Reddit) refuse automated readers. The Actor then returns an error item such as "blocked by the site's bot protection", free of charge. It does not solve CAPTCHAs.
- **Login and paywalls.** Only what a logged-out visitor sees. No cookies, no sign-in.
- **PDFs.** Text-based PDFs return their text (first 100 pages) in `markdown`. Scanned PDFs without a text layer return an error.
- **Search.** Uses DuckDuckGo's HTML results, then Google, with no SERP proxy or key. Search engines can refuse datacenter IPs; if a query returns no results, pass the URLs directly.
- **Main content only.** The extractor keeps the article or documentation body. A page that is mostly navigation or a dashboard returns little.
- **Speed.** Plain pages come back over HTTP in a few seconds. The browser fallback is slower and is used only for pages that come back empty or need JavaScript.

### FAQ

**How do I convert a URL to Markdown?** Put the URL in `urls` and run the Actor. Each page comes back as one item whose `markdown` field holds the main content. Add `text` or `html` to `outputFormats` if you want those too.

**Is there an API for web to Markdown?** Yes. Every Apify Actor is an API: use the Python or JavaScript client above, or the `run-sync-get-dataset-items` endpoint, which returns the Markdown in the same HTTP response.

**Can I use it as an MCP tool?** Yes. Add `https://mcp.apify.com?tools=datagleaner/web-to-markdown` to Claude, Cursor or any MCP client, as shown above.

**Can it convert a whole website to Markdown?** Not by crawling. It fetches exactly the URLs you give and does not follow links. To get every page of a site, list its URLs first with the Sitemap Extractor Actor, then pass them in `urls`.

**When does it use a browser?** By default only when the plain HTTP answer has almost no readable text or looks like an empty JavaScript app. `fetchedVia` tells you which path each page took. Set `renderJs` to `browser` to always render first (for single-page apps that slip through) or to `http` to never use a browser.

### Responsible use

This Actor reads only publicly visible pages. Respect each site's terms and robots rules, keep your request volume reasonable, and remember that page content is often copyrighted: the output is for analysis, retrieval and research, not for republishing. Pages may contain personal data: **you** are responsible for having a lawful basis and meeting your obligations under GDPR, CCPA and any other law that applies to your use. This is not legal advice.

# Actor input Schema

## `urls` (type: `array`):

Pages to fetch and convert to Markdown. Each URL returns one item. A URL without http(s):// gets https:// added.

## `query` (type: `string`):

Optional. Runs a web search (DuckDuckGo, then Google if blocked), takes the top results and fetches them like the URLs above. Each item carries searchRank, searchQuery, searchTitle and searchSnippet. A query that is a full URL (starting with http:// or https://) is fetched directly instead of searched.

## `maxResults` (type: `integer`):

How many search results to fetch for the query.

## `outputFormats` (type: `array`):

Which content fields to return. markdown is always the most useful for LLMs; text and html are optional extras.

## `includeLinks` (type: `boolean`):

Keep hyperlinks in the Markdown as [text](url). Turn off for the leanest text.

## `maxCharsPerPage` (type: `integer`):

Cut each content field to this many characters (0 = unlimited). Useful to protect an LLM context window. Cut items have truncated: true and originalLength set.

## `timeoutSecs` (type: `integer`):

Give up on a page after this long (HTTP and browser each).

## `concurrency` (type: `integer`):

Pages fetched in parallel.

## `renderJs` (type: `string`):

auto (default): fast HTTP first, a headless browser only when the page comes back empty or looks like a JavaScript app. http: never use a browser (cheapest, fails on JS-only pages). browser: always render the page in a browser first, for single-page apps that auto-detection misses.

## `removeElementsCssSelector` (type: `string`):

CSS selectors of elements to delete before conversion, comma separated, e.g. nav, footer, .sidebar. Leave empty to use the built-in list (nav, footer, aside and common cookie banners). Your list replaces the built-in one; enter a single space to remove nothing.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy. Default is no proxy. Residential proxy helps on sites that block datacenter IPs.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://docs.python.org/3/tutorial/introduction.html",
    "https://www.paulgraham.com/greatwork.html"
  ],
  "maxResults": 3,
  "outputFormats": [
    "markdown"
  ],
  "includeLinks": true,
  "maxCharsPerPage": 0,
  "timeoutSecs": 30,
  "concurrency": 10,
  "renderJs": "auto"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://docs.python.org/3/tutorial/introduction.html",
        "https://www.paulgraham.com/greatwork.html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("datagleaner/web-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://docs.python.org/3/tutorial/introduction.html",
        "https://www.paulgraham.com/greatwork.html",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("datagleaner/web-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://docs.python.org/3/tutorial/introduction.html",
    "https://www.paulgraham.com/greatwork.html"
  ]
}' |
apify call datagleaner/web-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datagleaner/web-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4UTArhSH5x8m5OWAv/builds/2tkUPByoAzBgi3k7G/openapi.json