# URL to Markdown for LLMs, RAG and AI Agents (`tan-solutions/url-to-markdown`) Actor

Convert web pages to clean Markdown for LLMs, RAG pipelines and AI agents. Keeps headings, tables, code and links, and adds title, metadata and optional RAG chunks. Opens a browser only for JavaScript pages and respects robots.txt. Pay only for converted pages.

- **URL**: https://apify.com/tan-solutions/url-to-markdown.md
- **Developed by:** [Tan Solutions](https://apify.com/tan-solutions) (community)
- **Categories:** AI, Developer tools, Integrations
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 converted pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown for LLMs, RAG and AI Agents

Convert web pages to clean Markdown for LLMs, RAG pipelines and AI agents. Give it URLs; get back each page as Markdown with its title and
metadata, ready for RAG pipelines, vector stores, AI agents and prompts. It reads plain HTML first and opens a real
browser only for pages that need JavaScript. You pay only for converted pages: failed, blocked and skipped pages are
free.

### Run it from the API or on a schedule

Get the Markdown back in one call (replace `<YOUR_API_TOKEN>` with your Apify API token from Console, **Settings**, **API & Integrations**):

```bash
curl -X POST "https://api.apify.com/v2/acts/tan-solutions~url-to-markdown/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.python.org/3/tutorial/introduction.html"}],"chunkSize":2000}'
```

Send `{}` (no input) to try it: the run converts one Apify docs page.

Convert the same pages again every day at 06:00 UTC, so your RAG index stays fresh, with an Apify schedule (`Qxw1MitrgVkJQisiY` is this Actor's ID):

```bash
curl -X POST "https://api.apify.com/v2/schedules?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"name": "url-to-markdown-daily", "cronExpression": "0 6 * * *", "timezone": "UTC", "isEnabled": true, "isExclusive": true,
       "actions": [{"type": "RUN_ACTOR", "actorId": "Qxw1MitrgVkJQisiY",
         "runInput": {"body": "{\"startUrls\":[{\"url\":\"https://docs.python.org/3/tutorial/introduction.html\"}],\"chunkSize\":2000}", "contentType": "application/json"}}]}'
```

In Apify Console the same is: **Schedules**, **Create new**, cron `0 6 * * *`, add this Actor with your input.

### What it does

- **Whole pages as Markdown.** By default ("Full page") it keeps the page body and drops menus, footers and cookie
  banners. Headings, lists, tables, code blocks, links and images stay. Choose "Article" for the main text only.
- **Metadata per page:** title, description, language, site name, published and modified time, canonical URL, image.
- **Chunks for RAG:** set a chunk size and get the Markdown split at headings and paragraphs, so no chunk cuts a
  sentence or a code block in half.
- **Optional front matter** (YAML) at the top of each page: title, url, description, language, date.
- **Fast:** plain HTTP first; a browser only when a page is an empty JavaScript shell.
- **JSON, XML and plain-text URLs** come back as fenced blocks or plain text instead of failing.
- **Reliable:** a timeout per page, retries on network errors, polite pacing per site, and a clear status and error
  code for every URL. Up to 1,000 URLs per run.

### Who it is for

- Developers building RAG search, chatbots or knowledge bases from websites and docs.
- AI agents and MCP clients that need to read a web page as text.
- Anyone who wants a page saved as Markdown for notes, prompts, archives or diffing.

### Quick start

1. Paste one or more URLs into **URLs**.
2. Click **Start**.
3. Open the **Output** tab and download the Markdown (JSON, CSV, Excel), or read the dataset through the API.

### Input

| Field | What it does | Default |
| --- | --- | --- |
| `startUrls` (URLs) | The pages to convert | `https://example.com` |
| `maxUrls` (Max URLs) | Stop after this many URLs (1 to 1,000); the rest are listed as skipped, free | 100 |
| `extractMode` (What to convert) | `full` = whole page body; `article` = main text only (smaller) | `full` |
| `renderMode` (Rendering) | `auto` = browser only when needed; `http` = never; `browser` = every page | `auto` |
| `includeLinks` / `includeImages` | Keep links and images in the Markdown | on / on |
| `frontMatter` | Add YAML front matter to each page | off |
| `chunkSize` | Split into chunks of about this many characters (500 to 100,000; 0 = off) | 0 |
| `timeoutSecs` | Time limit per page (5 to 120 seconds) | 30 |
| `maxConcurrency` | Pages converted at the same time (1 to 20) | 5 |
| `proxyConfiguration` | Optional proxy for the whole run | off |

Example:

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/platform/actors" }, { "url": "https://go.dev/doc/tutorial/getting-started" }],
  "extractMode": "full",
  "chunkSize": 2000
}
```

### Output example

One dataset item per URL, in input order. A real item (Markdown cut short here):

```json
{
  "url": "https://go.dev/doc/tutorial/getting-started",
  "finalUrl": "https://go.dev/doc/tutorial/getting-started",
  "status": "ok",
  "title": "Tutorial: Get started with Go - The Go Programming Language",
  "description": null,
  "markdown": "1.  [Documentation](https://go.dev/doc/)\n2.  [Tutorials](https://go.dev/doc/tutorial/)\n3.  [Tutorial: Get started with Go](https://go.dev/doc/tutorial/getting-started)\n\n# Tutorial: Get started with Go\n\nIn this tutorial, you'll get a brief introduction to Go programming. ...",
  "wordCount": 1041,
  "chunks": null,
  "metadata": {
    "title": "Tutorial: Get started with Go - The Go Programming Language",
    "lang": "en",
    "canonicalUrl": "https://go.dev/doc/tutorial/getting-started",
    "image": "https://go.dev/doc/gopher/gopher5logo.jpg"
  },
  "fetchedWith": "http",
  "httpStatus": 200,
  "contentType": "text/html",
  "crawledAt": "2026-11-13T07:49:36.789Z",
  "error": null,
  "totalMs": 1073,
  "fetchMs": 1052,
  "convertMs": 20
}
```

`status` is `ok`, `robots_disallowed`, `blocked` or `error`. With a chunk size set, `chunks` is a list of Markdown
strings. Failed items carry `error: { "code": "...", "message": "..." }`. Codes include `ROBOTS_DISALLOWED`,
`BLOCKED`, `CAPTCHA_DETECTED`, `HTTP_ERROR`, `TIMEOUT`, `UNSUPPORTED_CONTENT_TYPE`, `EMPTY_CONTENT`,
`OUTPUT_TOO_LARGE`, `CONTENT_TOO_LARGE` and `MAX_PAGES_REACHED`.

### Pricing

You pay per converted page (the `page` event). Failed, blocked and robots-disallowed pages are free, and so are
skipped URLs. If your spending limit is reached, the run stops cleanly and tells you so.

| Apify plan | Price per 1,000 converted pages |
| --- | --- |
| Free | $1.50 |
| Bronze | $1.20 |
| Silver | $1.10 |
| Gold | $1.00 |

Apify's standard Actor start event is charged when a run starts, at the platform's default price: one event per GB of
run memory (4 events at the default 4 GB).

### Limits

- robots.txt is respected. Pages it disallows are returned as `robots_disallowed` and are free.
- Public pages only: no login, no cookies, no form filling, no paywalls.
- It stops at CAPTCHAs and bot walls (returned as `blocked`, free). It does not solve CAPTCHAs or hide who it is.
- Up to 1,000 URLs per run (default 100). URLs over "Max URLs" are listed as `MAX_PAGES_REACHED` and are free.
- Page size: one item holds at most 5 million characters of Markdown (`OUTPUT_TOO_LARGE`). Pages whose HTML is too
  big for the run memory come back as `CONTENT_TOO_LARGE` (free). Memory is 4 GB by default, 2 GB minimum.
- The crawler identifies itself as `TanForgeBot/1.0 (+https://apify.com/tan-solutions)`. Site owners can allow or block
  it in robots.txt with the `TanForgeBot` token.

### FAQ

**Can AI agents and MCP clients use it?** Yes. Add it as a tool through Apify's MCP server, or call it through the
Apify API or SDKs: pass `startUrls`, then read the dataset items. Each item has the Markdown, title and URL.

**Does it work on JavaScript pages?** Yes. In auto mode, a page whose HTML has almost no text and looks like a
single-page app is opened again in a real browser. Choose "Browser for every page" to always render.

**Can it convert PDFs or Word files?** Use our **PDF, Word and PowerPoint to Markdown for LLMs and RAG** Actor for
documents. This Actor converts web pages (plus JSON, XML and plain text).

**How fast is it, and how much at once?** It converts 5 pages at the same time by default (up to 20 with "Max
parallel pages") and paces requests to each site politely. A typical page takes about 3 seconds.

**How long are my results kept?** Results go to the run's dataset in your own Apify account and follow your plan's
data retention. The Actor keeps no copy anywhere else.

### Related tools by Tan Solutions

- [Document to Markdown](https://apify.com/tan-solutions/document-to-markdown): PDF, Word and PowerPoint files to Markdown for LLMs and RAG.
- [SEO Audit Tool](https://apify.com/tan-solutions/seo-audit): on-page SEO checker with PageSpeed for any list of pages.
- [Domain Intelligence](https://apify.com/tan-solutions/domain-intel): DNS, WHOIS/RDAP, SSL and tech stack for up to 1,000 domains.
- [US Company Filings & Financials](https://apify.com/tan-solutions/company-filings): 10-K, 10-Q, 8-K filings and XBRL facts by ticker.

### More from Tan Solutions

- [URL to Markdown tool page](https://www.tansolutions.org/tools/url-to-markdown/): details and examples for this Actor on our site.
- [Guide: URL to Markdown: converter, Python, CLI or API compared](https://www.tansolutions.org/guides/url-to-markdown-converter-python-api/)
- [All Tan Solutions data tools](https://www.tansolutions.org/tools/)

### Your responsibilities

You are responsible for the URLs you submit and for lawful use of the text you get back (copyright, the site's terms,
any AI-training restrictions the site states, and privacy law for any personal data in a page). The Actor obeys
robots.txt, does not log in and stops at CAPTCHAs and bot walls. It is provided without warranty; liability is capped
at the fees you paid in the last 12 months. Questions and issues are answered within 14 days through the Actor's
Issues tab.

Not affiliated with OpenAI, Anthropic, LangChain or LlamaIndex. All product names belong to their owners.

Made by Tan Solutions.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to convert to Markdown, each with http:// or https://. Left out, the run converts one Apify docs page as a demo.

## `maxUrls` (type: `integer`):

URLs past this number are listed as not processed (free). Hard limit 1000 per run.

## `renderMode` (type: `string`):

Auto fetches the HTML and opens a browser only for pages that need JavaScript. HTTP never opens a browser. Browser renders every page.

## `extractMode` (type: `string`):

Full page (default) keeps the whole page body except menus and footers, so index, listing and product pages come through complete. Article is main content only: drops menus, footers and index-page lists; smaller output. Headings and links stay in both.

## `includeImages` (type: `boolean`):

Keep images as Markdown image links.

## `includeLinks` (type: `boolean`):

Keep hyperlinks. Off keeps only the link text.

## `frontMatter` (type: `boolean`):

Start each Markdown file with a YAML block (title, url, description, language, date).

## `chunkSize` (type: `integer`):

0 = off. Otherwise also split the Markdown into chunks of at most this many characters for RAG (500 to 100000). Code blocks are never split.

## `timeoutSecs` (type: `integer`):

Per page, from request to full body.

## `maxConcurrency` (type: `integer`):

Across all sites. Each site still gets at most 2 requests at once, 1 second apart.

## `proxyConfiguration` (type: `object`):

Off by default. One proxy address is used for the whole run; it is never changed to get past a block.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxUrls": 100,
  "renderMode": "auto",
  "extractMode": "full",
  "includeImages": true,
  "includeLinks": true,
  "frontMatter": false,
  "chunkSize": 0,
  "timeoutSecs": 30,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One item per URL: status, title, word count and any error.

## `markdown` (type: `string`):

The Markdown of each page with its URL and title.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tan-solutions/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("tan-solutions/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call tan-solutions/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tan-solutions/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Qxw1MitrgVkJQisiY/builds/Uh9baspwuhoyVjLiK/openapi.json
