# URL to Markdown: Website & PDF to Markdown for LLM/RAG (`mosaicked_sponge/url-to-markdown-for-llm-rag`) Actor

Convert web pages, sitemaps, docs sites and PDFs to clean Markdown for LLMs and RAG, in bulk. Keeps headings, code blocks and tables, adds exact token counts and ready-made RAG chunks. Automatic JavaScript rendering for dynamic sites.

- **URL**: https://apify.com/mosaicked_sponge/url-to-markdown-for-llm-rag.md
- **Developed by:** [Jason](https://apify.com/mosaicked_sponge) (community)
- **Categories:** AI, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.75 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown: Website & PDF to Markdown for LLM/RAG

Turn **hundreds or thousands of web pages, PDFs, sitemaps, Google Sheets of links or whole docs sites into clean, structure-preserving Markdown in one run**, ready for ChatGPT, Claude, LangChain, LlamaIndex, vector databases and RAG pipelines.

![URL to Markdown: Website & PDF to Markdown for LLM/RAG - sample output](https://api.apify.com/v2/key-value-stores/QpfrAb6wndEDB0tDd/records/output-zaxntKdwVuVVn2iyp.png)

**$1.10 per 1,000 pages** (down to $0.75 on higher Apify plans). Failed, blocked and duplicate URLs are never charged.

- 🧹 **Only the content, not the website.** Menus, headers, footers, cookie banners, sidebars, "read more" cards and comment threads are removed. Headings, lists, tables, links and code are kept.
- 💻 **Real code blocks.** Fenced code with the language tag (` ```python `), indentation intact. No more code flattened into one line.
- 📊 **Real tables.** Data tables become Markdown tables; layout tables are flattened so they don't turn into junk.
- ⚡ **Fast HTTP first, browser only when needed.** Pages built by JavaScript (empty app shells) are rendered automatically in a headless browser. Same price.
- 🛡️ **Gets past more "403 Forbidden" pages.** Refused requests are retried with a real Chrome/Safari TLS fingerprint, then in a browser, then through your proxy if you set one.
- 🔢 **Exact token counts** (OpenAI o200k_base tokenizer) per page and per chunk, so you can budget LLM context before you spend it.
- ✂️ **RAG chunks built in.** Heading-aware chunks that never exceed your token size (exact count), with overlap. Each chunk carries its heading path.
- 🕸️ **Crawl a docs site or blog.** Follow links 1–5 levels deep inside the same folder, host or domain, with include/exclude patterns. Or feed sitemaps.
- 📄 **PDFs → text**, Markdown/text files (`llms.txt`, `README.md`) pass through unchanged.
- 📥 **Bring your list any way you like:** paste URLs, add a sitemap, or link a TXT/CSV file or a Google Sheet.
- 📚 **One combined file:** every converted page also lands in `all-pages.md`, ready to upload to a Custom GPT, Claude Project or NotebookLM.
- 🧾 **Honest errors.** Every URL that failed is listed with the reason in a free `failed-urls` dataset (CSV export) and the `ERRORS` record. Failures never become empty billed rows.

### Head-to-head: this Actor vs the most-used Store fetcher

Same 10 real URLs, run on Oct 20, 2026 (Apify cloud, default settings for both). "Junk lines" are navigation/share/sign-in/comment-form lines left in the Markdown. "Prose kept" = how many of the competitor's real text paragraphs (150+ characters) also appear in our output.

| URL type | Competitor: tokens / junk lines | **This Actor: tokens / junk lines** | Prose kept | Notes |
|---|---|---|---|---|
| BBC news article | 1,776 / 14 | **1,176 / 0** | 22 of 22 | 34% fewer tokens, zero menu junk |
| Guardian long read | 5,781 / 11 | **4,295 / 2** | 30 of 30 | |
| Python docs (asyncio) | 17,266 / 45 | **13,208 / 1** | 88 of 88 | 32 code blocks fenced, language tagged where the page says it |
| Wikipedia comparison (tables) | 23,153 / 9 | **14,539 / 9** | 14 of 14 (checked by hand) | Data tables kept, nav boxes and citation markers dropped |
| JavaScript-only page (quotes.toscrape.com/js) | 30 tokens, **no content** | **276 tokens, all 10 quotes** | n/a | Rendered automatically; competitor needs a manual `wait` setting |
| Blog guide with 300+ comments (CSS-Tricks) | 67,390 / 345 | **6,294 / 11** | article kept | Comment thread removed: 10x cheaper to embed |
| arXiv PDF | 18,166 | **18,211** | same | |
| GitHub repo page | 4,278 / 13 | **1,397 / 0** | README kept | Sidebar and "you signed in" banners removed |
| G2 reviews (heavy bot protection) | ✅ 12,881 | ❌ HTTP 403 | – | Needs residential proxies; not included at this price (bring your own proxy) |
| Stack Overflow (Cloudflare) | ✅ 31,592 | ❌ HTTP 403 | – | Same |

**Totals:** we converted 8 of 10 vs 9 of 10 (the competitor returned no content for the JavaScript page, we failed the two bot-protected sites). On the 7 pages both converted we used **57% fewer tokens (59,120 vs 137,810) for the same content, with 23 junk lines vs 437**. Speed: our pages took 0.05–2.7 s each inside one batch run; the competitor took 3.8–61 s per URL, one URL per call.

**Price:** $1.10 per 1,000 pages vs $1.50 (27% cheaper), and a run takes thousands of URLs instead of one.

When to use the competitor instead: when your list is mostly sites with heavy bot protection (G2, Stack Overflow, Indeed...) and you don't have your own proxy.

### What can I use it for?

- **Feeding a chatbot or RAG app** your docs, help center, blog or knowledge base (crawl it or use its sitemap).
- **Building a Custom GPT / Claude Project knowledge file** in one click: download `all-pages.md`.
- **Bulk-converting research links** (articles, papers, PDFs) into LLM-ready text.
- **Monitoring content changes:** every row has a `contentHash`, so you can diff runs.
- **Pre-processing for summarization, translation, classification or embeddings.**

### How to use it

1. Paste URLs into **URLs**, add a **Sitemap URL**, link a **TXT/CSV/Google Sheet**, or set **Crawl links** to follow a site's links.
2. Optionally set **RAG chunk size** (for example 800 tokens with 100 overlap).
3. Click **Start**, then export the dataset as JSON, CSV or Excel, download `all-pages.md`, or pull it via the API, the MCP server, Make, Zapier or n8n.

#### Example inputs

Convert a list:

```json
{
  "urls": ["https://docs.python.org/3/tutorial/introduction.html", "https://arxiv.org/pdf/2005.11401"],
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 100
}
```

Crawl a docs section two levels deep, skipping changelogs:

```json
{
  "urls": ["https://docs.apify.com/platform/actors"],
  "crawlDepth": 2,
  "crawlScope": "samePath",
  "excludeUrlPatterns": ["*/changelog*"],
  "maxUrls": 300
}
```

Convert every URL in a Google Sheet (shared as "Anyone with the link"):

```json
{ "urls": [], "urlsFileUrl": "https://docs.google.com/spreadsheets/d/YOUR_SHEET_ID/edit#gid=0", "maxUrls": 5000 }
```

#### Example output (one row per page, shortened)

````json
{
  "url": "https://docs.python.org/3/library/asyncio-task.html",
  "finalUrl": "https://docs.python.org/3/library/asyncio-task.html",
  "statusCode": 200,
  "title": "Coroutines and tasks",
  "description": "This section outlines high-level asyncio APIs to work with coroutines and Tasks...",
  "language": "en",
  "canonicalUrl": "https://docs.python.org/3/library/asyncio-task.html",
  "markdown": "# Coroutines and tasks\n\nThis section outlines high-level asyncio APIs ...\n\n```python\n>>> import asyncio\n\n>>> async def main():\n...     print('hello')\n```",
  "wordCount": 6745,
  "tokens": 13208,
  "tokensEstimate": 14377,
  "chunkCount": 22,
  "chunks": [{"index": 0, "heading": "Coroutines and tasks", "text": "# Coroutines and tasks\n\n...", "tokens": 780, "tokensEstimate": 766, "charCount": 3062}],
  "extractor": "main",
  "fetchMethod": "http",
  "depth": 0,
  "contentHash": "3bcf912531e8…",
  "fetchMs": 1048,
  "fetchedAt": "2026-10-20T23:58:56+00:00"
}
````

`fetchMethod` tells you how the page was fetched: `http`, `http-impersonate` (real-browser fingerprint retry), `browser` (JavaScript rendering) or `proxy`.

Other outputs of each run:

- `all-pages.md`: every converted page in one Markdown file (key-value store).
- `failed-urls` dataset: one row per failed URL with the reason (free, export as CSV/Excel). Its link is in `OUTPUT.failedUrlsCsv`.
- `OUTPUT`: counts, total tokens, fallbacks used, and the status of each URL.
- `ERRORS`: the same failures as JSON: `http-error`, `blocked-by-site`, `blocked-by-robots`, `no-content`, `unsupported-type`, `blocked-private-network`, `invalid-url`.

### Input options

| Option | What it does | Default |
|---|---|---|
| URLs | Pages, PDFs, `.md`/`.txt` files | 3 sample pages |
| Sitemap URLs | Adds every page in the sitemap(s) | none |
| URL list file | Public TXT/CSV link or Google Sheet; any column | none |
| Max URLs | Hard cap per run (listed + sitemap + crawled pages) | 100 |
| Crawl links (depth) / scope / include / exclude | Follow links 0–5 levels; same folder, host or domain; glob or /regex/ filters | 0 (off) / same folder |
| Extraction mode | `main` (content only) or `full` (cleaned full page) | main |
| Content CSS selector / Remove elements | Override detection or strip extra blocks | none |
| Output formats | `markdown`, `text`, `html` | markdown |
| RAG chunk size / overlap | Heading-aware chunks in real tokens (0 = off) | off / 100 |
| Keep links / images | Keep `[text](url)` and `![alt](src)` | links on, images off |
| Add list of all page links | `links[]` with absolute URL + anchor text | off |
| JavaScript rendering | `auto` (only when needed), `never`, `always` | auto |
| Retry refused pages with a real-browser fingerprint | Free 403/429 retry | on |
| Proxy for blocked pages | Apify datacenter proxy or your own proxy URLs, used only after other retries fail | off |
| Combined Markdown file / exact token counts | `all-pages.md`; o200k_base token counts | on / on |
| Respect robots.txt | Skip disallowed URLs | on |
| Parallel requests / per domain | Speed vs. politeness | 10 / 3 |

### Pricing

Pay per page converted: **$1.10 per 1,000** on the Free and Starter plans, $0.95 on Scale, $0.75 on Business and above, plus a tiny $0.00005 run start fee. JavaScript rendering, fingerprint retries, chunking, exact tokens, the combined file and crawling are all included. Failed, blocked, skipped and duplicate URLs cost nothing. Set **Maximum cost per run** and the Actor stops cleanly when it is reached.

### Limitations

- **Heavily protected sites** (G2, Stack Overflow, Indeed and similar) often refuse datacenter traffic even from a real browser. They are reported as `http-error 403` and not charged. Add your own residential proxy URLs under **Proxy** to reach them (Apify residential proxy isn't supported because its cost would exceed the page price).
- **robots.txt is respected by default**, so some news sites (for example WSJ, Reuters) are skipped as `blocked-by-robots`. Turn it off only for your own sites or with permission.
- **No logins or cookies.** Content behind a login isn't available.
- **PDF extraction reads the text layer.** Scanned PDFs without text aren't OCR'd.
- `tokensEstimate` (about 4 characters per token) is kept for older integrations; `tokens` is the exact count.

### FAQ

**Is it compatible with my existing runs?** Yes. Every 1.0 input field works the same and every 1.0 output field is still there. New fields were added (`tokens`, `fetchMethod`, `depth`, `parentUrl`, `modifiedDate`, `image`, `keywords`).

**Does JavaScript rendering cost more?** No. It's the same price per page; the browser only starts when a page needs it.

**How do I get one file for ChatGPT / Claude?** Open the run's Storage → Key-value store → `all-pages.md`, or use the "All pages as one Markdown file" link on the Output tab.

**Can I call it from an AI agent?** Yes: through the Apify API, the Apify MCP server, or integrations like Make, Zapier and n8n.

### Is it legal?

This Actor only reads publicly accessible pages, follows robots.txt by default, and doesn't collect personal data by design. Respect each website's terms and copyright when you store or republish content. If you're unsure, check with a lawyer.

### Feedback

Found a page that converts badly? Open an issue with the URL, and it will be looked at quickly.

# Actor input Schema

## `urls` (type: `array`):

Web pages, PDFs or Markdown/text files to convert. One URL per line; up to 10,000 per run. Pages behind a login are not supported.

## `sitemapUrls` (type: `array`):

sitemap.xml, sitemap index, .xml.gz or plain-text sitemap. Every page listed is added to the run (up to Max URLs).

## `urlsFileUrl` (type: `string`):

A public link to a .txt or .csv file, or a Google Sheet shared as "Anyone with the link". Every cell that looks like a web address is added to the run (any column, header row ignored).

## `maxUrls` (type: `integer`):

Hard cap on pages processed in one run (URLs + sitemap pages).

## `crawlDepth` (type: `integer`):

0 converts only the URLs you list. 1 also converts the pages those URLs link to (inside the crawl scope), 2 goes one level further, and so on. Max URLs still caps the total.

## `crawlScope` (type: `string`):

Which links are followed when crawling. Same folder keeps a docs crawl inside /docs/.

## `includeUrlPatterns` (type: `array`):

Glob patterns like https://example.com/blog/\* or regular expressions wrapped in slashes like //docs/v2//. Empty = all links in scope.

## `excludeUrlPatterns` (type: `array`):

Glob or /regex/ patterns to skip, e.g. *?page=* or */tag/*.

## `extractionMode` (type: `string`):

Main content: the article/body without menus, headers, footers, cookie banners or sidebars (best for RAG). Full page: everything visible in the HTML except scripts, navigation and footers.

## `contentSelector` (type: `string`):

Override automatic main-content detection with your own CSS selector, e.g. article.post or #docs-content. Pages where it matches nothing fall back to automatic detection.

## `removeSelectors` (type: `string`):

Elements to strip before conversion, e.g. .newsletter-box, .author-card, table.ads

## `outputFormats` (type: `array`):

Markdown keeps headings, lists, tables, code blocks and links. Plain text strips the Markdown syntax. Raw HTML is the page source (large).

## `chunkSizeTokens` (type: `integer`):

Split each page into heading-aware chunks of about this many tokens, ready for embeddings and vector databases. Each chunk keeps its heading path.

## `chunkOverlapTokens` (type: `integer`):

Tokens repeated between consecutive chunks so context is not lost at the boundary.

## `keepLinksInMarkdown` (type: `boolean`):

Keep [text](url) links (as absolute URLs). Turn off for shorter, cheaper LLM input.

## `includeLinksList` (type: `boolean`):

Adds a de-duplicated links\[] array (absolute URL + anchor text) to each row. Useful for building your own crawl.

## `keepImages` (type: `boolean`):

Keep ![alt](src) image references in main-content mode.

## `convertPdfs` (type: `boolean`):

Extract the text layer of PDF links (no OCR for scanned PDFs).

## `maxContentChars` (type: `integer`):

Truncate very long pages to keep LLM context and dataset size under control.

## `minContentChars` (type: `integer`):

Pages with less text than this are reported in ERRORS as no-content (for example JavaScript-only apps) instead of returning an empty row.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs the site's robots.txt disallows. Turn off only for your own sites or with permission.

## `maxConcurrency` (type: `integer`):

Total pages fetched in parallel across all sites.

## `maxConcurrencyPerDomain` (type: `integer`):

Keeps the load on each website polite.

## `requestTimeoutSecs` (type: `integer`):

Give up on a URL after this many seconds (per attempt).

## `maxRetries` (type: `integer`):

Retries on network errors, 429 and 5xx.

## `renderJavaScript` (type: `string`):

Auto fetches every page with fast HTTP and switches to a headless browser only when the page is built by JavaScript (empty app shell) or shows a bot check. Same price either way.

## `renderWaitMs` (type: `integer`):

Additional time to wait in the browser for late-loading content. Usually not needed: the browser already waits for the network to go quiet.

## `retryBlockedWithBrowserFingerprint` (type: `boolean`):

When a site answers 401/403/429/503 or a bot check, retry with a Chrome/Safari TLS fingerprint before giving up. Free and on by default.

## `proxyConfiguration` (type: `object`):

Off by default. If you turn on Apify Proxy (datacenter) or add your own proxy URLs, it is used ONLY for pages that still refuse after the fingerprint retry. Residential proxy is not supported.

## `saveCombinedMarkdown` (type: `boolean`):

Saves all converted pages as a single all-pages.md file in the run's key-value store: ready to upload to a Custom GPT, Claude Project or NotebookLM.

## `exactTokenCounts` (type: `boolean`):

Adds tokens (exact count with OpenAI's o200k_base tokenizer, used by GPT-4o/4.1/o-series) to each page and chunk, and sizes chunks by real tokens.

## `userAgent` (type: `string`):

Leave empty to use a standard desktop browser User-Agent.

## Actor input object example

```json
{
  "urls": [
    "https://docs.python.org/3/tutorial/introduction.html",
    "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview",
    "https://llmstxt.org/"
  ],
  "maxUrls": 100,
  "crawlDepth": 0,
  "crawlScope": "samePath",
  "extractionMode": "main",
  "outputFormats": [
    "markdown"
  ],
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 100,
  "keepLinksInMarkdown": true,
  "includeLinksList": false,
  "keepImages": false,
  "convertPdfs": true,
  "maxContentChars": 0,
  "minContentChars": 50,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "maxConcurrencyPerDomain": 3,
  "requestTimeoutSecs": 30,
  "maxRetries": 2,
  "renderJavaScript": "auto",
  "renderWaitMs": 0,
  "retryBlockedWithBrowserFingerprint": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "saveCombinedMarkdown": true,
  "exactTokenCounts": true
}
```

# Actor output Schema

## `results` (type: `string`):

One row per extracted page: Markdown, metadata, token counts, chunks.

## `summary` (type: `string`):

OUTPUT record: counts, total tokens, per-URL status.

## `errors` (type: `string`):

ERRORS record: failed, blocked and skipped URLs with the reason.

## `combined` (type: `string`):

all-pages.md: every converted page in one file (for Custom GPTs, Claude Projects, NotebookLM).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.python.org/3/tutorial/introduction.html",
        "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview",
        "https://llmstxt.org/"
    ],
    "chunkSizeTokens": 800,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("mosaicked_sponge/url-to-markdown-for-llm-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://docs.python.org/3/tutorial/introduction.html",
        "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview",
        "https://llmstxt.org/",
    ],
    "chunkSizeTokens": 800,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("mosaicked_sponge/url-to-markdown-for-llm-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.python.org/3/tutorial/introduction.html",
    "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview",
    "https://llmstxt.org/"
  ],
  "chunkSizeTokens": 800,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call mosaicked_sponge/url-to-markdown-for-llm-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mosaicked_sponge/url-to-markdown-for-llm-rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zaxntKdwVuVVn2iyp/builds/dS5GckeHagybzhZ4O/openapi.json