# URL to Markdown for RAG: Pages, PDFs, Sitemaps to Clean Chunks (`atesen-software/url-to-markdown-rag`) Actor

Turn a URL, sitemap or docs site into clean Markdown and RAG chunks with heading paths, token counts and metadata. PDFs included, no headless browser. Failed, blocked and duplicate URLs are not charged.

- **URL**: https://apify.com/atesen-software/url-to-markdown-rag.md
- **Developed by:** [Atesen Software](https://apify.com/atesen-software) (community)
- **Categories:** AI, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.96 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Turn web pages, PDFs, sitemaps or whole docs sites into **clean Markdown with RAG-ready chunks**. No headless browser, so it runs at **512 MB**, converts a 200-page docs site in about 2.5 minutes and a 10-URL batch in seconds. It costs **$1.20 per 1,000 pages**. URLs that fail are **free**.

Use it to load knowledge bases, docs and blogs into a vector database. It also feeds LLM agents, custom GPTs and fine-tuning datasets.

### What you get per page

- **Markdown** of the main content. Menus, footers, cookie banners, edit links and citation markers are removed, and tables, code blocks and headings are kept.
- **Chunks** of at most `chunkSize` tokens (default 512, OpenAI cl100k tokenizer):
  - split along headings, then paragraphs, then sentences;
  - each chunk carries its **heading path** (`Guide > Install > Linux`) as context;
  - large tables are split by rows and the **header row is repeated** in every chunk;
  - code blocks are never cut in the middle when they fit;
  - optional overlap between chunks.
- **Token count, word count and content hash (SHA-1)**, so you can budget embeddings and detect changes.
- **Metadata:**
  - title, description, author, published and modified dates, language and site name;
  - Open Graph image, canonical URL and noindex flag;
  - HTTP status, final URL after redirects and crawl depth.
- **PDF text:** PDFs linked or listed get the same treatment, including page count, title and chunks. Plain-text and Markdown files are passed through.

### Why this instead of a browser-based crawler

| | **This actor** | RAG Web Browser | Website Content Crawler |
|---|---|---|---|
| Price | **$1.20 / 1,000 pages**, lower on paid plans. Failures are free. | $2.28 / 1,000 fetches on the Free plan, plus a charge per search | Free actor, but you pay for compute: browser runs with 8 GB default memory |
| Default memory | **512 MB** (runs on 256 MB) | 8 GB | 8 GB |
| Speed | **HTTP-only, parallel**: 10 URLs in ~6 s, 200-page docs site in ~2.5 min | Browser per page | Browser by default |
| Chunks for RAG | **Built in:** token-sized, heading path, table header repeated | No (whole page Markdown) | No (whole page text/Markdown) |
| Token count per page/chunk | **Yes** | No | No |
| Duplicate handling | URL, canonical URL and identical-content dedupe (free rows) | – | URL |
| Failed / blocked / non-HTML URLs | **Free status row, run still succeeds** | – | Compute still used |
| PDFs | Yes, same price | – | Yes |
| JavaScript-only pages (SPAs) | No (flagged as `empty_content`, free) | Yes | Yes |

**When to choose a browser crawler instead:** the site renders its content only with JavaScript, needs a login, or sits behind heavy anti-bot protection. Those pages come back here as a free `empty_content` or `blocked` row, so testing costs you nothing.

### Input examples

Convert a few pages:

```json
{ "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }, { "url": "https://arxiv.org/pdf/2005.11401" }] }
```

Crawl a docs site through its sitemap plus links, with up to 500 pages and chunks as separate rows for a vector DB:

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "crawlMode": "sitemap+crawl",
  "maxPages": 500,
  "chunkSize": 400,
  "chunkOverlap": 40,
  "chunkOutput": "rows",
  "excludeUrlPatterns": ["**/changelog/**"]
}
```

Plain API usage (`urls` takes a string array):

```bash
curl -X POST "https://api.apify.com/v2/acts/atesen-software~url-to-markdown-rag/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H 'Content-Type: application/json' -d '{"urls":["https://example.com"],"chunkSize":0}'
```

### Output

Page row (charged), shortened:

```json
{
  "type": "page",
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "statusCode": 200,
  "contentType": "html",
  "title": "Retrieval-augmented generation - Wikipedia",
  "language": "en",
  "extraction": "article",
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique ...",
  "wordCount": 3740,
  "tokenCount": 5561,
  "contentHash": "5f0c…",
  "chunkCount": 17,
  "chunks": [
    { "index": 0, "tokens": 398, "headings": ["Retrieval-augmented generation"], "text": "# Retrieval-augmented generation\n\n..." },
    { "index": 1, "tokens": 377, "headings": ["Retrieval-augmented generation", "RAG and LLM limitations"], "text": "Retrieval-augmented generation > RAG and LLM limitations\n\n..." }
  ],
  "fetchMs": 412,
  "crawledAt": "2026-10-20T02:10:00.000Z"
}
```

Chunk rows (`chunkOutput: "rows"`, free) have the fields `chunkId`, `url`, `title`, `chunkIndex`, `chunkCount`, `tokens`, `headings`, `text` and `contentHash`. You can upsert them directly into Pinecone, Qdrant, Weaviate, pgvector, Chroma and similar stores.

Status rows (free) look like `{ "type": "status", "status": "http_error", "url": "...", "message": "HTTP 404. Not charged." }` with these statuses:

- `invalid_url`, `invalid_input`;
- `duplicate`, `duplicate_content`;
- `robots_disallowed`, `unreachable`, `blocked`, `http_error`, `too_large`;
- `unsupported_content_type`, `empty_content`;
- `skipped_pdf`, `pdf_error`, `no_sitemap_urls`.

The dataset has three views: **Pages**, **RAG chunks** and **Skipped & failed URLs**. A `RUN_SUMMARY` record in the key-value store holds the totals.

### Pricing

Pay per event: you pay only for **pages converted to Markdown**. A PDF document counts as one page.

| Apify plan | Price per 1,000 pages |
|---|---|
| Free / Starter | $1.20 |
| Scale | $1.08 |
| Business and higher | $0.96 |

Status rows, duplicates and chunk rows are free. Platform compute is minimal because there is no browser.

### Tips

- **Sitemap mode** is the fastest way to cover a docs site. **Sitemap + links** also catches pages missing from the sitemap.
- `maxPages` caps your cost exactly. A run stops cleanly when you reach your max charge per run.
- Set `chunkSize: 0` if you only need the Markdown.
- Some docs sites (e.g. Docusaurus/Mintlify with Markdown for agents) serve Markdown directly. It is passed through as `extraction: "markdown"`.
- Respect site terms. robots.txt is honored while crawling by default.

### FAQ

**Do I need a browser?** No. The Actor fetches HTTP pages at 512 MB. A page that returns no readable text is a failed row and is not charged.

**What is a RAG chunk?** The Markdown is split with `chunkSize` (default 512 tokens) so you can embed it. You pay for the page, not for each chunk.

**What does the saved example return?** It converts one public page to Markdown with chunks and a token count.

# Actor input Schema

## `startUrls` (type: `array`):

Pages, PDFs or website start URLs. Plain domains such as example.com work too. With crawl mode "Single pages" every URL is converted on its own; the other modes treat each URL as the start of a crawl.

## `crawlMode` (type: `string`):

<b>Single pages</b>: convert only the given URLs.<br><b>Sitemap</b>: read robots.txt/sitemap.xml and convert the pages under each start URL.<br><b>Follow links</b>: crawl links from the start URL.<br><b>Sitemap + links</b>: both, best coverage for docs sites.

## `maxPages` (type: `integer`):

Stop after this many converted pages (charged rows). Default: 100 when crawling, 1000 for single pages.

## `maxDepth` (type: `integer`):

How many clicks away from the start URL to follow links (Follow links modes only).

## `crawlScope` (type: `string`):

Which URLs a crawl may visit. <b>Path</b>: only URLs under the start URL (e.g. /docs/...). <b>Host</b>: the whole host. <b>Domain</b>: the host and its subdomains.

## `includeUrlPatterns` (type: `array`):

Optional. Globs (https://example.com/blog/\*\*), /regex/ or plain text. A crawled URL must match one of them.

## `excludeUrlPatterns` (type: `array`):

Optional. Globs, /regex/ or plain text, e.g. /tag/ or \**/login*.

## `respectRobotsTxt` (type: `boolean`):

Skip crawled URLs that robots.txt disallows (free status row). URLs you list yourself are always fetched.

## `chunkSize` (type: `integer`):

Split each page into RAG chunks of at most this many tokens (cl100k, the OpenAI tokenizer). Chunks follow headings, paragraphs and sentences; tables keep their header row. 0 = no chunking.

## `chunkOverlap` (type: `integer`):

Repeat up to this many tokens of trailing paragraphs at the start of the next chunk.

## `chunkOutput` (type: `string`):

<b>Inline</b>: chunks array inside each page row. <b>Rows</b>: one extra dataset row per chunk (free), ready for vector DB import. <b>Both</b>.

## `outputFormats` (type: `array`):

Markdown is always the main output. Add plain text and/or the cleaned HTML.

## `extractionMode` (type: `string`):

<b>Main content</b>: readability-style article extraction that drops menus, footers, cookie banners and ads. <b>Full page</b>: the whole body minus navigation and boilerplate.

## `includeImages` (type: `boolean`):

Keep ![alt](url) image references.

## `saveLinks` (type: `boolean`):

Add the list of absolute links found on each page.

## `includePdfs` (type: `boolean`):

Extract text from PDF URLs (same price per document). Off: PDFs get a free skipped status row.

## `maxPdfPages` (type: `integer`):

Read at most this many pages of each PDF.

## `minContentLength` (type: `integer`):

Pages with less text are returned as a free empty_content status row instead of a charged page (e.g. JavaScript-only apps, login walls).

## `dedupeContent` (type: `boolean`):

Skip pages whose Markdown is identical to a page already saved in this run (free status row). URL and canonical-URL duplicates are always skipped.

## `maxConcurrency` (type: `integer`):

Upper limit; it is lowered automatically for small memory sizes.

## `urls` (type: `array`):

Same as URLs, as a plain string array.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ],
  "crawlMode": "single",
  "maxDepth": 3,
  "crawlScope": "path",
  "respectRobotsTxt": true,
  "chunkSize": 512,
  "chunkOverlap": 50,
  "chunkOutput": "inline",
  "outputFormats": [
    "markdown"
  ],
  "extractionMode": "article",
  "includeImages": true,
  "saveLinks": false,
  "includePdfs": true,
  "maxPdfPages": 100,
  "minContentLength": 100,
  "dedupeContent": true,
  "maxConcurrency": 10
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `chunks` (type: `string`):

No description

## `problems` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("atesen-software/url-to-markdown-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }] }

# Run the Actor and wait for it to finish
run = client.actor("atesen-software/url-to-markdown-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ]
}' |
apify call atesen-software/url-to-markdown-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,atesen-software/url-to-markdown-rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/9M8DXF6mhREKS4St5/builds/pLYa0Y3RtWTLfpaCd/openapi.json