URL to Markdown for RAG: Pages, PDFs, Sitemaps to Clean Chunks

Turn web pages, PDFs, sitemaps or whole docs sites into clean Markdown with RAG-ready chunks. No headless browser, so it runs at 512 MB, converts a 200-page docs site in about 2.5 minutes and a 10-URL batch in seconds. It costs $1.20 per 1,000 pages. URLs that fail are free.

Use it to load knowledge bases, docs and blogs into a vector database. It also feeds LLM agents, custom GPTs and fine-tuning datasets.

What you get per page

• Markdown of the main content. Menus, footers, cookie banners, edit links and citation markers are removed, and tables, code blocks and headings are kept.
• Chunks of at most chunkSize tokens (default 512, OpenAI cl100k tokenizer):
  • split along headings, then paragraphs, then sentences;
  • each chunk carries its heading path (Guide > Install > Linux) as context;
  • large tables are split by rows and the header row is repeated in every chunk;
  • code blocks are never cut in the middle when they fit;
  • optional overlap between chunks.
• Token count, word count and content hash (SHA-1), so you can budget embeddings and detect changes.
• Metadata:
  • title, description, author, published and modified dates, language and site name;
  • Open Graph image, canonical URL and noindex flag;
  • HTTP status, final URL after redirects and crawl depth.
• PDF text: PDFs linked or listed get the same treatment, including page count, title and chunks. Plain-text and Markdown files are passed through.

Why this instead of a browser-based crawler

	This actor	RAG Web Browser	Website Content Crawler
Price	$1.20 / 1,000 pages, lower on paid plans. Failures are free.	$2.28 / 1,000 fetches on the Free plan, plus a charge per search	Free actor, but you pay for compute: browser runs with 8 GB default memory
Default memory	512 MB (runs on 256 MB)	8 GB	8 GB
Speed	HTTP-only, parallel: 10 URLs in ~6 s, 200-page docs site in ~2.5 min	Browser per page	Browser by default
Chunks for RAG	Built in: token-sized, heading path, table header repeated	No (whole page Markdown)	No (whole page text/Markdown)
Token count per page/chunk	Yes	No	No
Duplicate handling	URL, canonical URL and identical-content dedupe (free rows)	–	URL
Failed / blocked / non-HTML URLs	Free status row, run still succeeds	–	Compute still used
PDFs	Yes, same price	–	Yes
JavaScript-only pages (SPAs)	No (flagged as empty_content, free)	Yes	Yes

When to choose a browser crawler instead: the site renders its content only with JavaScript, needs a login, or sits behind heavy anti-bot protection. Those pages come back here as a free empty_content or blocked row, so testing costs you nothing.

Input examples

Convert a few pages:

{ "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }, { "url": "https://arxiv.org/pdf/2005.11401" }] }

Crawl a docs site through its sitemap plus links, with up to 500 pages and chunks as separate rows for a vector DB:

{
  "startUrls": [{ "url": "https://docs.apify.com/platform" }],
  "crawlMode": "sitemap+crawl",
  "maxPages": 500,
  "chunkSize": 400,
  "chunkOverlap": 40,
  "chunkOutput": "rows",
  "excludeUrlPatterns": ["**/changelog/**"]
}

Plain API usage (urls takes a string array):

curl -X POST "https://api.apify.com/v2/acts/atesen-software~url-to-markdown-rag/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H 'Content-Type: application/json' -d '{"urls":["https://example.com"],"chunkSize":0}'

Output

Page row (charged), shortened:

{
  "type": "page",
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "statusCode": 200,
  "contentType": "html",
  "title": "Retrieval-augmented generation - Wikipedia",
  "language": "en",
  "extraction": "article",
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique ...",
  "wordCount": 3740,
  "tokenCount": 5561,
  "contentHash": "5f0c…",
  "chunkCount": 17,
  "chunks": [
    { "index": 0, "tokens": 398, "headings": ["Retrieval-augmented generation"], "text": "# Retrieval-augmented generation\n\n..." },
    { "index": 1, "tokens": 377, "headings": ["Retrieval-augmented generation", "RAG and LLM limitations"], "text": "Retrieval-augmented generation > RAG and LLM limitations\n\n..." }
  ],
  "fetchMs": 412,
  "crawledAt": "2026-10-20T02:10:00.000Z"
}

Chunk rows (chunkOutput: "rows", free) have the fields chunkId, url, title, chunkIndex, chunkCount, tokens, headings, text and contentHash. You can upsert them directly into Pinecone, Qdrant, Weaviate, pgvector, Chroma and similar stores.

Status rows (free) look like { "type": "status", "status": "http_error", "url": "...", "message": "HTTP 404. Not charged." } with these statuses:

• invalid_url, invalid_input;
• duplicate, duplicate_content;
• robots_disallowed, unreachable, blocked, http_error, too_large;
• unsupported_content_type, empty_content;
• skipped_pdf, pdf_error, no_sitemap_urls.

The dataset has three views: Pages, RAG chunks and Skipped & failed URLs. A RUN_SUMMARY record in the key-value store holds the totals.

Pricing

Pay per event: you pay only for pages converted to Markdown. A PDF document counts as one page.

Apify plan	Price per 1,000 pages
Free / Starter	$1.20
Scale	$1.08
Business and higher	$0.96

Status rows, duplicates and chunk rows are free. Platform compute is minimal because there is no browser.

Tips

• Sitemap mode is the fastest way to cover a docs site. Sitemap + links also catches pages missing from the sitemap.
• maxPages caps your cost exactly. A run stops cleanly when you reach your max charge per run.
• Set chunkSize: 0 if you only need the Markdown.
• Some docs sites (e.g. Docusaurus/Mintlify with Markdown for agents) serve Markdown directly. It is passed through as extraction: "markdown".
• Respect site terms. robots.txt is honored while crawling by default.

FAQ

Do I need a browser? No. The Actor fetches HTTP pages at 512 MB. A page that returns no readable text is a failed row and is not charged.

What is a RAG chunk? The Markdown is split with chunkSize (default 512 tokens) so you can embed it. You pay for the page, not for each chunk.

What does the saved example return? It converts one public page to Markdown with chunks and a token count.