URL to Markdown Converter for RAG: Pages, PDFs & Sitemaps
Pricing
from $0.96 / 1,000 page converteds
URL to Markdown Converter for RAG: Pages, PDFs & Sitemaps
Convert any URL, sitemap or docs site to clean Markdown with RAG-ready chunks: heading paths, token counts and metadata. Reads PDFs too. No headless browser, 512 MB. $1.20 per 1,000 pages; failed, blocked and duplicate URLs are free.
Pricing
from $0.96 / 1,000 page converteds
Rating
0.0
(0)
Developer
Atesen Software
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
3 days ago
Last modified
Categories
Features
Share
Converts web pages, PDFs, sitemaps and docs sites to clean Markdown with RAG-ready chunks as JSON, CSV or Excel. For AI and RAG developers; pay per result: $1.20 per 1,000 pages (see the Pricing tab for the current price).
Convert any URL to Markdown: web pages, PDFs, sitemaps or whole docs sites become clean Markdown with RAG-ready chunks (heading path, token count, metadata). No headless browser, so it runs at 512 MB, converts a 200-page docs site in about 2.5 minutes and a 10-URL batch in seconds. It costs $1.20 per 1,000 pages. URLs that fail are free.
Use it to load knowledge bases, docs and blogs into a vector database. It also feeds LLM agents, custom GPTs and fine-tuning datasets.
What you get per page
- Markdown of the main content. Menus, footers, cookie banners, edit links and citation markers are removed, and tables, code blocks and headings are kept.
- Chunks of at most
chunkSizetokens (default 512, OpenAI cl100k tokenizer):- split along headings, then paragraphs, then sentences;
- each chunk carries its heading path (
Guide > Install > Linux) as context; - large tables are split by rows and the header row is repeated in every chunk;
- code blocks are never cut in the middle when they fit;
- optional overlap between chunks.
- Token count, word count and content hash (SHA-1), so you can budget embeddings and detect changes.
- Metadata:
- title, description, author, published and modified dates, language and site name;
- Open Graph image, canonical URL and noindex flag;
- HTTP status, final URL after redirects and crawl depth.
- PDF text: PDFs linked or listed get the same treatment, including page count, title and chunks. Plain-text and Markdown files are passed through.
Why this instead of a browser-based crawler
| This actor | RAG Web Browser | Website Content Crawler | |
|---|---|---|---|
| Price | $1.20 / 1,000 pages, lower on paid plans. Failures are free. | $2.28 / 1,000 fetches on the Free plan, plus a charge per search | Free actor, but you pay for compute: browser runs with 8 GB default memory |
| Default memory | 512 MB (runs on 256 MB) | 8 GB | 8 GB |
| Speed | HTTP-only, parallel: 10 URLs in ~6 s, 200-page docs site in ~2.5 min | Browser per page | Browser by default |
| Chunks for RAG | Built in: token-sized, heading path, table header repeated | No (whole page Markdown) | No (whole page text/Markdown) |
| Token count per page/chunk | Yes | No | No |
| Duplicate handling | URL, canonical URL and identical-content dedupe (free rows) | – | URL |
| Failed / blocked / non-HTML URLs | Free status row, run still succeeds | – | Compute still used |
| PDFs | Yes, same price | – | Yes |
| JavaScript-only pages (SPAs) | No (flagged as empty_content, free) | Yes | Yes |
When to choose a browser crawler instead: the site renders its content only with JavaScript, needs a login, or sits behind heavy anti-bot protection. Those pages come back here as a free empty_content or blocked row, so testing costs you nothing.
Input examples
Convert a few pages:
{ "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }, { "url": "https://arxiv.org/pdf/2005.11401" }] }
Crawl a docs site through its sitemap plus links, with up to 500 pages and chunks as separate rows for a vector DB:
{"startUrls": [{ "url": "https://docs.apify.com/platform" }],"crawlMode": "sitemap+crawl","maxPages": 500,"chunkSize": 400,"chunkOverlap": 40,"chunkOutput": "rows","excludeUrlPatterns": ["**/changelog/**"]}
Plain API usage (urls takes a string array):
curl -X POST "https://api.apify.com/v2/acts/atesen-software~url-to-markdown-rag/run-sync-get-dataset-items" \-H "Authorization: Bearer $APIFY_TOKEN" \-H 'Content-Type: application/json' -d '{"urls":["https://example.com"],"chunkSize":0}'
Output
Page row (charged), shortened:
{"type": "page","url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation","statusCode": 200,"contentType": "html","title": "Retrieval-augmented generation - Wikipedia","language": "en","extraction": "article","markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique ...","wordCount": 3740,"tokenCount": 5561,"contentHash": "5f0c…","chunkCount": 17,"chunks": [{ "index": 0, "tokens": 398, "headings": ["Retrieval-augmented generation"], "text": "# Retrieval-augmented generation\n\n..." },{ "index": 1, "tokens": 377, "headings": ["Retrieval-augmented generation", "RAG and LLM limitations"], "text": "Retrieval-augmented generation > RAG and LLM limitations\n\n..." }],"fetchMs": 412,"crawledAt": "2026-10-20T02:10:00.000Z"}
Chunk rows (chunkOutput: "rows", free) have the fields chunkId, url, title, chunkIndex, chunkCount, tokens, headings, text and contentHash. You can upsert them directly into Pinecone, Qdrant, Weaviate, pgvector, Chroma and similar stores.
Status rows (free) look like { "type": "status", "status": "http_error", "url": "...", "message": "HTTP 404. Not charged." } with these statuses:
invalid_url,invalid_input;duplicate,duplicate_content;robots_disallowed,unreachable,blocked,http_error,too_large;unsupported_content_type,empty_content;skipped_pdf,pdf_error,no_sitemap_urls.
The dataset has three views: Pages, RAG chunks and Skipped & failed URLs. A RUN_SUMMARY record in the key-value store holds the totals.
Pricing
Pay per event: you pay only for pages converted to Markdown. A PDF document counts as one page.
| Apify plan | Price per 1,000 pages |
|---|---|
| Free / Starter | $1.20 |
| Scale | $1.08 |
| Business and higher | $0.96 |
Status rows, duplicates and chunk rows are free. Platform compute is minimal because there is no browser.
Tips
- Sitemap mode is the fastest way to cover a docs site. Sitemap + links also catches pages missing from the sitemap.
maxPagescaps your cost exactly. A run stops cleanly when you reach your max charge per run.- Set
chunkSize: 0if you only need the Markdown. - Some docs sites (e.g. Docusaurus/Mintlify with Markdown for agents) serve Markdown directly. It is passed through as
extraction: "markdown". - Respect site terms. robots.txt is honored while crawling by default.
Use via API
One request runs the Actor, waits for it and returns the dataset items as JSON. Send your token in the Authorization header, not in the URL:
curl -X POST "https://api.apify.com/v2/acts/atesen-software~url-to-markdown-rag/run-sync-get-dataset-items?timeout=120" \-H "Authorization: Bearer $APIFY_TOKEN" \-H "Content-Type: application/json" \-d '{"startUrls":[{"url":"https://books.toscrape.com"}],"maxPages":1,"crawlMode":"single"}'
The synchronous endpoint waits up to 300 seconds. For longer jobs start a run with POST https://api.apify.com/v2/acts/atesen-software~url-to-markdown-rag/runs and read the dataset when it finishes.
JavaScript (npm install apify-client):
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: process.env.APIFY_TOKEN });const run = await client.actor('atesen-software/url-to-markdown-rag').call({ startUrls: [{url: "https://books.toscrape.com"}], maxPages: 1, crawlMode: "single" });const { items } = await client.dataset(run.defaultDatasetId).listItems();console.log(items);
Use with AI agents (MCP)
Claude Desktop, Cursor, VS Code and other MCP clients can call this Actor as a tool through the Apify MCP server. Load only this Actor with:
https://mcp.apify.com?tools=atesen-software/url-to-markdown-rag
Client config (sign in with Apify OAuth when the browser opens, or add an Authorization: Bearer <APIFY_TOKEN> header instead):
{"mcpServers": {"apify": { "url": "https://mcp.apify.com?tools=atesen-software/url-to-markdown-rag" }}}
Then ask in plain words, for example: "Convert https://books.toscrape.com to clean Markdown so I can add it to my RAG index." The agent fills the same input fields you see on the Input tab, and runs are billed to your Apify account at the prices on the Pricing tab.
FAQ
What does URL to Markdown Converter return? One row per page with clean Markdown, title, language, word and token counts, a content hash and RAG chunks with heading paths. Chunks can also come as separate rows that you can upsert into a vector database. Output is JSON, CSV or Excel.
How much does it cost? Pay per result: $1.20 per 1,000 pages converted, $1.08 on the Scale plan and $0.96 on Business and higher (see the Pricing tab for the current prices). Failed URLs, duplicates, status rows and chunk rows are free, and maxPages caps your cost.
Do I need a login or proxy? No. The Actor fetches pages over plain HTTP with no browser, no login and no proxy, and runs at 512 MB.
Does it respect robots.txt? Yes. robots.txt is honored while crawling by default, and a URL it disallows becomes a free robots_disallowed row. You are still responsible for following each site's terms.
Do I need a browser? No. The Actor fetches HTTP pages at 512 MB. A page that returns no readable text is a failed row and is not charged.
What is a RAG chunk? The Markdown is split with chunkSize (default 512 tokens) so you can embed it. You pay for the page, not for each chunk.
What does the saved example return? It converts one public page to Markdown with chunks and a token count.