URL to Markdown for LLMs, RAG and AI Agents
Pricing
from $1.00 / 1,000 converted pages
URL to Markdown for LLMs, RAG and AI Agents
Convert web pages to clean Markdown for LLMs, RAG pipelines and AI agents. Keeps headings, tables, code and links, and adds title, metadata and optional RAG chunks. Opens a browser only for JavaScript pages and respects robots.txt. Pay only for converted pages.
Pricing
from $1.00 / 1,000 converted pages
Rating
0.0
(0)
Developer
Tan Solutions
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
5 days ago
Last modified
Categories
Features
Share
Convert web pages to clean Markdown for LLMs, RAG pipelines and AI agents. Give it URLs; get back each page as Markdown with its title and metadata, ready for RAG pipelines, vector stores, AI agents and prompts. It reads plain HTML first and opens a real browser only for pages that need JavaScript. You pay only for converted pages: failed, blocked and skipped pages are free.
Run it from the API or on a schedule
Get the Markdown back in one call (replace <YOUR_API_TOKEN> with your Apify API token from Console, Settings, API & Integrations):
curl -X POST "https://api.apify.com/v2/acts/tan-solutions~url-to-markdown/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \-H "Content-Type: application/json" \-d '{"startUrls":[{"url":"https://docs.python.org/3/tutorial/introduction.html"}],"chunkSize":2000}'
Send {} (no input) to try it: the run converts one Apify docs page.
Convert the same pages again every day at 06:00 UTC, so your RAG index stays fresh, with an Apify schedule (Qxw1MitrgVkJQisiY is this Actor's ID):
curl -X POST "https://api.apify.com/v2/schedules?token=<YOUR_API_TOKEN>" \-H "Content-Type: application/json" \-d '{"name": "url-to-markdown-daily", "cronExpression": "0 6 * * *", "timezone": "UTC", "isEnabled": true, "isExclusive": true,"actions": [{"type": "RUN_ACTOR", "actorId": "Qxw1MitrgVkJQisiY","runInput": {"body": "{\"startUrls\":[{\"url\":\"https://docs.python.org/3/tutorial/introduction.html\"}],\"chunkSize\":2000}", "contentType": "application/json"}}]}'
In Apify Console the same is: Schedules, Create new, cron 0 6 * * *, add this Actor with your input.
What it does
- Whole pages as Markdown. By default ("Full page") it keeps the page body and drops menus, footers and cookie banners. Headings, lists, tables, code blocks, links and images stay. Choose "Article" for the main text only.
- Metadata per page: title, description, language, site name, published and modified time, canonical URL, image.
- Chunks for RAG: set a chunk size and get the Markdown split at headings and paragraphs, so no chunk cuts a sentence or a code block in half.
- Optional front matter (YAML) at the top of each page: title, url, description, language, date.
- Fast: plain HTTP first; a browser only when a page is an empty JavaScript shell.
- JSON, XML and plain-text URLs come back as fenced blocks or plain text instead of failing.
- Reliable: a timeout per page, retries on network errors, polite pacing per site, and a clear status and error code for every URL. Up to 1,000 URLs per run.
Who it is for
- Developers building RAG search, chatbots or knowledge bases from websites and docs.
- AI agents and MCP clients that need to read a web page as text.
- Anyone who wants a page saved as Markdown for notes, prompts, archives or diffing.
Quick start
- Paste one or more URLs into URLs.
- Click Start.
- Open the Output tab and download the Markdown (JSON, CSV, Excel), or read the dataset through the API.
Input
| Field | What it does | Default |
|---|---|---|
startUrls (URLs) | The pages to convert | https://example.com |
maxUrls (Max URLs) | Stop after this many URLs (1 to 1,000); the rest are listed as skipped, free | 100 |
extractMode (What to convert) | full = whole page body; article = main text only (smaller) | full |
renderMode (Rendering) | auto = browser only when needed; http = never; browser = every page | auto |
includeLinks / includeImages | Keep links and images in the Markdown | on / on |
frontMatter | Add YAML front matter to each page | off |
chunkSize | Split into chunks of about this many characters (500 to 100,000; 0 = off) | 0 |
timeoutSecs | Time limit per page (5 to 120 seconds) | 30 |
maxConcurrency | Pages converted at the same time (1 to 20) | 5 |
proxyConfiguration | Optional proxy for the whole run | off |
Example:
{"startUrls": [{ "url": "https://docs.apify.com/platform/actors" }, { "url": "https://go.dev/doc/tutorial/getting-started" }],"extractMode": "full","chunkSize": 2000}
Output example
One dataset item per URL, in input order. A real item (Markdown cut short here):
{"url": "https://go.dev/doc/tutorial/getting-started","finalUrl": "https://go.dev/doc/tutorial/getting-started","status": "ok","title": "Tutorial: Get started with Go - The Go Programming Language","description": null,"markdown": "1. [Documentation](https://go.dev/doc/)\n2. [Tutorials](https://go.dev/doc/tutorial/)\n3. [Tutorial: Get started with Go](https://go.dev/doc/tutorial/getting-started)\n\n# Tutorial: Get started with Go\n\nIn this tutorial, you'll get a brief introduction to Go programming. ...","wordCount": 1041,"chunks": null,"metadata": {"title": "Tutorial: Get started with Go - The Go Programming Language","lang": "en","canonicalUrl": "https://go.dev/doc/tutorial/getting-started","image": "https://go.dev/doc/gopher/gopher5logo.jpg"},"fetchedWith": "http","httpStatus": 200,"contentType": "text/html","crawledAt": "2026-11-13T07:49:36.789Z","error": null,"totalMs": 1073,"fetchMs": 1052,"convertMs": 20}
status is ok, robots_disallowed, blocked or error. With a chunk size set, chunks is a list of Markdown
strings. Failed items carry error: { "code": "...", "message": "..." }. Codes include ROBOTS_DISALLOWED,
BLOCKED, CAPTCHA_DETECTED, HTTP_ERROR, TIMEOUT, UNSUPPORTED_CONTENT_TYPE, EMPTY_CONTENT,
OUTPUT_TOO_LARGE, CONTENT_TOO_LARGE and MAX_PAGES_REACHED.
Pricing
You pay per converted page (the page event). Failed, blocked and robots-disallowed pages are free, and so are
skipped URLs. If your spending limit is reached, the run stops cleanly and tells you so.
| Apify plan | Price per 1,000 converted pages |
|---|---|
| Free | $1.50 |
| Bronze | $1.20 |
| Silver | $1.10 |
| Gold | $1.00 |
Apify's standard Actor start event is charged when a run starts, at the platform's default price: one event per GB of run memory (4 events at the default 4 GB).
Limits
- robots.txt is respected. Pages it disallows are returned as
robots_disallowedand are free. - Public pages only: no login, no cookies, no form filling, no paywalls.
- It stops at CAPTCHAs and bot walls (returned as
blocked, free). It does not solve CAPTCHAs or hide who it is. - Up to 1,000 URLs per run (default 100). URLs over "Max URLs" are listed as
MAX_PAGES_REACHEDand are free. - Page size: one item holds at most 5 million characters of Markdown (
OUTPUT_TOO_LARGE). Pages whose HTML is too big for the run memory come back asCONTENT_TOO_LARGE(free). Memory is 4 GB by default, 2 GB minimum. - The crawler identifies itself as
TanForgeBot/1.0 (+https://apify.com/tan-solutions). Site owners can allow or block it in robots.txt with theTanForgeBottoken.
FAQ
Can AI agents and MCP clients use it? Yes. Add it as a tool through Apify's MCP server, or call it through the
Apify API or SDKs: pass startUrls, then read the dataset items. Each item has the Markdown, title and URL.
Does it work on JavaScript pages? Yes. In auto mode, a page whose HTML has almost no text and looks like a single-page app is opened again in a real browser. Choose "Browser for every page" to always render.
Can it convert PDFs or Word files? Use our PDF, Word and PowerPoint to Markdown for LLMs and RAG Actor for documents. This Actor converts web pages (plus JSON, XML and plain text).
How fast is it, and how much at once? It converts 5 pages at the same time by default (up to 20 with "Max parallel pages") and paces requests to each site politely. A typical page takes about 3 seconds.
How long are my results kept? Results go to the run's dataset in your own Apify account and follow your plan's data retention. The Actor keeps no copy anywhere else.
Related tools by Tan Solutions
- Document to Markdown: PDF, Word and PowerPoint files to Markdown for LLMs and RAG.
- SEO Audit Tool: on-page SEO checker with PageSpeed for any list of pages.
- Domain Intelligence: DNS, WHOIS/RDAP, SSL and tech stack for up to 1,000 domains.
- US Company Filings & Financials: 10-K, 10-Q, 8-K filings and XBRL facts by ticker.
More from Tan Solutions
- URL to Markdown tool page: details and examples for this Actor on our site.
- Guide: URL to Markdown: converter, Python, CLI or API compared
- All Tan Solutions data tools
Your responsibilities
You are responsible for the URLs you submit and for lawful use of the text you get back (copyright, the site's terms, any AI-training restrictions the site states, and privacy law for any personal data in a page). The Actor obeys robots.txt, does not log in and stops at CAPTCHAs and bot walls. It is provided without warranty; liability is capped at the fees you paid in the last 12 months. Questions and issues are answered within 14 days through the Actor's Issues tab.
Not affiliated with OpenAI, Anthropic, LangChain or LlamaIndex. All product names belong to their owners.
Made by Tan Solutions.