URL to Markdown for LLMs, RAG and AI Agents avatar

URL to Markdown for LLMs, RAG and AI Agents

Pricing

from $1.00 / 1,000 converted pages

Go to Apify Store
URL to Markdown for LLMs, RAG and AI Agents

URL to Markdown for LLMs, RAG and AI Agents

Convert web pages to clean Markdown for LLMs, RAG pipelines and AI agents. Keeps headings, tables, code and links, and adds title, metadata and optional RAG chunks. Opens a browser only for JavaScript pages and respects robots.txt. Pay only for converted pages.

Pricing

from $1.00 / 1,000 converted pages

Rating

0.0

(0)

Developer

Tan Solutions

Tan Solutions

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

5 days ago

Last modified

Features

Limited permissions

Share

Convert web pages to clean Markdown for LLMs, RAG pipelines and AI agents. Give it URLs; get back each page as Markdown with its title and metadata, ready for RAG pipelines, vector stores, AI agents and prompts. It reads plain HTML first and opens a real browser only for pages that need JavaScript. You pay only for converted pages: failed, blocked and skipped pages are free.

Run it from the API or on a schedule

Get the Markdown back in one call (replace <YOUR_API_TOKEN> with your Apify API token from Console, Settings, API & Integrations):

curl -X POST "https://api.apify.com/v2/acts/tan-solutions~url-to-markdown/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
-H "Content-Type: application/json" \
-d '{"startUrls":[{"url":"https://docs.python.org/3/tutorial/introduction.html"}],"chunkSize":2000}'

Send {} (no input) to try it: the run converts one Apify docs page.

Convert the same pages again every day at 06:00 UTC, so your RAG index stays fresh, with an Apify schedule (Qxw1MitrgVkJQisiY is this Actor's ID):

curl -X POST "https://api.apify.com/v2/schedules?token=<YOUR_API_TOKEN>" \
-H "Content-Type: application/json" \
-d '{"name": "url-to-markdown-daily", "cronExpression": "0 6 * * *", "timezone": "UTC", "isEnabled": true, "isExclusive": true,
"actions": [{"type": "RUN_ACTOR", "actorId": "Qxw1MitrgVkJQisiY",
"runInput": {"body": "{\"startUrls\":[{\"url\":\"https://docs.python.org/3/tutorial/introduction.html\"}],\"chunkSize\":2000}", "contentType": "application/json"}}]}'

In Apify Console the same is: Schedules, Create new, cron 0 6 * * *, add this Actor with your input.

What it does

  • Whole pages as Markdown. By default ("Full page") it keeps the page body and drops menus, footers and cookie banners. Headings, lists, tables, code blocks, links and images stay. Choose "Article" for the main text only.
  • Metadata per page: title, description, language, site name, published and modified time, canonical URL, image.
  • Chunks for RAG: set a chunk size and get the Markdown split at headings and paragraphs, so no chunk cuts a sentence or a code block in half.
  • Optional front matter (YAML) at the top of each page: title, url, description, language, date.
  • Fast: plain HTTP first; a browser only when a page is an empty JavaScript shell.
  • JSON, XML and plain-text URLs come back as fenced blocks or plain text instead of failing.
  • Reliable: a timeout per page, retries on network errors, polite pacing per site, and a clear status and error code for every URL. Up to 1,000 URLs per run.

Who it is for

  • Developers building RAG search, chatbots or knowledge bases from websites and docs.
  • AI agents and MCP clients that need to read a web page as text.
  • Anyone who wants a page saved as Markdown for notes, prompts, archives or diffing.

Quick start

  1. Paste one or more URLs into URLs.
  2. Click Start.
  3. Open the Output tab and download the Markdown (JSON, CSV, Excel), or read the dataset through the API.

Input

FieldWhat it doesDefault
startUrls (URLs)The pages to converthttps://example.com
maxUrls (Max URLs)Stop after this many URLs (1 to 1,000); the rest are listed as skipped, free100
extractMode (What to convert)full = whole page body; article = main text only (smaller)full
renderMode (Rendering)auto = browser only when needed; http = never; browser = every pageauto
includeLinks / includeImagesKeep links and images in the Markdownon / on
frontMatterAdd YAML front matter to each pageoff
chunkSizeSplit into chunks of about this many characters (500 to 100,000; 0 = off)0
timeoutSecsTime limit per page (5 to 120 seconds)30
maxConcurrencyPages converted at the same time (1 to 20)5
proxyConfigurationOptional proxy for the whole runoff

Example:

{
"startUrls": [{ "url": "https://docs.apify.com/platform/actors" }, { "url": "https://go.dev/doc/tutorial/getting-started" }],
"extractMode": "full",
"chunkSize": 2000
}

Output example

One dataset item per URL, in input order. A real item (Markdown cut short here):

{
"url": "https://go.dev/doc/tutorial/getting-started",
"finalUrl": "https://go.dev/doc/tutorial/getting-started",
"status": "ok",
"title": "Tutorial: Get started with Go - The Go Programming Language",
"description": null,
"markdown": "1. [Documentation](https://go.dev/doc/)\n2. [Tutorials](https://go.dev/doc/tutorial/)\n3. [Tutorial: Get started with Go](https://go.dev/doc/tutorial/getting-started)\n\n# Tutorial: Get started with Go\n\nIn this tutorial, you'll get a brief introduction to Go programming. ...",
"wordCount": 1041,
"chunks": null,
"metadata": {
"title": "Tutorial: Get started with Go - The Go Programming Language",
"lang": "en",
"canonicalUrl": "https://go.dev/doc/tutorial/getting-started",
"image": "https://go.dev/doc/gopher/gopher5logo.jpg"
},
"fetchedWith": "http",
"httpStatus": 200,
"contentType": "text/html",
"crawledAt": "2026-11-13T07:49:36.789Z",
"error": null,
"totalMs": 1073,
"fetchMs": 1052,
"convertMs": 20
}

status is ok, robots_disallowed, blocked or error. With a chunk size set, chunks is a list of Markdown strings. Failed items carry error: { "code": "...", "message": "..." }. Codes include ROBOTS_DISALLOWED, BLOCKED, CAPTCHA_DETECTED, HTTP_ERROR, TIMEOUT, UNSUPPORTED_CONTENT_TYPE, EMPTY_CONTENT, OUTPUT_TOO_LARGE, CONTENT_TOO_LARGE and MAX_PAGES_REACHED.

Pricing

You pay per converted page (the page event). Failed, blocked and robots-disallowed pages are free, and so are skipped URLs. If your spending limit is reached, the run stops cleanly and tells you so.

Apify planPrice per 1,000 converted pages
Free$1.50
Bronze$1.20
Silver$1.10
Gold$1.00

Apify's standard Actor start event is charged when a run starts, at the platform's default price: one event per GB of run memory (4 events at the default 4 GB).

Limits

  • robots.txt is respected. Pages it disallows are returned as robots_disallowed and are free.
  • Public pages only: no login, no cookies, no form filling, no paywalls.
  • It stops at CAPTCHAs and bot walls (returned as blocked, free). It does not solve CAPTCHAs or hide who it is.
  • Up to 1,000 URLs per run (default 100). URLs over "Max URLs" are listed as MAX_PAGES_REACHED and are free.
  • Page size: one item holds at most 5 million characters of Markdown (OUTPUT_TOO_LARGE). Pages whose HTML is too big for the run memory come back as CONTENT_TOO_LARGE (free). Memory is 4 GB by default, 2 GB minimum.
  • The crawler identifies itself as TanForgeBot/1.0 (+https://apify.com/tan-solutions). Site owners can allow or block it in robots.txt with the TanForgeBot token.

FAQ

Can AI agents and MCP clients use it? Yes. Add it as a tool through Apify's MCP server, or call it through the Apify API or SDKs: pass startUrls, then read the dataset items. Each item has the Markdown, title and URL.

Does it work on JavaScript pages? Yes. In auto mode, a page whose HTML has almost no text and looks like a single-page app is opened again in a real browser. Choose "Browser for every page" to always render.

Can it convert PDFs or Word files? Use our PDF, Word and PowerPoint to Markdown for LLMs and RAG Actor for documents. This Actor converts web pages (plus JSON, XML and plain text).

How fast is it, and how much at once? It converts 5 pages at the same time by default (up to 20 with "Max parallel pages") and paces requests to each site politely. A typical page takes about 3 seconds.

How long are my results kept? Results go to the run's dataset in your own Apify account and follow your plan's data retention. The Actor keeps no copy anywhere else.

More from Tan Solutions

Your responsibilities

You are responsible for the URLs you submit and for lawful use of the text you get back (copyright, the site's terms, any AI-training restrictions the site states, and privacy law for any personal data in a page). The Actor obeys robots.txt, does not log in and stops at CAPTCHAs and bot walls. It is provided without warranty; liability is capped at the fees you paid in the last 12 months. Questions and issues are answered within 14 days through the Actor's Issues tab.

Not affiliated with OpenAI, Anthropic, LangChain or LlamaIndex. All product names belong to their owners.

Made by Tan Solutions.