Website to Markdown: Fast Crawler for LLM & RAG (No Browser) avatar

Website to Markdown: Fast Crawler for LLM & RAG (No Browser)

Pricing

from $0.70 / 1,000 page converteds

Go to Apify Store
Website to Markdown: Fast Crawler for LLM & RAG (No Browser)

Website to Markdown: Fast Crawler for LLM & RAG (No Browser)

Turn web pages into clean Markdown for LLMs, RAG and AI agents: main content only, menus and footers removed, can follow site links. $1 per 1,000 pages.

Pricing

from $0.70 / 1,000 page converteds

Rating

0.0

(0)

Developer

Open Data Tools

Open Data Tools

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

9 hours ago

Last modified

Features

Limited permissions

Share

Give page links, get clean Markdown ready for an LLM, a RAG index or an AI agent: headings, paragraphs, lists, links, tables and code, without the menus, headers, footers, sidebars and cookie banners. One page, or a whole documentation: the Actor can follow the links of a site under the same path. It reads pages directly, without a browser, so it is fast and cheap: $1 per 1,000 pages.

What you get

One row per page:

  • Markdown of the main content (or of the whole page if you prefer).
  • Title, description, language and canonical link.
  • Dates the page gives: published and modified, in ISO 8601.
  • Counts: words, links and images in the Markdown.
  • Depth of the page from the link you gave (0 for the page itself).

Use cases

  • RAG and chatbots: feed a documentation, a help center or a blog into a vector database.
  • AI agents: one call to read a page as text an LLM understands.
  • Content audits: word counts and structure of many pages at once.
  • Archives and monitoring: keep the text of pages that matter to you.

How to use

  • Without changing anything: the Apify documentation page about Actors.
  • Pages: paste links or bare domains, one per line, up to 10,000 per run.
  • Pages per link: 1 converts the page alone. More follows its links on the same site, under the same path: from https://example.com/docs the Actor reads /docs/... pages only, up to the number you choose.
  • Main content only, Keep links, Keep images: switches for shorter or fuller text.

Input example

{
"urls": ["https://docs.apify.com/platform/actors"],
"maxPagesPerUrl": 5,
"onlyMainContent": true,
"includeLinks": true,
"includeImages": false
}

Output example

{
"recordType": "page",
"id": "https://docs.apify.com/actors",
"url": "https://docs.apify.com/actors",
"startUrl": "https://docs.apify.com/platform/actors",
"depth": 0,
"title": "Actors | Platform | Apify Documentation",
"description": "Learn how to develop, run and share serverless cloud programs.",
"lang": "en",
"canonicalUrl": "https://docs.apify.com/actors",
"publishedAt": null,
"modifiedAt": null,
"markdown": "# Actors\n\nActors are serverless cloud programs that take a structured JSON input, perform a task…",
"wordCount": 360,
"linkCount": 16,
"imageCount": 0,
"status": "ok",
"sourceName": "Website docs.apify.com",
"sourceUrl": "https://docs.apify.com/actors",
"licence": "Text of a public page, as its owner publishes it; check the site terms before reuse",
"sourceUpdatedAt": null,
"actorVersion": "0.1.1",
"retrievedAt": "2026-11-17T21:00:00.000Z"
}

Pricing

Pay per event: $1.00 per 1,000 pages converted (page-converted, $0.001 each), plus Apify's start fee of $0.00005 per run. Pages that do not answer, are missing or are not web pages give a free error row.

PagesPrice
100$0.10
1,000$1
100,000$100

Data source

The pages you give, read directly as their server sends them. The Actor introduces itself as OpenDataToolsBot, follows robots.txt (RFC 9309), waits between two pages of the same site, and never logs in, solves no CAPTCHA and uses no proxy to get round a block.

Limit: pages that build their text with JavaScript after loading (some single-page apps) give little or no Markdown, since no browser runs here. Most sites, documentations, blogs and news pages send their text directly.

The Actor reads public pages, as each site's robots.txt allows. The text stays its owner's: check the site's terms before you republish it or use it to train a model.

FAQ

  • Why so few words on a page? The page may build its text with JavaScript, or the main part may be small. Try "Main content only" off.
  • Why did the crawl stop early? It reads links under the path you gave, on the same site, and skips pages that robots.txt closes, files (PDF, images) and pages that fail.
  • Can I get shorter text for an AI model? Turn "Keep links" off: links keep their text only.

Every Actor here also works as a tool for AI agents (Claude, ChatGPT, Cursor…) through Apify’s MCP server.

Changelog

See CHANGELOG.md.

En français

Pages web en Markdown propre, pour les IA

Donnez des liens de pages, obtenez leur texte en Markdown (titres, listes, liens, tableaux, code) sans les menus, en-têtes, pieds de page et bandeaux cookies. Le robot peut suivre les liens d'un site sous le même chemin (toute une documentation, par exemple).

Prix

1 $ les 1 000 pages. Les lignes d'erreur sont gratuites.

Source et légalité

Les pages publiques elles-mêmes, lues dans le respect de leur robots.txt, sans contournement. Le texte reste à son auteur : vérifiez les conditions du site avant de le republier.