Website to Markdown Crawler for AI, RAG and LLMs avatar

Website to Markdown Crawler for AI, RAG and LLMs

Pricing

Pay per event + usage

Go to Apify Store
Website to Markdown Crawler for AI, RAG and LLMs

Website to Markdown Crawler for AI, RAG and LLMs

Crawl any public website into clean Markdown or text for AI chatbots, RAG and LLM training. Fast browserless crawler, sitemaps, robots.txt, change-only re-crawls. Flat price per page.

Pricing

Pay per event + usage

Rating

0.0

(0)

Developer

Musedenny

Musedenny

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

9 hours ago

Last modified

Categories

Share

Turn any public website into clean Markdown or plain text for AI chatbots, RAG pipelines, vector databases and LLM fine-tuning, at a flat, predictable price per page.

  • Fast and lightweight: a browserless HTTP crawler. 18 Python docs pages (44,000 words) took 4.6 seconds in testing.
  • Clean output: keeps headings, lists, tables, code and links in the main content; drops menus, headers, footers, cookie banners, forms and scripts. Add your own CSS selectors to drop more.
  • Finds pages for you: follows links within your chosen scope (whole site, subdomain, path, or just the URLs you give) and reads sitemap.xml.
  • Polite: respects robots.txt by default, retries with backoff, identifies itself.
  • Change-only re-crawls: schedule it and get only pages that are new or changed since the last run. Unchanged pages cost a fraction of a full page.
  • Flags JavaScript-only pages (likelyNeedsJavascript) instead of silently returning empty content.

Input

{
"startUrls": ["https://docs.example.com/"],
"crawlScope": "path-prefix",
"maxPages": 500,
"outputFormat": "markdown",
"excludeUrlGlobs": ["*/changelog/*"],
"onlyChangedPages": false
}
FieldDefaultNotes
startUrls–One or more sites or sections
crawlScopepath-prefixsame-domain (incl. subdomains), same-subdomain, path-prefix, start-urls-only
maxPages100Hard cap on pages fetched (and on cost)
maxDepth5Clicks from the start URLs
useSitemapstrue
includeUrlGlobs / excludeUrlGlobs–e.g. https://example.com/blog/*, *?page=*
outputFormatmarkdownmarkdown, text, both
removeSelectors–Extra CSS selectors to drop
keepNavigationAndFooterfalse
includeLinksfalseLinks found on each page
onlyChangedPagesfalseRe-crawl mode
respectRobotsTxttrue
maxConcurrency10

Output

{
"url": "https://docs.python.org/3/tutorial/classes.html",
"title": "9. Classes — Python 3 documentation",
"description": null,
"language": "en",
"canonicalUrl": "https://docs.python.org/3/tutorial/classes.html",
"depth": 1,
"wordCount": 5961,
"likelyNeedsJavascript": false,
"contentHash": "…",
"changedSinceLastRun": null,
"crawledAt": "2026-11-01T20:00:00Z",
"markdown": "# 9. Classes\n\nClasses provide a means of bundling data and functionality together…"
}

Pricing (pay per event + usage)

EventPriceWhen
apify-actor-start$0.001Once per run
page$0.0005One page returned ($0.50 per 1,000 pages)
unchanged-page$0.0001Change-only mode: page re-checked, unchanged

Plus Apify platform usage, which is small because there's no browser: a few hundredths of a cent per 100 pages.

Limitations

  • Pages that render their content only with JavaScript are flagged rather than rendered.
  • Only public pages; no logins.
  • Use it on sites you're allowed to crawl.