Website to Markdown Crawler for AI, RAG and LLMs
Pricing
Pay per event + usage
Go to Apify Store
Website to Markdown Crawler for AI, RAG and LLMs
Crawl any public website into clean Markdown or text for AI chatbots, RAG and LLM training. Fast browserless crawler, sitemaps, robots.txt, change-only re-crawls. Flat price per page.
Pricing
Pay per event + usage
Rating
0.0
(0)
Developer
Musedenny
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
9 hours ago
Last modified
Categories
Share
Turn any public website into clean Markdown or plain text for AI chatbots, RAG pipelines, vector databases and LLM fine-tuning, at a flat, predictable price per page.
- Fast and lightweight: a browserless HTTP crawler. 18 Python docs pages (44,000 words) took 4.6 seconds in testing.
- Clean output: keeps headings, lists, tables, code and links in the main content; drops menus, headers, footers, cookie banners, forms and scripts. Add your own CSS selectors to drop more.
- Finds pages for you: follows links within your chosen scope (whole site, subdomain, path, or just the URLs you give) and reads
sitemap.xml. - Polite: respects robots.txt by default, retries with backoff, identifies itself.
- Change-only re-crawls: schedule it and get only pages that are new or changed since the last run. Unchanged pages cost a fraction of a full page.
- Flags JavaScript-only pages (
likelyNeedsJavascript) instead of silently returning empty content.
Input
{"startUrls": ["https://docs.example.com/"],"crawlScope": "path-prefix","maxPages": 500,"outputFormat": "markdown","excludeUrlGlobs": ["*/changelog/*"],"onlyChangedPages": false}
| Field | Default | Notes |
|---|---|---|
startUrls | – | One or more sites or sections |
crawlScope | path-prefix | same-domain (incl. subdomains), same-subdomain, path-prefix, start-urls-only |
maxPages | 100 | Hard cap on pages fetched (and on cost) |
maxDepth | 5 | Clicks from the start URLs |
useSitemaps | true | |
includeUrlGlobs / excludeUrlGlobs | – | e.g. https://example.com/blog/*, *?page=* |
outputFormat | markdown | markdown, text, both |
removeSelectors | – | Extra CSS selectors to drop |
keepNavigationAndFooter | false | |
includeLinks | false | Links found on each page |
onlyChangedPages | false | Re-crawl mode |
respectRobotsTxt | true | |
maxConcurrency | 10 |
Output
{"url": "https://docs.python.org/3/tutorial/classes.html","title": "9. Classes — Python 3 documentation","description": null,"language": "en","canonicalUrl": "https://docs.python.org/3/tutorial/classes.html","depth": 1,"wordCount": 5961,"likelyNeedsJavascript": false,"contentHash": "…","changedSinceLastRun": null,"crawledAt": "2026-11-01T20:00:00Z","markdown": "# 9. Classes\n\nClasses provide a means of bundling data and functionality together…"}
Pricing (pay per event + usage)
| Event | Price | When |
|---|---|---|
apify-actor-start | $0.001 | Once per run |
page | $0.0005 | One page returned ($0.50 per 1,000 pages) |
unchanged-page | $0.0001 | Change-only mode: page re-checked, unchanged |
Plus Apify platform usage, which is small because there's no browser: a few hundredths of a cent per 100 pages.
Limitations
- Pages that render their content only with JavaScript are flagged rather than rendered.
- Only public pages; no logins.
- Use it on sites you're allowed to crawl.