Website Content Crawler: Markdown for AI (Half Price)
Pricing
Pay per usage
Website Content Crawler: Markdown for AI (Half Price)
Crawl any website and get each page's main content as clean Markdown and text for LLMs, RAG and search. Sitemap-seeded, de-duplicated, no nav or cookie banners. Respects robots.txt and AI-crawler opt-outs. $1.50 per 1,000 pages.
Pricing
Pay per usage
Rating
0.0
(0)
Developer
Unadhi
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
4 hours ago
Last modified
Categories
Share
Website Content Crawler: Clean Markdown for AI and RAG at Half Price
Give it one or more website URLs. The Actor crawls each site (sitemap first, then links, same site only) and returns one row per page: clean main-content Markdown (or plain text), title, description, language, headings, canonical URL, word count and a content hash.
Price: $1.50 per 1,000 pages (pay per event page). That is 50% of 6sigmag/fast-website-content-crawler ($3 per 1,000). Pages that are skipped, duplicated, too short or opted out are free.
Why it's better
| This Actor | |
|---|---|
| Price | $0.0015 per page, flat on every plan. No per-GB start fee |
| Clean content | Removes nav, footers, cookie banners and boilerplate, keeping the headings, lists, tables and code blocks. On 60 pages compared with trafilatura 2.2.0: precision 0.89, recall 0.96 |
| No duplicates | Canonical URLs plus content-hash dedupe, so you never pay twice for the same page |
| Respects AI opt-outs | Honours robots.txt (RFC 9309), including AI-bot groups such as GPTBot, noai meta tags and TDM-Rep (honorAiOptOut, on by default). Blocklisted platforms (LinkedIn, etc.) are never fetched |
| Control | maxPagesPerSite, maxDepth, stayUnderStartPath, include and exclude URL patterns, minWords |
| Spend limit | Stops cleanly at your max-charge limit |
Input
Output fields
url, site, depth, title, description, language, canonicalUrl, headings, wordCount, markdown, text, contentHash, httpStatus, fetchedAt. See test/sample_output.json for real rows.
Measured on 18 real sites (29 Sep 2026)
- 280 pages at 20 pages per site in 38.5 s wall time, 27.4 CPU-s and 230 MB peak memory.
- Correctly refused:
- nytimes.com and medium.com (their robots.txt disallows AI crawlers)
- linkedin.com (blocklist)
Use it responsibly
Crawl only content you have the right to use. The Actor never logs in and never gets around blocks.
Pricing
| Event | Price |
|---|---|
page (primary) | $0.0015 per page = $1.50 per 1,000 pages (all tiers) |
apify-actor-start | Apify default |
