Website Content Crawler for AI - Markdown & RAG Chunks
Pricing
from $0.13 / 1,000 pages
Website Content Crawler for AI - Markdown & RAG Chunks
Crawl a website to clean Markdown, text and token-counted RAG chunks for LLMs, vector DBs, n8n and Make. Sitemaps, robots.txt, JS pages, linked PDF/DOCX, llms.txt, change-only re-crawls. Flat price per page.
Pricing
from $0.13 / 1,000 pages
Rating
0.0
(0)
Developer
Kanto Labs
Maintained by CommunityActor stats
0
Bookmarked
3
Total users
2
Monthly active users
13 days ago
Last modified
Categories
Features
Share
What does Website Content Crawler for AI do?
Website Content Crawler for AI is a website content crawler that turns a whole website into clean Markdown, plain text and RAG-ready chunks for LLMs, vector databases, chatbots and AI agents. Give it the start of a docs site, blog, help centre or company site and it follows the links and the sitemap, keeps the main content of every page (navigation, headers, footers, sidebars, cookie banners, ads and share buttons are removed), and returns one row per page with the Markdown, text, title, description, language, a content hash and heading-aware chunks with token counts.
You pay a flat price per page, so you know the cost before you press Start.
Why use this website content crawler?
- Flat, predictable price per page. No compute-unit maths, no surprise bills from slow pages, retries or a big memory setting. $0.15 per 1,000 pages (plain HTTP), $0.90 per 1,000 pages that need a headless browser.
- RAG-ready output. Every page comes pre-split into heading-aware chunks (default 512 tokens, 64 tokens overlap, OpenAI
cl100k_basetoken counts). Each chunk carries the page URL, title and heading path (["Guide", "Install", "Windows"]), so you can embed it straight away. - Linked documents too. Turn on Convert linked documents and the PDF, DOCX, PPTX and XLSX files linked from the pages become Markdown + chunks as well.
llms.txtand the whole site in one file. Each run writesllms.txt(a page index),llms-full.txt(every page's Markdown in one file) andchunks.jsonl(all chunks, one per line) to the key-value store.- Cheap re-crawls: only changed pages. Point a new run at the dataset of an earlier run: pages that did not change (same content hash, or
304 Not Modified) come back as freeunchangedrows. You pay only for new and changed pages. - Duplicates removed for free. Same canonical URL, identical text or near-identical text (SimHash: print views, tracking-parameter and sort variants) become free
duplicaterows that point to the first copy. - JavaScript sites. Fast plain HTTP by default; pages that only show content after JavaScript runs are automatically re-rendered in headless Chromium.
- Polite and safe. robots.txt is respected by default (with
Crawl-delay), the crawler identifies itself asKantoCrawler, and failed, blocked, robots-disallowed, duplicate and unchanged pages are never charged. Bot walls, CAPTCHAs and logins are never bypassed.
How to crawl a website to Markdown
- Put the start of the site in Start URLs, e.g.
https://docs.example.com/(pages under the same path are followed). - Set Max pages (and, if you want, include / exclude URL globs and a max depth).
- Keep Crawler type = auto (HTTP, browser only when needed). Choose http for the lowest price or browser for app-like sites.
- Optional: Convert linked documents, a different chunk size, or Only changed pages since this dataset for a re-crawl.
- Click Start. Pages appear in the Pages view; the OUTPUT record links
llms.txt,llms-full.txtandchunks.jsonl.
Input example
{"startUrls": [{ "url": "https://docs.example.com/" }],"maxPages": 500,"crawlerType": "auto","excludeUrlGlobs": ["https://docs.example.com/**/changelog/**"],"chunkSize": 512,"chunkOverlap": 64,"convertLinkedDocuments": true}
Output example
One row per page (Markdown, text and chunks shortened):
{"url": "https://www.joelonsoftware.com/2018/04/13/gamification/","status": "ok","type": "page","renderedWith": "http","httpStatus": 200,"depth": 1,"title": "A Dusting of Gamification – Joel on Software","description": "I had to think for a minute to realize that Stack Overflow has \"gamification\" too...","language": "en-US","publishedAt": "2018-04-13T13:40:21+00:00","markdown": "# A Dusting of Gamification\n\nAround 2010 the success of Stack Overflow had led us into some conversations with VCs...","text": "A Dusting of Gamification\n\nAround 2010 the success of Stack Overflow...","wordCount": 1235,"tokenCount": 1690,"contentHash": "bb8a180d7aac4889f277c95b4f07b695dea90850d9f8dd13ad94a59f93e09289","simHash": "9f1c...","chunkCount": 6,"chunks": [{"chunkId": "198844c52f-2","index": 2,"url": "https://www.joelonsoftware.com/2018/04/13/gamification/","title": "A Dusting of Gamification – Joel on Software","headingPath": ["A Dusting of Gamification"],"text": "We didn't want to punish you so much as we wanted to show other people that your answer was wrong...","tokenCount": 424}],"billedEvent": "page","billedUnits": 1,"error": null}
Linked documents are rows with "type": "document" plus fileName, format and pageCount. Free rows have status = failed, blocked, robots-disallowed, duplicate, unchanged or skipped, an empty markdown and a plain-language error / duplicateOf. Filter on status = "ok" to get the content.
How much does it cost to crawl a website?
| Event | Price (Free / Bronze) | Silver | Gold | When |
|---|---|---|---|---|
| Page | $0.00015 | $0.000135 | $0.00013 | a page delivered over plain HTTP |
| Browser page | $0.0009 | $0.00081 | $0.00078 | a page that needed headless Chromium |
| Document | $0.002 | $0.0018 | $0.0017 | a linked PDF / DOCX / PPTX / XLSX converted |
Plus the standard Apify start fee ($0.00005 per GB of memory, $0.0001 at the default 2 GB). One event covers up to 25,000 tokens of Markdown (about 18,000 words, longer than nearly every web page); a longer page or file counts one event per started 25,000 tokens. Failed, blocked, robots-disallowed, duplicate and unchanged pages cost $0.
Why a flat price beats usage billing (worked example)
You crawl a 1,000-page documentation site once a week.
- Here: 1,000 pages x $0.00015 = $0.15 per crawl, the same every week. Next week, with Only changed pages, the 950 pages that did not change are free: about $0.01.
- Usage-billed crawlers charge for compute time instead (memory x run time). Their own estimates for raw-HTTP crawling are around $0.2 per 1,000 pages, and $0.5 to $5 per 1,000 pages with a headless browser. The real bill moves with page weight, slow servers, retries and the memory you pick, so you only know it after the run.
Set Max cost per run in the run options and the crawler stops cleanly when the budget is reached.
Use it with n8n, Make, LangChain and LlamaIndex
- n8n / Make / Zapier: run the Actor (or a saved task) with the Apify integration, then loop over the dataset items where
statusisokand sendmarkdownor each item ofchunksto your vector store, Google Sheet or LLM step. - LangChain (Python):
from langchain_apify import ApifyWrapperfrom langchain_core.documents import Documentloader = ApifyWrapper().call_actor(actor_id="kantolabs/website-content-crawler-ai",run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200},dataset_mapping_function=lambda item: Document(page_content=item["markdown"], metadata={"source": item["url"], "title": item["title"]}),)docs = [d for d in loader.load() if d.page_content]
- LlamaIndex: use
ApifyActorfromllama-index-readers-apifywith the sameactor_idand mapitem["markdown"](or each chunk'stext) to aDocument. - Plain API / agents:
chunks.jsonlfrom the OUTPUT record is one chunk per line withurl,headingPath,textandtokenCount, ready to embed.
Crawl scope, sitemaps and robots.txt
- Scope: by default only pages on the same host and under the start URL's folder are followed (
https://example.com/blog/crawls/blog/...). Choose Same host or Same domain and subdomains, or set Include URLs (globs) such ashttps://example.com/docs/**. - Sitemaps: sitemaps from robots.txt,
/sitemap.xmland/sitemap_index.xml(including gzipped and nested ones) add in-scope pages that are not linked. - robots.txt: respected by default for the
KantoCrawleruser agent. If robots.txt cannot be reached at all, the site is not crawled (RFC 9309). Turn Respect robots.txt off only for a website you own or have written permission to crawl.
Limits
- Sites that block datacenter traffic or show a bot check return free
blockedrows; this crawler does not use residential proxies, fingerprint tricks or CAPTCHA solving. - No logins: pages behind a login are not crawled.
- Scanned PDFs without a text layer give a free error row (no OCR).
- Pages larger than Max page size (10 MB by default) are skipped with a free error row.
Related Kanto Labs actors
- Video to LLM Context - keyframes per scene + transcript of a video, packaged for an LLM next to your website content.
- Video OCR - on-screen text of video files (subtitles, captions, slides) as timed segments and SRT/VTT.
- AI Image Caption & Alt Text Generator - captions, alt text and tags for images, e.g. the images of the pages you crawled.
FAQ
Is it free to crawl a website?
You only pay for pages that are delivered with content: $0.15 per 1,000 plain-HTTP pages. Errors, blocked pages, duplicates and unchanged pages are free, and Apify's free plan credit covers thousands of pages per month.
Does it work on JavaScript websites (React, Vue, Angular)?
Yes. In auto mode the crawler fetches the plain HTML first and re-renders only the pages that are empty without JavaScript in headless Chromium (billed as Browser page). Choose browser to render every page.
How do I keep my vector database up to date?
Schedule the crawl and set Only changed pages since this dataset to the previous run's dataset. Unchanged pages come back free and marked unchanged, changed and new pages come back in full, and the OUTPUT record lists possiblyRemovedUrls.
What is the chunk size in?
Tokens of OpenAI's cl100k_base tokenizer (used by text-embedding-3-* and GPT-4-class models). Other tokenizers give similar counts for English.
Can I crawl many sites in one run?
Yes. Add several start URLs; each one keeps its own scope.
Something went wrong?
Every problem gets a clear error in its row and is free. If the Markdown of a page looks wrong, open an Issue with the URL and we will look at it.