Website Content Crawler for AI - Markdown & RAG Chunks avatar

Website Content Crawler for AI - Markdown & RAG Chunks

Pricing

from $0.13 / 1,000 pages

Go to Apify Store
Website Content Crawler for AI - Markdown & RAG Chunks

Website Content Crawler for AI - Markdown & RAG Chunks

Crawl a website to clean Markdown, text and token-counted RAG chunks for LLMs, vector DBs, n8n and Make. Sitemaps, robots.txt, JS pages, linked PDF/DOCX, llms.txt, change-only re-crawls. Flat price per page.

Pricing

from $0.13 / 1,000 pages

Rating

0.0

(0)

Developer

Kanto Labs

Kanto Labs

Maintained by Community

Actor stats

0

Bookmarked

3

Total users

2

Monthly active users

13 days ago

Last modified

Features

Limited permissions

Share

What does Website Content Crawler for AI do?

Website Content Crawler for AI is a website content crawler that turns a whole website into clean Markdown, plain text and RAG-ready chunks for LLMs, vector databases, chatbots and AI agents. Give it the start of a docs site, blog, help centre or company site and it follows the links and the sitemap, keeps the main content of every page (navigation, headers, footers, sidebars, cookie banners, ads and share buttons are removed), and returns one row per page with the Markdown, text, title, description, language, a content hash and heading-aware chunks with token counts.

You pay a flat price per page, so you know the cost before you press Start.

Why use this website content crawler?

  • Flat, predictable price per page. No compute-unit maths, no surprise bills from slow pages, retries or a big memory setting. $0.15 per 1,000 pages (plain HTTP), $0.90 per 1,000 pages that need a headless browser.
  • RAG-ready output. Every page comes pre-split into heading-aware chunks (default 512 tokens, 64 tokens overlap, OpenAI cl100k_base token counts). Each chunk carries the page URL, title and heading path (["Guide", "Install", "Windows"]), so you can embed it straight away.
  • Linked documents too. Turn on Convert linked documents and the PDF, DOCX, PPTX and XLSX files linked from the pages become Markdown + chunks as well.
  • llms.txt and the whole site in one file. Each run writes llms.txt (a page index), llms-full.txt (every page's Markdown in one file) and chunks.jsonl (all chunks, one per line) to the key-value store.
  • Cheap re-crawls: only changed pages. Point a new run at the dataset of an earlier run: pages that did not change (same content hash, or 304 Not Modified) come back as free unchanged rows. You pay only for new and changed pages.
  • Duplicates removed for free. Same canonical URL, identical text or near-identical text (SimHash: print views, tracking-parameter and sort variants) become free duplicate rows that point to the first copy.
  • JavaScript sites. Fast plain HTTP by default; pages that only show content after JavaScript runs are automatically re-rendered in headless Chromium.
  • Polite and safe. robots.txt is respected by default (with Crawl-delay), the crawler identifies itself as KantoCrawler, and failed, blocked, robots-disallowed, duplicate and unchanged pages are never charged. Bot walls, CAPTCHAs and logins are never bypassed.

How to crawl a website to Markdown

  1. Put the start of the site in Start URLs, e.g. https://docs.example.com/ (pages under the same path are followed).
  2. Set Max pages (and, if you want, include / exclude URL globs and a max depth).
  3. Keep Crawler type = auto (HTTP, browser only when needed). Choose http for the lowest price or browser for app-like sites.
  4. Optional: Convert linked documents, a different chunk size, or Only changed pages since this dataset for a re-crawl.
  5. Click Start. Pages appear in the Pages view; the OUTPUT record links llms.txt, llms-full.txt and chunks.jsonl.

Input example

{
"startUrls": [{ "url": "https://docs.example.com/" }],
"maxPages": 500,
"crawlerType": "auto",
"excludeUrlGlobs": ["https://docs.example.com/**/changelog/**"],
"chunkSize": 512,
"chunkOverlap": 64,
"convertLinkedDocuments": true
}

Output example

One row per page (Markdown, text and chunks shortened):

{
"url": "https://www.joelonsoftware.com/2018/04/13/gamification/",
"status": "ok",
"type": "page",
"renderedWith": "http",
"httpStatus": 200,
"depth": 1,
"title": "A Dusting of Gamification – Joel on Software",
"description": "I had to think for a minute to realize that Stack Overflow has \"gamification\" too...",
"language": "en-US",
"publishedAt": "2018-04-13T13:40:21+00:00",
"markdown": "# A Dusting of Gamification\n\nAround 2010 the success of Stack Overflow had led us into some conversations with VCs...",
"text": "A Dusting of Gamification\n\nAround 2010 the success of Stack Overflow...",
"wordCount": 1235,
"tokenCount": 1690,
"contentHash": "bb8a180d7aac4889f277c95b4f07b695dea90850d9f8dd13ad94a59f93e09289",
"simHash": "9f1c...",
"chunkCount": 6,
"chunks": [
{
"chunkId": "198844c52f-2",
"index": 2,
"url": "https://www.joelonsoftware.com/2018/04/13/gamification/",
"title": "A Dusting of Gamification – Joel on Software",
"headingPath": ["A Dusting of Gamification"],
"text": "We didn't want to punish you so much as we wanted to show other people that your answer was wrong...",
"tokenCount": 424
}
],
"billedEvent": "page",
"billedUnits": 1,
"error": null
}

Linked documents are rows with "type": "document" plus fileName, format and pageCount. Free rows have status = failed, blocked, robots-disallowed, duplicate, unchanged or skipped, an empty markdown and a plain-language error / duplicateOf. Filter on status = "ok" to get the content.

How much does it cost to crawl a website?

EventPrice (Free / Bronze)SilverGoldWhen
Page$0.00015$0.000135$0.00013a page delivered over plain HTTP
Browser page$0.0009$0.00081$0.00078a page that needed headless Chromium
Document$0.002$0.0018$0.0017a linked PDF / DOCX / PPTX / XLSX converted

Plus the standard Apify start fee ($0.00005 per GB of memory, $0.0001 at the default 2 GB). One event covers up to 25,000 tokens of Markdown (about 18,000 words, longer than nearly every web page); a longer page or file counts one event per started 25,000 tokens. Failed, blocked, robots-disallowed, duplicate and unchanged pages cost $0.

Why a flat price beats usage billing (worked example)

You crawl a 1,000-page documentation site once a week.

  • Here: 1,000 pages x $0.00015 = $0.15 per crawl, the same every week. Next week, with Only changed pages, the 950 pages that did not change are free: about $0.01.
  • Usage-billed crawlers charge for compute time instead (memory x run time). Their own estimates for raw-HTTP crawling are around $0.2 per 1,000 pages, and $0.5 to $5 per 1,000 pages with a headless browser. The real bill moves with page weight, slow servers, retries and the memory you pick, so you only know it after the run.

Set Max cost per run in the run options and the crawler stops cleanly when the budget is reached.

Use it with n8n, Make, LangChain and LlamaIndex

  • n8n / Make / Zapier: run the Actor (or a saved task) with the Apify integration, then loop over the dataset items where status is ok and send markdown or each item of chunks to your vector store, Google Sheet or LLM step.
  • LangChain (Python):
from langchain_apify import ApifyWrapper
from langchain_core.documents import Document
loader = ApifyWrapper().call_actor(
actor_id="kantolabs/website-content-crawler-ai",
run_input={"startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 200},
dataset_mapping_function=lambda item: Document(
page_content=item["markdown"], metadata={"source": item["url"], "title": item["title"]}),
)
docs = [d for d in loader.load() if d.page_content]
  • LlamaIndex: use ApifyActor from llama-index-readers-apify with the same actor_id and map item["markdown"] (or each chunk's text) to a Document.
  • Plain API / agents: chunks.jsonl from the OUTPUT record is one chunk per line with url, headingPath, text and tokenCount, ready to embed.

Crawl scope, sitemaps and robots.txt

  • Scope: by default only pages on the same host and under the start URL's folder are followed (https://example.com/blog/ crawls /blog/...). Choose Same host or Same domain and subdomains, or set Include URLs (globs) such as https://example.com/docs/**.
  • Sitemaps: sitemaps from robots.txt, /sitemap.xml and /sitemap_index.xml (including gzipped and nested ones) add in-scope pages that are not linked.
  • robots.txt: respected by default for the KantoCrawler user agent. If robots.txt cannot be reached at all, the site is not crawled (RFC 9309). Turn Respect robots.txt off only for a website you own or have written permission to crawl.

Limits

  • Sites that block datacenter traffic or show a bot check return free blocked rows; this crawler does not use residential proxies, fingerprint tricks or CAPTCHA solving.
  • No logins: pages behind a login are not crawled.
  • Scanned PDFs without a text layer give a free error row (no OCR).
  • Pages larger than Max page size (10 MB by default) are skipped with a free error row.
  • Video to LLM Context - keyframes per scene + transcript of a video, packaged for an LLM next to your website content.
  • Video OCR - on-screen text of video files (subtitles, captions, slides) as timed segments and SRT/VTT.
  • AI Image Caption & Alt Text Generator - captions, alt text and tags for images, e.g. the images of the pages you crawled.

FAQ

Is it free to crawl a website?

You only pay for pages that are delivered with content: $0.15 per 1,000 plain-HTTP pages. Errors, blocked pages, duplicates and unchanged pages are free, and Apify's free plan credit covers thousands of pages per month.

Does it work on JavaScript websites (React, Vue, Angular)?

Yes. In auto mode the crawler fetches the plain HTML first and re-renders only the pages that are empty without JavaScript in headless Chromium (billed as Browser page). Choose browser to render every page.

How do I keep my vector database up to date?

Schedule the crawl and set Only changed pages since this dataset to the previous run's dataset. Unchanged pages come back free and marked unchanged, changed and new pages come back in full, and the OUTPUT record lists possiblyRemovedUrls.

What is the chunk size in?

Tokens of OpenAI's cl100k_base tokenizer (used by text-embedding-3-* and GPT-4-class models). Other tokenizers give similar counts for English.

Can I crawl many sites in one run?

Yes. Add several start URLs; each one keeps its own scope.

Something went wrong?

Every problem gets a clear error in its row and is free. If the Markdown of a page looks wrong, open an Issue with the URL and we will look at it.