Website Content Crawler for LLMs & RAG

Point it at a site. Get clean Markdown back, with the navigation and footer gone, the token count already worked out, and — if you want them — chunks ready to embed.

No browser. No API key. No proxy for most sites.

docs.apify.com/academy      24 pages    5.1s    1.3 MB    15 nav blocks stripped 180×
en.wikipedia.org             9 pages    2.1s    2.2 MB
react.dev/learn             15 pages    3.0s    3.1 MB

About 0.2 seconds a page. Every one of those sites serves its content over plain HTTP, so that is how this reads them.

---

The three things it does that a page-at-a-time extractor cannot

1. It learns the site's furniture from the crawl

A single page cannot tell its navigation from its article. Both are text in tags, and rules about position or tag name are wrong often enough to ruin an index.

A crawl can. The nav, the footer, the cookie bar and the "on this page" sidebar are exactly the blocks that appear on nearly every page — and the article is the part that does not. So the filter is learned from your corpus rather than guessed per page.

Two signals, because one is not enough:

• Frequency. A block on most pages of the crawl is furniture. Counted once per page, so a nav of forty links on one page cannot out-vote the same nav appearing on forty pages.
• Link density. Frequency alone missed the sidebars on docs.apify.com: documentation renders a different sidebar per section, so each nav sat on only three or four pages of twelve — under any sane threshold, and all of them survived into the content. A block that is almost entirely link text with no prose between the links is a menu whatever its frequency. Ordinary writing does not look like that; a paragraph with three citations still has sentences around them.

Blocks are matched with the link targets stripped, keeping only the link text. A nav highlights the page you are on, which makes it textually unique on every page it appears on — with the hrefs left in, a twelve-page crawl saw twelve different navs and removed none of them.

2. It counts tokens

Every page and every chunk carries tokenCount. RAG budgets are in tokens, not bytes, and the number you need at chunking time is the one nobody gives you.

It is an estimate, not a tokenizer call — shipping tiktoken would add megabytes of vocabulary to an actor whose whole argument is that it is cheap. Within roughly 10% on mixed prose, code and markup, and it deliberately errs high: a chunk smaller than budgeted is a non-event, one larger is a rejected embedding call. CJK is counted near one token per character rather than by the chars-over-four rule, which under-counts a Chinese page threefold.

3. It admits when a page needs a browser

Some pages really are client-rendered. Those come back with needsJavaScript: true instead of an empty body, so you can send those few URLs to a browser rather than discovering blank documents in your index three weeks later.

It does not fire on merely short pages. Across a crawl of linear.app — a Next.js site — it fired zero times, because linear.app server-renders.

---

What a page looks like

{
  "url": "https://docs.apify.com/api",
  "title": "Apify API documentation",
  "content": "# Apify API documentation\n\nLearn how to use the [Apify platform](https://docs.apify.com/) programmatically.\n\n## REST API\n\nThe Apify API is built around HTTP REST...\n\n```bash\ncurl https://api.apify.com/v2/acts\n```",
  "wordCount": 357,
  "tokenCount": 943,
  "headings": ["Apify API documentation", "REST API", "OpenAPI schema", "API clients"],
  "needsJavaScript": false,
  "boilerplateBlocksRemoved": 17,
  "fetchedInMs": 184
}

Headings, lists, tables and fenced code survive with their language hints. Links stay as [text](url) so the model can cite and you can follow.

And a chunk

{
  "chunkId": "a3f9c21b0e44-0002",
  "url": "https://docs.apify.com/api",
  "chunkIndex": 2,
  "chunkCount": 6,
  "heading": "API clients",
  "content": "## API clients\n\nThe client libraries are a more convenient way...",
  "tokenCount": 780,
  "overlapTokens": 96
}

chunkId is a hash of the URL and the index, so a re-crawl updates rows instead of duplicating them. Chunks break on block boundaries — never mid-sentence, and never through the middle of a code fence. heading carries forward, so a chunk taken from halfway down a page still says what section it belongs to.

---

What it will not do

• It does not run JavaScript. That is the point — it is why it costs what it costs. Pages that need a browser are flagged, not rendered.
• It does not pretend a 403 is a rate limit. Some sites refuse datacenter IPs; you get HTTP_403 immediately rather than three retries and twenty wasted seconds. Add a proxy for those.
• It does not normalise dates. publishedAt is whatever the page's meta tag said. Guessing a timezone the page never stated would invent precision.
• It respects robots.txt by default, including Crawl-delay, longest-rule-wins Allow over Disallow, and */$ wildcards. An empty Disallow: means permission, not a ban on everything.

Scope

The crawl stays on the hosts you started from. includeSubdomains widens it to the same registrable domain — and that is computed properly: taking the last two labels of bbc.co.uk would give co.uk, and a crawl scoped to that would accept every .co.uk address on the internet.

URLs are normalised before they are queued: fragment dropped, default port dropped, query sorted, tracking parameters removed. Left alone, a crawl spends its whole budget re-reading one article under a hundred different utm strings.

Input

{
  "startUrls": ["https://docs.apify.com/academy"],
  "maxPages": 200,
  "maxDepth": 3,
  "chunk": true,
  "chunkTokens": 800,
  "chunkOverlapTokens": 100
}

Everything else has a sensible default. useSitemap is on, which is the cheapest way to find every page on a site: one request, and it follows a sitemap index to the nested sitemaps it points at.

Pricing

$0.005 to start, $0.0015 per page, $0.0002 per chunk. Roughly $1.50 per 1,000 pages.

Not charged: pages below your minimum word count, pages robots.txt refused, URLs that errored, and responses that were not readable pages. A client-rendered page is charged — telling you it needs a browser is the finding you came for, and the fetch happened either way.

See docs/PRICING.md.