# Website Content Crawler for LLMs & RAG (`eiv/llm-content-crawler`) Actor

Crawl any site to clean Markdown for RAG, with no browser. Learns each site's navigation and footer from the crawl itself and strips them, counts tokens per page, and emits ready-to-embed chunks. Says NEEDS\_JS instead of returning a blank page.

- **URL**: https://apify.com/eiv/llm-content-crawler.md
- **Developed by:** [Eimantas V](https://apify.com/eiv) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 page crawls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Content Crawler for LLMs & RAG

Point it at a site. Get clean Markdown back, with the navigation and footer gone, the token count already worked out, and — if you want them — chunks ready to embed.

No browser. No API key. No proxy for most sites.

```
docs.apify.com/academy      24 pages    5.1s    1.3 MB    15 nav blocks stripped 180×
en.wikipedia.org             9 pages    2.1s    2.2 MB
react.dev/learn             15 pages    3.0s    3.1 MB
```

**About 0.2 seconds a page.** Every one of those sites serves its content over plain HTTP, so that is how this reads them.

***

### The three things it does that a page-at-a-time extractor cannot

#### 1. It learns the site's furniture from the crawl

A single page cannot tell its navigation from its article. Both are text in tags, and rules about position or tag name are wrong often enough to ruin an index.

A crawl can. The nav, the footer, the cookie bar and the "on this page" sidebar are exactly the blocks that appear on nearly every page — and the article is the part that does not. So the filter is learned from your corpus rather than guessed per page.

Two signals, because one is not enough:

- **Frequency.** A block on most pages of the crawl is furniture. Counted once per page, so a nav of forty links on one page cannot out-vote the same nav appearing on forty pages.
- **Link density.** Frequency alone missed the sidebars on docs.apify.com: documentation renders a *different* sidebar per section, so each nav sat on only three or four pages of twelve — under any sane threshold, and all of them survived into the content. A block that is almost entirely link text with no prose between the links is a menu whatever its frequency. Ordinary writing does not look like that; a paragraph with three citations still has sentences around them.

Blocks are matched with the link targets stripped, keeping only the link text. A nav highlights the page you are on, which makes it textually unique on every page it appears on — with the hrefs left in, a twelve-page crawl saw twelve different navs and removed none of them.

#### 2. It counts tokens

Every page and every chunk carries `tokenCount`. RAG budgets are in tokens, not bytes, and the number you need at chunking time is the one nobody gives you.

It is an estimate, not a tokenizer call — shipping tiktoken would add megabytes of vocabulary to an actor whose whole argument is that it is cheap. Within roughly 10% on mixed prose, code and markup, and it deliberately errs high: a chunk smaller than budgeted is a non-event, one larger is a rejected embedding call. CJK is counted near one token per character rather than by the chars-over-four rule, which under-counts a Chinese page threefold.

#### 3. It admits when a page needs a browser

Some pages really are client-rendered. Those come back with `needsJavaScript: true` instead of an empty body, so you can send those few URLs to a browser rather than discovering blank documents in your index three weeks later.

It does not fire on merely short pages. Across a crawl of linear.app — a Next.js site — it fired zero times, because linear.app server-renders.

***

### What a page looks like

````json
{
  "url": "https://docs.apify.com/api",
  "title": "Apify API documentation",
  "content": "# Apify API documentation\n\nLearn how to use the [Apify platform](https://docs.apify.com/) programmatically.\n\n## REST API\n\nThe Apify API is built around HTTP REST...\n\n```bash\ncurl https://api.apify.com/v2/acts\n```",
  "wordCount": 357,
  "tokenCount": 943,
  "headings": ["Apify API documentation", "REST API", "OpenAPI schema", "API clients"],
  "needsJavaScript": false,
  "boilerplateBlocksRemoved": 17,
  "fetchedInMs": 184
}
````

Headings, lists, tables and fenced code survive with their language hints. Links stay as `[text](url)` so the model can cite and you can follow.

### And a chunk

```json
{
  "chunkId": "a3f9c21b0e44-0002",
  "url": "https://docs.apify.com/api",
  "chunkIndex": 2,
  "chunkCount": 6,
  "heading": "API clients",
  "content": "## API clients\n\nThe client libraries are a more convenient way...",
  "tokenCount": 780,
  "overlapTokens": 96
}
```

`chunkId` is a hash of the URL and the index, so a re-crawl **updates** rows instead of duplicating them. Chunks break on block boundaries — never mid-sentence, and never through the middle of a code fence. `heading` carries forward, so a chunk taken from halfway down a page still says what section it belongs to.

***

### What it will not do

- **It does not run JavaScript.** That is the point — it is why it costs what it costs. Pages that need a browser are flagged, not rendered.
- **It does not pretend a 403 is a rate limit.** Some sites refuse datacenter IPs; you get `HTTP_403` immediately rather than three retries and twenty wasted seconds. Add a proxy for those.
- **It does not normalise dates.** `publishedAt` is whatever the page's meta tag said. Guessing a timezone the page never stated would invent precision.
- **It respects robots.txt by default**, including `Crawl-delay`, longest-rule-wins `Allow` over `Disallow`, and `*`/`$` wildcards. An empty `Disallow:` means permission, not a ban on everything.

### Scope

The crawl stays on the hosts you started from. `includeSubdomains` widens it to the same registrable domain — and that is computed properly: taking the last two labels of `bbc.co.uk` would give `co.uk`, and a crawl scoped to that would accept every `.co.uk` address on the internet.

URLs are normalised before they are queued: fragment dropped, default port dropped, query sorted, tracking parameters removed. Left alone, a crawl spends its whole budget re-reading one article under a hundred different `utm` strings.

### Input

```json
{
  "startUrls": ["https://docs.apify.com/academy"],
  "maxPages": 200,
  "maxDepth": 3,
  "chunk": true,
  "chunkTokens": 800,
  "chunkOverlapTokens": 100
}
```

Everything else has a sensible default. `useSitemap` is on, which is the cheapest way to find every page on a site: one request, and it follows a sitemap index to the nested sitemaps it points at.

### Pricing

$0.005 to start, **$0.0015 per page**, $0.0002 per chunk. Roughly **$1.50 per 1,000 pages**.

Not charged: pages below your minimum word count, pages robots.txt refused, URLs that errored, and responses that were not readable pages. A client-rendered page *is* charged — telling you it needs a browser is the finding you came for, and the fetch happened either way.

See [docs/PRICING.md](docs/PRICING.md).

# Actor input Schema

## `startUrls` (type: `array`):

Where to begin. A bare domain works — example.com becomes https://example.com. Everything crawled stays on these hosts unless you turn on Include subdomains.

## `maxPages` (type: `integer`):

How many pages to read at most. This is the number you are charged for, and the crawl stops the moment it is reached.

## `maxDepth` (type: `integer`):

How many links deep to follow from a start URL. 0 reads only the start URLs themselves. Pages found in the sitemap enter at depth 1.

## `followLinks` (type: `boolean`):

Discover more pages by following links found on each page. Turn this off to read only the start URLs and whatever the sitemap lists.

## `useSitemap` (type: `boolean`):

Read sitemap.xml first, including any sitemaps robots.txt points at and any nested sitemap index. It is the fastest way to find every page on a site and costs one request.

## `respectRobotsTxt` (type: `boolean`):

Honour the site's crawl rules and its Crawl-delay. Turn this off only for sites you own or have permission to crawl.

## `includeSubdomains` (type: `boolean`):

Also crawl other hosts on the same root domain, so a start URL on example.com will also read docs.example.com and blog.example.com.

## `includeUrlPatterns` (type: `array`):

Only crawl URLs matching one of these. Globs by default (*/blog/*), or write /regex/ for a regular expression. Leave empty to allow everything in scope.

## `excludeUrlPatterns` (type: `array`):

Skip URLs matching any of these. Same glob or /regex/ syntax. Useful for /tag/, /author/ and other pages that hold no content.

## `outputFormat` (type: `string`):

markdown keeps headings, lists, tables and code fences, which is what an LLM reads best. text is the same content with the markup stripped. html returns the original source untouched.

## `removeBoilerplate` (type: `boolean`):

Drop the navigation, footer, cookie bar and sidebar. Blocks repeating across the crawl are learned and removed, and link-only menus are removed on sight even when the crawl is too small to learn from.

## `keepLinks` (type: `boolean`):

Keep links as Markdown [text](url). Turn off for prose only — smaller, but the model can no longer cite or follow anything.

## `keepImages` (type: `boolean`):

Include images as ![alt](url). Off by default: alt text is rarely worth the tokens unless you are indexing figures.

## `keepTables` (type: `boolean`):

Render tables as Markdown rows. Off means the cells still appear, but the row structure is lost.

## `minWords` (type: `integer`):

Skip pages with fewer words than this once boilerplate is gone — redirect stubs, empty tag pages, bare listings. Skipped pages are not charged. A page that needs JavaScript is always kept, because its emptiness is the finding.

## `chunk` (type: `boolean`):

Also emit ready-to-embed chunks as their own records, each with a stable id, its token count and the heading it sits under.

## `chunkTokens` (type: `integer`):

Target size per chunk. Chunks break on block boundaries, so a chunk never starts mid-sentence and a code fence is never cut in half.

## `chunkOverlapTokens` (type: `integer`):

Tokens carried from the end of one chunk into the start of the next, so a fact spanning a boundary survives. Must be below the chunk size.

## `flattenOutput` (type: `boolean`):

Turn arrays into pipe-delimited strings so the CSV and Excel exports have a fixed, readable column set.

## `maxConcurrency` (type: `integer`):

Pages fetched at once. Raise it for large crawls of sites that can take it; lower it if you get rate limited.

## `requestDelayMs` (type: `integer`):

Milliseconds to wait between requests. Raised automatically if robots.txt asks for a longer Crawl-delay.

## `requestTimeoutSecs` (type: `integer`):

Seconds to wait for one page before giving up on it.

## `maxRetries` (type: `integer`):

Retries for a page that times out or answers with a retryable status.

## `proxyConfiguration` (type: `object`):

Optional. Not needed for most sites — this actor sends plain HTTP requests and does not run a browser.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "followLinks": true,
  "useSitemap": true,
  "respectRobotsTxt": true,
  "includeSubdomains": false,
  "outputFormat": "markdown",
  "removeBoilerplate": true,
  "keepLinks": true,
  "keepImages": false,
  "keepTables": true,
  "minWords": 25,
  "chunk": false,
  "chunkTokens": 800,
  "chunkOverlapTokens": 100,
  "flattenOutput": false,
  "maxConcurrency": 5,
  "requestDelayMs": 200,
  "requestTimeoutSecs": 45,
  "maxRetries": 2
}
```

# Actor output Schema

## `results` (type: `string`):

Page records carry recordType 'page'; chunks carry 'chunk'; URLs that errored or were refused carry 'page-error'.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("eiv/llm-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://docs.apify.com/academy"] }

# Run the Actor and wait for it to finish
run = client.actor("eiv/llm-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ]
}' |
apify call eiv/llm-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eiv/llm-content-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Tfbh5hOPg02CjaFDm/builds/MwVIQZpxohWGsMf4L/openapi.json