# Website Content Crawler (`pumpkinapi/website-content-crawler`) Actor

Website content crawler for RAG and LLMs: each page as clean Markdown, menus and footers removed, optional chunks. $3 per 1,000 saved pages ($2.10 on Business); refused and failed pages are free. HTML only: no JavaScript rendering, proxy or PDF. Max 5,000 pages per website.

- **URL**: https://apify.com/pumpkinapi/website-content-crawler.md
- **Developed by:** [Pumpkin Farm](https://apify.com/pumpkinapi) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 10 total users, 10 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Website content crawler for RAG and LLM apps. Give it a website, a section of one or a list of pages, and it saves each page's own text as clean Markdown: menus, footers and cookie banners that repeat on most pages are taken out and kept once, and a page that has the same text as another address is saved once. Turn on chunks and every page comes split into pieces that know the headings above them, ready for embeddings and a vector database.

Found a bug, need a field, or want a new Actor? Open an issue on the **Issues** tab. If it works well for you, a short review helps others find it.

- **Price:** $3.00 per 1,000 saved pages on the Free plan, $2.10 on Business, platform usage included. You pay only for pages saved with their text: pages that refuse the crawler, fail, copy another page or have no text are free, and so are chunks and notices.
- **HTML only:** no JavaScript rendering, proxy or PDF. A website that needs them gets a free notice row.
- **Max pages per website:** up to 5,000 in one run. It also caps what one website can cost.
- **Input:** a website (`example.com`), a section (`example.com/docs/`) or single pages. An input written for apify/website-content-crawler works as it is.
- **Output:** one row per page, or one row per chunk, with `text` in Markdown, `title`, `description`, `language` and the `headings` above each chunk.
- **Measured:** 197 pages from 4 websites in 51 seconds on Apify. Of 100 websites in two tests, 82 gave their text, 16 refused the crawler and 2 need JavaScript.

### How to use it

1. Put the address in **Websites to crawl**: `example.com` for a whole site, `example.com/docs/` for one section, or several pages, one per line.
2. Choose **Crawl scope**: pages under the start address, the whole website, or only the addresses you gave.
3. Set **Max pages per website**. It also caps what one website can cost.
4. For a vector database, turn on **Split text into chunks** and set **Chunk size**.
5. Click **Start**, then open **Pages** or **Chunks** in the Output tab.

```json
{
    "startUrls": ["https://docs.python.org/3/library/"],
    "maxPages": 60,
    "splitIntoChunks": true,
    "chunkSize": 1000,
    "chunkOverlap": 100
}
```

The crawl follows links nearest first, then reads the pages the website's sitemap lists that no link led to. It keeps to the scope you chose, so `example.com/docs/` never leaves `/docs/`, and `example.com/en/5.2` stays in that one version of the docs.

### What it takes out and what it keeps

| Part of the page | What happens |
| --- | --- |
| Menus, footers, cookie banners | A text block found on at least half of the pages of a website, and on at least 3, is taken out of every page and saved once, in a row of type `shared` |
| Bars under the page title | "Edit this page" or breadcrumbs that most pages have right under their title go to the `shared` row too |
| Several languages | Each language is counted on its own and gets its own `shared` row |
| Navigation elements | `<nav>` is left out by default. **Remove HTML elements** takes any CSS selector |
| The same page at two addresses | Saved once, under the shorter address. Copies are listed in the run summary |
| Tables | Markdown tables. A price list without a header row gets an empty one |
| Code | Fenced code blocks with every line and space as the browser shows them, and the language when the page names it. Line numbers, copy buttons and prompts you cannot select are left out |
| FAQ accordions | A question on a button becomes a heading over its answer, so a chunk keeps the two together |
| Contact links | A "Call us" link gets its number in brackets, a "Write to us" link its address |
| Images | Their alt text stays, file names do not |
| Embedded maps and booking forms | A line such as `Map: https://...` or `Booking form: https://...` |
| Scripts, styles, forms, other buttons | Left out |

### What data you get

| Field | What it holds |
| --- | --- |
| `type` | `page`, `chunk`, `shared` (the text repeated across the website) or `notice` (why something could not be read) |
| `url`, `title`, `description`, `language` | The address after redirects, the page title, meta description and the `lang` of the page |
| `text`, `chars` | The page's own text as Markdown, and its length in characters |
| `chunkIndex`, `chunkCount`, `headings` | For chunks: the place of the chunk in its page, how many the page has, and the headings above it |
| `depth`, `foundBy` | Links from a start address, and whether the page came from a `start` address, a `link` or the `sitemap` |
| `site`, `crawledAt`, `error` | The website, the time, and on a notice row what happened |

A real chunk row, the first of 22 from one Python documentation page:

```json
{
    "type": "chunk",
    "url": "https://docs.python.org/3/library/",
    "title": "The Python standard library — Python 3.15.2 documentation",
    "text": "# The Python standard library\n\nThis library reference manual describes the standard library distributed with Python. It also describes some of the optional components that are commonly included in Python distributions.\n\nElsewhere, The Python Language Reference describes the exact syntax and semantics of the Python language, and Python built-ins reference describes the built-in functions.",
    "chars": 390,
    "chunkIndex": 0,
    "chunkCount": 22,
    "headings": ["The Python standard library"],
    "language": "en",
    "depth": 0,
    "foundBy": "start",
    "site": "https://docs.python.org"
}
```

A long table is cut between rows with its header repeated, a long code block between lines with its fences repeated. With **Chunk overlap**, a chunk starts with the last words, rows or code lines of the one before it.

A real page row with chunks off:

```json
{
    "type": "page",
    "url": "https://www.usa.gov/phone",
    "title": "Call USAGov | USAGov",
    "text": "# Call USAGov\n\nTalk with a USAGov agent for information on federal benefits, services, programs, and more. They will answer your question or direct you to an agency that can. Or call to report an issue with or content update for the USA.gov website.\n\nFor your protection, do not share personal information like telephone or Social Security numbers.\n\nCalls may be monitored for quality assurance.\n\n## Hours of operation\n\nAgents are available Monday - Friday, 8:00 AM - 8:00 PM Eastern Time (except federal holidays).\n\n## Call USAGov\n\nTo speak to a live agent, call toll-free from the U.S. or Canada 1-844-USA-GOV1 (1-844-872-4681).\n\nLAST UPDATED: May 22, 2025",
    "chars": 658,
    "chunkIndex": null,
    "headings": null,
    "language": "en",
    "depth": 0,
    "foundBy": "start",
    "site": "https://www.usa.gov"
}
```

The `SUMMARY` record in the key-value store lists, for each website, the pages saved, the duplicates skipped, the pages that failed and why, and how many found pages the scope, the filters, robots.txt or the page limit left out.

### How much does it cost?

You pay for each page saved with its text. Chunks, `shared` rows and notices cost nothing, and neither do pages that fail, copies of another page or pages with no text of their own.

| Plan | Per 1,000 pages |
| --- | --- |
| Free | $3.00 |
| Starter | $2.70 |
| Scale | $2.40 |
| Business | $2.10 |

A run also pays $0.00005 when it starts. Example: a website of 500 pages costs $1.50 on the Free plan, the same with chunks on or off. **Max pages per website** and the run's maximum cost both stop the crawl before it goes over.

### How to build a chatbot knowledge base from a website

1. Start at the home page with **Crawl scope** set to the whole website, and turn on chunks.
2. Keep **Remove menus, footers and banners** on. The `shared` row then holds the footer once: contacts, addresses and opening hours stay findable without being repeated in every chunk.
3. Embed `text`. Keep `url`, `title` and `headings` as metadata, so an answer can name the page and the section it came from.

A website in two languages gives two `shared` rows. Filter rows by `language` to build one assistant per language.

### How to crawl only the docs

Start at the docs section, for example `https://kubernetes.io/docs/concepts/`, and leave **Crawl scope** at pages under the start address. Use **Skip pages whose address contains** for parts you do not want, such as `/blog/` or `?page=`. To read only the main text of each page, put its element in **Keep only HTML elements**, for example `main` or `article`.

### An input written for apify/website-content-crawler

Paste it as it is. `startUrls` with `{ "url": ... }`, `includeUrlGlobs` (they decide the scope instead of the start path), `excludeUrlGlobs`, `maxCrawlDepth`, `keepElementsCssSelector`, `removeElementsCssSelector`, `useSitemaps`, `respectRobotsTxtFile`, `customHttpHeaders` and `initialCookies` work the same way; `maxCrawlPages` caps each website and `maxResults` the whole run. Headers go only to the hosts of the start addresses and their subdomains, cookies only where their domain matches. Settings for a proxy, a browser or files get a free notice row, and the rest of the input is used.

### Limits

- It reads HTML and runs no JavaScript. A website that builds its text in the browser gets a notice, and its empty pages are not saved or charged. A part that a script fills in later, such as the ticket prices on one museum page, is missing without a notice.
- It uses no proxy. Websites with bot protection refuse it: 16 of the 100 websites in our tests did, 9 of them online shops. You get a notice row that names the protection when it can tell. A website that lets the first page through and then refuses 20 pages in a row is left, and what was read is saved.
- PDF, Word and other files are skipped. Pages behind a login are read only with the cookies or headers you give through the API.
- One website is read one request at a time, at least 0.25 seconds apart, slower when the site is slow or asks for a Crawl-delay. 60 pages of one website took between 20 and 49 seconds in our runs, 400 pages 7 to 8 minutes, and 5,000 pages of one documentation website 54 minutes. Several websites are read at the same time.
- The rows of a website are saved when its crawl ends, because what repeats across its pages is known only then. Until then the status line counts the pages read.
- Up to 5,000 pages per website in one run. Their text stays in memory until the crawl of the website ends: 5,000 documentation pages with chunks took 557 MB of the default 1 GB, and a website of very long pages needs a run with more memory.
- A run stops reading before its timeout, 12 minutes before the default 4 hours, and saves what it has. robots.txt is followed by default.
- When up to 3 pages of a website are read, or up to 3 in one language, they keep their menus and footers. A repeated line right under a section heading stays in its page.

### API and integrations

Run it from the Apify API, the Apify clients for JavaScript and Python, schedules, webhooks, Make, Zapier or any MCP client. The dataset exports to JSON, CSV and Excel, and the `chunks` view gives just the fields an embedding step needs.

### FAQ

**Do I pay per chunk?** No. You pay per page. A page split into 30 chunks costs the same as one row.

**Do code blocks and FAQ accordions come through?** Yes. On the 20 pages of the example input, 178 of 178 code blocks matched what a browser shows, line by line, without line numbers. On 5 pricing pages, 77 of 77 FAQ questions stayed with their answers.

**A website gave 0 pages. Why?** Look at its `notice` row: it says whether the site refused the crawler, robots.txt does not allow it, the address did not open, or the text needs JavaScript. You are not charged for those pages.

**The dataset is empty while the run is going. Is it stuck?** No. Each website's rows arrive together when its crawl ends. The status line counts the pages read so far.

**Can I keep menus and footers in every page?** Yes. Turn off **Remove menus, footers and banners** and clear **Remove HTML elements**.

**Does it obey robots.txt?** Yes, its rules and its Crawl-delay up to 10 seconds. Turn **Respect robots.txt** off only for a website you own or may crawl.

**What chunk size should I use?** The default is 1,500 characters, about 375 tokens of English text. Sizes from 400 to 20,000 characters are allowed.

*This Actor is an independent tool. It has no connection to the websites it reads, and their content belongs to their owners.*

# Actor input Schema

## `startUrls` (type: `array`):

One address per line: a whole website (example.com), a section (https://example.com/docs/) or single pages. Addresses of the same website are crawled together. Crawl scope below says how far the crawl goes from them.

## `maxPages` (type: `integer`):

The crawl of one website stops after reading this many pages. You pay only for the pages it saves, so this also caps the cost of each website: copies of another page and pages without text of their own are read but not saved. Up to 5,000.

## `crawlScope` (type: `string`):

Which pages the crawl takes. Pages under the start address: https://example.com/docs/ reads only pages whose path starts with /docs/. Whole website: every page of the same website, www or not. Only the start addresses: reads the given pages and follows no links.

## `splitIntoChunks` (type: `boolean`):

Save each page as several rows of up to Chunk size characters, ready for embeddings and a vector database. Each chunk keeps the headings above it, tables are cut between rows with the header row repeated. You still pay per page, not per chunk.

## `chunkSize` (type: `integer`):

The longest a chunk may be, in characters. About 4 characters make one token in English text.

## `chunkOverlap` (type: `integer`):

Each chunk after the first starts with up to this many characters from the end of the chunk before it, cut at a word, a table row or a line of code. At most a quarter of the chunk size. 0 means no overlap.

## `removeRepeatedText` (type: `boolean`):

Text blocks found on at least half of the pages of a website (and on at least 3) are taken out of every page and saved once, in a row of type shared. A website in several languages is counted language by language, and each language gets its own shared row. A website of up to 3 pages keeps every page whole. Turn off to keep every page as it is.

## `removeElementsCssSelector` (type: `string`):

Elements matching this CSS selector are left out of the text, besides scripts, styles, forms and buttons, which are always left out (a FAQ question on a button stays). The default leaves out navigation menus. Clear it to keep them.

## `keepElementsCssSelector` (type: `string`):

Optional. Only the elements matching this CSS selector are read, for example main or article, and the rest of the page is left out. A page where nothing matches has no text and is not saved or charged. Empty reads the whole page.

## `includeUrls` (type: `array`):

Optional. A found page is read only when its address contains one of these, in any case. \* stands for anything, for example /blog/ or /docs/\*/guide. Start addresses are always read.

## `excludeUrls` (type: `array`):

Optional. A found page is skipped when its address contains one of these, in any case. \* stands for anything, for example /tag/ or ?page=. Skipped pages are not downloaded.

## `useSitemaps` (type: `boolean`):

After the links run out, also read the pages listed in the website's sitemaps (from robots.txt, or /sitemap.xml and the usual places) that no link led to. They follow the same scope and filters.

## `respectRobotsTxt` (type: `boolean`):

Skip the pages the website's robots.txt does not allow, and wait as long as its Crawl-delay asks, up to 10 seconds. Turn off only for a website you own or may crawl anyway.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "maxPages": 20,
  "crawlScope": "path",
  "splitIntoChunks": false,
  "chunkSize": 1500,
  "chunkOverlap": 0,
  "removeRepeatedText": true,
  "removeElementsCssSelector": "nav, [role=\"navigation\"]",
  "keepElementsCssSelector": "",
  "includeUrls": [],
  "excludeUrls": [],
  "useSitemaps": true,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `chunks` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy/scraping-basics-javascript"
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("pumpkinapi/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy/scraping-basics-javascript"],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("pumpkinapi/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "maxPages": 20
}' |
apify call pumpkinapi/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pumpkinapi/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kkCvihfaNzbFBeWjM/builds/F3zsPuoVyi1hw7bjh/openapi.json