# Website Content Crawler (`neuro-scraper/website-content-crawler`) Actor

Website Content Crawler SD - Website Content Crawler is a web crawling tool that extracts the main content of every page of a website as clean Markdown and text, with title, language and metadata, for LLMs, RAG, vector databases and search indexes.

- **URL**: https://apify.com/neuro-scraper/website-content-crawler.md
- **Developed by:** [Neuro Scraper](https://apify.com/neuro-scraper) (community)
- **Categories:** AI, Developer tools
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.49 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Website Content Crawler — clean Markdown and text from whole websites

Website Content Crawler crawls a website from one or more start URLs and saves every page's **main content** as Markdown and plain text, with title, description, language, canonical URL, author, dates and metadata. Menus, footers and cookie banners are removed, so the output is ready for LLMs, RAG pipelines, vector databases and search indexes.

It reads pages over HTTP (no browser), follows links inside the scope you choose, can seed the crawl from the site's sitemaps, respects robots.txt by default and fetches one page at a time per site with a pause between requests.

#### Input

| Field | Type | Default | Meaning |
|---|---|---|---|
| `startUrls` | URL list | crawlee.dev/python/docs/guides (prefill) | Where the crawl starts |
| `maxPages` | integer | 20 | Pages saved (one row each) |
| `maxCrawlDepth` | integer | 5 | Links away from a start URL; 0 = start URLs only |
| `crawlScope` | select | `startUrlPath` | `startUrlPath` (URLs starting with the start URL), `sameDomain`, `sameDomainAndSubdomains` |
| `includeUrlGlobs` / `excludeUrlGlobs` | glob lists | – | e.g. `https://site.com/docs/**`; include globs replace the scope |
| `useSitemaps` | boolean | false | Also queue in-scope URLs from robots.txt sitemaps / `/sitemap.xml` |
| `respectRobotsTxt` | boolean | true | Skip disallowed pages, honour Crawl-delay (max 10 s) |
| `htmlTransformer` | select | `trafilatura` | `trafilatura`, `readability`, or `none` (whole cleaned page) |
| `saveMarkdown` / `saveText` / `saveHtml` | booleans | true / true / false | Output formats |
| `keepMarkdownLinks` | boolean | false | Keep `[text](url)` links in the Markdown |
| `requestDelayMs` | integer | 1000 | Pause between requests to one host |
| `maxConcurrency` | integer | 2 | Pages processed at once across hosts |
| `proxyConfiguration` | proxy | Apify datacenter | Use RESIDENTIAL for sites that block datacenter IPs |

#### Output (one row per page)

| Field | Meaning |
|---|---|
| `url`, `requestUrl`, `startUrl`, `referrerUrl`, `depth`, `statusCode` | Where the page came from |
| `title`, `description`, `language`, `canonicalUrl`, `author`, `publishedAt` | Page metadata |
| `markdown`, `text`, `html` | Main content in the formats you chose |
| `wordCount`, `extractor` | Words in the main content; which extractor produced it |
| `likelyJsRendered` | True when the server sent almost no text (page builds itself with JavaScript) |
| `truncated` | True if a field was cut at 1.5 M characters |
| `metadata` | keywords, robots, modifiedAt, Open Graph, Twitter card, JSON-LD types, hreflang, content type, response time, size |
| `crawledAt` | ISO time (UTC) |

#### Measured (2026-10-03, see TEST_REPORT.md)

- crawlee.dev guides: 20 pages in 22 s; with sitemaps + globs: 60 pages in 67 s (59 found through the sitemap).
- books.toscrape.com: 12 pages in 12 s; `/` and `/index.html` saved once (content de-duplication).
- About 1 page per second per site with the default delay.

#### Limits

- No browser: pages that render their text with JavaScript come out (nearly) empty and are flagged `likelyJsRendered`.
- PDFs, images and other non-HTML files are skipped. No login or cookies.
- Default scope follows only URLs that start with the start URL — start from a section or the homepage, or pick `sameDomain`.
- Sites behind anti-bot challenges (Cloudflare, DataDome, ...) are logged as blocked and skipped; try a RESIDENTIAL proxy.

#### Example

```json
{"startUrls": [{"url": "https://crawlee.dev/python/docs/guides"}], "maxPages": 50, "useSitemaps": true,
 "excludeUrlGlobs": ["**/docs/next/**"]}
```

# Actor input Schema

## `startUrls` (type: `array`):

Pages where the crawl starts, e.g. https://crawlee.dev/python/docs/guides or https://example.com. Links are followed inside the crawl scope below.

## `maxPages` (type: `integer`):

Maximum number of pages saved to the dataset (one row per page).

## `maxCrawlDepth` (type: `integer`):

How many links away from a start URL the crawler may go. 0 = only the start URLs.

## `crawlScope` (type: `string`):

Which links are followed. 'Under the start URL path' (default): https://site.com/docs/ only follows https://site.com/docs/...; start from the homepage to crawl the whole site. 'Same domain': any page on the start URL's host (www. ignored). 'Domain and subdomains': also blog.site.com, docs.site.com... Include URL globs replace this setting.

## `includeUrlGlobs` (type: `array`):

Only follow links matching one of these globs (replaces the crawl scope). \*\* = anything, \* = anything except '/'. Example: https://crawlee.dev/python/docs/\*\*

## `excludeUrlGlobs` (type: `array`):

Never follow links matching these globs. Example: https://crawlee.dev/**/api/** or **/tag/**

## `useSitemaps` (type: `boolean`):

Read the sitemaps listed in robots.txt (or /sitemap.xml) and queue their in-scope URLs, so pages that are not linked from the start page are found too.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that the site's robots.txt disallows and honour its Crawl-delay (capped at 10 s). Turn off only for sites you are allowed to crawl.

## `htmlTransformer` (type: `string`):

trafilatura (default) keeps the article/documentation text and drops menus, footers and cookie banners; it falls back to readability, then to the whole cleaned page, when it finds too little. 'readability' = Mozilla Readability port. 'none' = the whole page minus navigation, header, footer and banners.

## `saveMarkdown` (type: `boolean`):

Add the main content as Markdown (headings, lists, tables, code blocks) in the 'markdown' field.

## `saveText` (type: `boolean`):

Add the main content as plain text in the 'text' field.

## `saveHtml` (type: `boolean`):

Add the main content as cleaned HTML in the 'html' field.

## `keepMarkdownLinks` (type: `boolean`):

Keep hyperlinks as [text](url) in the Markdown. Off = link text only (less noise for LLMs).

## `requestDelayMs` (type: `integer`):

Pause between two requests to the same host. Pages of one site are fetched one at a time.

## `maxConcurrency` (type: `integer`):

How many pages are processed at once across different hosts (one host is always fetched one page at a time).

## `proxyConfiguration` (type: `object`):

Apify Proxy (datacenter by default). Choose RESIDENTIAL for sites that block datacenter IPs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/python/docs/guides"
    }
  ],
  "maxPages": 20,
  "maxCrawlDepth": 5,
  "crawlScope": "startUrlPath",
  "useSitemaps": false,
  "respectRobotsTxt": true,
  "htmlTransformer": "trafilatura",
  "saveMarkdown": true,
  "saveText": true,
  "saveHtml": false,
  "keepMarkdownLinks": false,
  "requestDelayMs": 1000,
  "maxConcurrency": 2,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/python/docs/guides"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("neuro-scraper/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/python/docs/guides" }],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("neuro-scraper/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/python/docs/guides"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call neuro-scraper/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neuro-scraper/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/DQqOiJ09pYweZ0Emq/builds/AkAn1bCP9EgP5Gojs/openapi.json