# Website Content Crawler (Anti-Bot) - $1.20 / 1000 Pages (`scrapeunblocker/website-content-crawler`) Actor

Crawl any website, even behind Cloudflare, DataDome, PerimeterX or Akamai, and get clean Markdown and text of every page for LLMs, RAG and AI agents. Main-content extraction, URL globs, sitemaps, depth and page limits. $1.20 per 1000 pages; failed pages are free.

- **URL**: https://apify.com/scrapeunblocker/website-content-crawler.md
- **Developed by:** [Scrapeunblocker](https://apify.com/scrapeunblocker) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.20 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Content Crawler (Anti-Bot) - Markdown for LLMs

Crawl a whole website, a docs section or a blog and get the **clean content of every page as Markdown and plain text**, ready for LLMs, RAG pipelines, vector databases, AI agents and fine-tuning. Every page is loaded through an anti-bot unblocker, so sites behind **Cloudflare, DataDome, PerimeterX (HUMAN), Akamai** and similar walls are crawled like any other site: no proxies to configure, no browser settings to tune, no CAPTCHA services.

The input uses the same field names as the most popular website content crawlers (`startUrls`, `maxCrawlPages`, `maxCrawlDepth`, `includeUrlGlobs`, `excludeUrlGlobs`, `removeElementsCssSelector`, ...), and the output has the same shape (`url`, `markdown`, `text`, `crawl`, `metadata`), so switching an existing pipeline over takes minutes.

### Features

- **Gets through anti-bot walls** - Cloudflare challenges, DataDome, PerimeterX, Akamai: pages come back as real content, not as "Just a moment..." screens.
- **Clean Markdown** - headings, lists, tables, links (absolute URLs) and fenced code blocks with their language; menus, footers, cookie banners and scripts removed.
- **Main content extraction** - keeps the article or documentation body and drops sidebars, or keep the whole page; plus your own CSS selectors to remove or keep.
- **Smart crawl scope** - stays under the start URL's path by default (`https://example.com/docs/` crawls `/docs/...` only), or use include / exclude URL globs.
- **Sitemaps** - optionally read the site's sitemaps to find pages that are not linked.
- **Duplicates skipped** - pages whose canonical URL was already crawled are skipped, and not billed.
- **Page metadata** - title, description, language, canonical URL, author, keywords, Open Graph tags and JSON-LD.
- **Pay only for content** - pages that fail, are blocked, do not exist (404) or have no readable text are listed in the `ERRORS` record and **never billed**.
- **Resumes after platform migrations** - the crawl state is saved, so a moved run does not start over or bill pages twice.

### Use cases

- **RAG and chatbots** - feed product docs, help centers and knowledge bases into a vector database.
- **LLM fine-tuning and evaluation datasets** - collect clean text from many sites.
- **AI agents** - give an agent the current content of a site that blocks ordinary crawlers.
- **Content monitoring and SEO** - snapshot a site's pages, titles and descriptions.

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `startUrls` | array | Python tutorial | Where to start. The crawl stays under each start URL's path. |
| `maxCrawlPages` | integer | `100` | The most pages to load in the run (1-100000). |
| `maxCrawlDepth` | integer | `20` | Links away from a start URL to follow (0 = start URLs only). |
| `useSitemaps` | boolean | `false` | Also add in-scope URLs from the site's sitemaps. |
| `includeUrlGlobs` | array | `[]` | Crawl only links matching these globs (replaces the path rule). |
| `excludeUrlGlobs` | array | `[]` | Never crawl links matching these globs. |
| `keepUrlFragments` | boolean | `false` | Treat `#fragment` URLs as separate pages. |
| `ignoreCanonicalUrl` | boolean | `false` | Do not skip pages whose canonical URL was already crawled. |
| `outputFormats` | array | `["markdown", "text"]` | Any of `markdown`, `text`, `html` (cleaned content HTML). |
| `htmlTransformer` | string | `readableText` | `readableText` = main content only, `none` = whole page. |
| `removeElementsCssSelector` | string | nav, footer, scripts, dialogs ... | Elements removed before extraction. |
| `keepElementsCssSelector` | string | - | Keep only the elements matching this selector. |
| `removeCookieWarnings` | boolean | `true` | Remove cookie consent banners. |
| `waitForSelector` | string | - | Wait for this CSS selector before capturing a page (JavaScript-loaded content). |
| `proxyCountry` | string | - | Exit-IP country (ISO-2). Leave empty for the most reliable exit. |
| `maxConcurrency` | integer | `5` | Pages loaded at the same time (1-10). |
| `requestTimeoutSecs` | integer | `150` | Timeout for one page (30-200 s). |
| `maxRequestRetries` | integer | `2` | Retries after a server error or timeout. |

Globs: `**` matches anything, `*` anything within one path segment, `{a,b}` either. Example: `https://example.com/blog/**`.

#### Example input

Crawl a Cloudflare-protected resource center, at most 200 pages, skipping tag pages:

```json
{
  "startUrls": [{ "url": "https://www.upwork.com/resources/" }],
  "maxCrawlPages": 200,
  "excludeUrlGlobs": [{ "glob": "https://www.upwork.com/resources/tag/**" }],
  "outputFormats": ["markdown"]
}
```

### Output

One dataset item per crawled page:

```json
{
  "url": "https://docs.python.org/3/tutorial/appetite.html",
  "title": "1. Whetting Your Appetite - Python 3.15.0 documentation",
  "description": "If you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "languageCode": "en",
  "markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "text": "1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "crawl": {
    "loadedUrl": "https://docs.python.org/3/tutorial/appetite.html",
    "loadedTime": "2026-10-12T17:36:01.942018+00:00",
    "referrerUrl": "https://docs.python.org/3/tutorial/",
    "depth": 1,
    "httpStatusCode": 200
  },
  "metadata": {
    "title": "1. Whetting Your Appetite - Python 3.15.0 documentation",
    "description": "If you do much work on computers...",
    "author": null,
    "keywords": null,
    "languageCode": "en",
    "canonicalUrl": "https://docs.python.org/3/tutorial/appetite.html",
    "openGraph": { "og:title": "1. Whetting Your Appetite", "og:type": "website" },
    "jsonLd": null
  },
  "crawledAt": "2026-10-12T17:36:01.942018+00:00"
}
```

The key-value store also holds:

- `SUMMARY` - pages fetched, delivered, failed, duplicates, skipped file links, and why the crawl stopped (page limit, charge limit, run timeout).
- `ERRORS` - pages that failed, were blocked, did not exist or had no readable text, with the reason. These are **not billed**.

### Pricing

**$1.20 per 1000 pages.** You pay only for pages delivered to the dataset. Failed, blocked, empty, not-found and duplicate pages are free. Set a maximum total charge on the run to cap the cost; the crawl stops when it is reached.

### Tips

- Start from the most specific URL you can: `https://example.com/docs/` instead of `https://example.com/`.
- Use `maxCrawlPages` to cap a first run, look at the result, then raise it.
- Pages that need a moment to load their content with JavaScript: set `waitForSelector` to an element of the content, e.g. `.article-body`.
- A site that sets the same canonical URL on every page: turn on `ignoreCanonicalUrl`.

### Notes

- Only HTML pages are crawled. Links to PDFs, images, archives, office documents and other files are skipped.
- `robots.txt` crawl rules are not applied; crawl responsibly and respect the sites' terms.
- Each page is loaded once (no infinite scrolling or clicking); content that only appears after user interaction is not captured.

# Actor input Schema

## `startUrls` (type: `array`):

One or more URLs to start crawling from. By default the crawler follows links to pages under the start URL's path only: https://example.com/docs/ crawls /docs/... and nothing else on the site.

## `maxCrawlPages` (type: `integer`):

The most pages to load in this run, start URLs included. Pages that fail are counted but not billed.

## `maxCrawlDepth` (type: `integer`):

How many links away from a start URL to go. 0 = only the start URLs, 1 = start URLs and the pages they link to.

## `useSitemaps` (type: `boolean`):

Also read the site's sitemap (robots.txt Sitemap entries, else /sitemap.xml) and add its URLs that fall in the crawl scope. Finds pages that are not linked from the start URL.

## `includeUrlGlobs` (type: `array`):

Crawl only links matching these patterns, e.g. https://example.com/blog/\*\*. Setting this replaces the default rule of staying under the start URL's path. \*\* matches anything, \* anything within one path segment.

## `excludeUrlGlobs` (type: `array`):

Never crawl links matching these patterns, e.g. https://example.com/**/tag/** or \*\*?page=\*.

## `keepUrlFragments` (type: `boolean`):

Treat page.html#a and page.html#b as different pages. Leave off unless the site is a single-page app routed by #fragments.

## `ignoreCanonicalUrl` (type: `boolean`):

By default a page whose canonical URL was already crawled is skipped as a duplicate (not billed). Turn this on if the site sets one canonical URL for many different pages.

## `outputFormats` (type: `array`):

What to save for every page: Markdown (headings, lists, tables, links, code blocks), plain text, and/or the cleaned HTML of the content.

## `htmlTransformer` (type: `string`):

Main content keeps the article or documentation body and drops menus, sidebars and footers. Whole page keeps everything except the removed elements.

## `removeElementsCssSelector` (type: `string`):

Elements to delete before the content is extracted, e.g. nav, footer, .sidebar, #comments.

## `keepElementsCssSelector` (type: `string`):

Optional: keep only the elements matching this selector, e.g. article or .post-content. Pages where nothing matches are kept whole.

## `removeCookieWarnings` (type: `boolean`):

Drop cookie consent banners (OneTrust, Cookiebot, Usercentrics, Didomi, Quantcast and common generic ones) from the content.

## `waitForSelector` (type: `string`):

Optional CSS selector to wait for before the page is captured (up to 15 s), for pages that load their content with JavaScript, e.g. .article-body.

## `proxyCountry` (type: `string`):

Optional exit-IP country (ISO-2, e.g. US, DE) for sites that show different content by country. Leave empty: the exit is chosen automatically, which is the most reliable.

## `maxConcurrency` (type: `integer`):

How many pages to load at the same time (1-10). Lower it for small sites that slow down under load.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one page. Most pages take a few seconds; pages behind a hard anti-bot challenge can take a minute or more.

## `maxRequestRetries` (type: `integer`):

How many times to retry a page after a server error or timeout. Pages that do not exist (404) are not retried.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "maxCrawlPages": 100,
  "maxCrawlDepth": 20,
  "useSitemaps": false,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "htmlTransformer": "readableText",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]",
  "removeCookieWarnings": true,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 150,
  "maxRequestRetries": 2
}
```

# Actor output Schema

## `results` (type: `string`):

One item per crawled page: URL, title, Markdown, text, crawl depth and page metadata

## `summary` (type: `string`):

Pages fetched, delivered, failed, duplicates and why the crawl stopped

## `errors` (type: `string`):

Pages that failed, were blocked, were not found or had no readable text, with the reason (not billed)

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.python.org/3/tutorial/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapeunblocker/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.python.org/3/tutorial/" }] }

# Run the Actor and wait for it to finish
run = client.actor("scrapeunblocker/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ]
}' |
apify call scrapeunblocker/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapeunblocker/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uaPLPxFIDOaqNH0x1/builds/Bfk45HtYmSt7JpfjF/openapi.json