# Website Crawler Scraper (`automation-lab/website-content-crawler`) Actor

Crawl public websites into page-level metadata, links, clean text, and Markdown for search indexes and LLM knowledge bases.

- **URL**: https://apify.com/automation-lab/website-content-crawler.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.26 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Crawler Scraper

Turn public website pages into clean text, Markdown, metadata, and link records with a bounded **website crawler**.

Website Crawler Scraper starts from one or more URLs, follows same-site links within your page and depth limits, and writes one structured dataset record per fetched page. The output is ready for search indexing, LLM knowledge bases, RAG ingestion, content inventories, and recurring documentation refreshes.

### What does Website Crawler Scraper do?

The Actor:

- crawls up to 20 public websites in one run;
- follows internal links to a configurable depth;
- extracts title, description, canonical URL, language, and headings;
- exports unique internal and external links;
- removes common navigation and layout boilerplate;
- returns both normalized visible text and structured Markdown;
- calculates a stable SHA-256 content hash for change detection;
- records HTTP status, response time, crawl depth, and failures;
- accepts an optional CSS selector for the main content region;
- supports optional user-configured Apify Proxy without an automatic paid fallback.

It uses lightweight HTTP requests rather than a browser. That makes it a practical website crawler tool for public HTML and Markdown pages where JavaScript rendering is not required.

### Who is it for?

**AI and RAG teams** can prepare source-attributed Markdown for embeddings, vector databases, assistants, and knowledge-base refreshes.

**Search engineers** can create page-level indexing documents with URLs, metadata, normalized text, and content hashes.

**Documentation teams** can export a bounded section of public docs before migration, analysis, or scheduled comparison.

**Developers and data teams** can feed a stable JSON dataset into Apify integrations, webhooks, cloud storage, or an ETL pipeline.

**Content operators** can inventory public pages and detect changed text by comparing `contentHash` between scheduled runs.

### Why use this Actor?

A general web request gives you HTML. This Actor gives you an integration-ready page record.

- **Two useful content formats:** `cleanedText` for indexing and `markdown` for structure-aware downstream tools.
- **Explicit crawl boundaries:** `maxPages`, `maxDepth`, and `maxLinksPerPage` prevent accidental unbounded crawls.
- **Multi-site runs:** refresh several unrelated public sites through one input and one dataset.
- **Stable change signal:** compare `contentHash` instead of diffing layout-heavy HTML.
- **Transparent failures:** blocked, unavailable, non-HTML, or selector-mismatched pages return typed error records instead of disappearing silently.
- **SSRF-safe requests:** private, local, credential-bearing, and reserved-network targets are rejected, including redirects.
- **No hidden paid fallback:** direct HTTP is the default; a proxy is used only when you configure one.

### What data does it extract?

| Field | Meaning |
| --- | --- |
| `url` | Normalized URL requested by the crawler |
| `finalUrl` | Final URL after validated redirects |
| `startUrl` | Supplied website root that led to the page |
| `depth` | Internal-link depth from the start URL |
| `statusCode` | Final HTTP status when a response was received |
| `contentType` | Response content type |
| `title` | HTML title or first Markdown heading |
| `metaDescription` | HTML meta description when available |
| `canonicalUrl` | Resolved HTML canonical URL when available |
| `language` | HTML language attribute when available |
| `headings` | Ordered H1-H6 level and text pairs |
| `internalLinks` | Unique links inside the website scope |
| `externalLinks` | Unique links to other websites |
| `cleanedText` | Whitespace-normalized visible content |
| `markdown` | Selected content converted to Markdown, or source Markdown |
| `wordCount` | Approximate cleaned-text word count |
| `contentHash` | SHA-256 hash of cleaned text |
| `responseTimeMs` | Request and redirect time in milliseconds |
| `fetchedAt` | ISO timestamp for the page record |
| `error` | Concise per-page failure reason, otherwise `null` |

Fields that a page does not expose are `null` or empty arrays. For example, a server-provided Markdown response does not have HTML canonical or language elements.

### How to crawl a website

1. Open the Actor input page.
2. Add one or more public HTTP(S) pages under **Start URLs**.
3. Choose a small **Maximum pages** value for your first run.
4. Set **Maximum link depth** to `0` for only supplied pages, `1` for their direct internal links, or a higher bounded depth.
5. Keep query parameters disabled unless they identify genuinely different content.
6. Optionally set a content CSS selector such as `main` or `article`.
7. Run the Actor and open the **Website content** dataset view.
8. Export JSON, CSV, Excel, XML, or connect the dataset to your next system.

Start with low concurrency when crawling a small site. Increase it only when the website can handle the request rate.

### Input parameters

#### `startUrls`

Required. One to 20 public HTTP(S) start pages. Entries can be request-list objects or URL strings through the API.

Each start URL defines its own crawl scope. Links remain on that hostname unless `includeSubdomains` is enabled.

#### `maxPages`

Maximum number of page records across the entire run. Default: `20`. Range: `1` to `10000`.

The limit includes success and failure records, so upstream problems cannot make the crawl exceed its requested scope.

#### `maxDepth`

Maximum internal-link depth. Default: `2`. Range: `0` to `10`.

Depth `0` fetches only the supplied URLs. Depth `1` also fetches links found directly on those pages.

#### `maxConcurrency`

Maximum requests in flight. Default: `5`. Range: `1` to `20`.

#### `maxLinksPerPage`

Maximum internal links accepted from one page for future crawling. Default: `20`.

#### `includeSubdomains`

When enabled, links such as `docs.example.org` can be followed from `example.org`. The default is `false`.

#### `followQueryParameters`

When enabled, distinct query strings may become distinct crawl targets. Common tracking parameters are always removed. The default is `false` to reduce duplicate and faceted pages.

#### `contentSelector`

CSS selector for the content converted to text and Markdown. The first match is used. Default: `body`.

Use `main`, `article`, or a source-specific selector to reduce layout content. A selector that matches nothing creates an explicit page error.

#### `removeSelectors`

Optional CSS selectors removed before conversion. When omitted, the Actor removes scripts, styles, templates, SVG, canvas, iframes, navigation, headers, footers, asides, and aria-hidden elements.

#### `requestTimeoutSecs`

Per-page timeout from `5` to `120` seconds. Default: `30`.

#### `proxyConfiguration`

Optional Apify Proxy or custom proxy configuration. Direct requests are used by default. The Actor never turns on residential proxy transfer automatically.

### Example input: website to Markdown

```json
{
  "startUrls": [
    { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }
  ],
  "maxPages": 5,
  "maxDepth": 1,
  "maxConcurrency": 2
}
```

For an LLM-focused documentation crawl, add `"contentSelector": "main"` when the source serves conventional HTML.

### Example output

```json
{
  "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview",
  "finalUrl": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview",
  "startUrl": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview",
  "depth": 0,
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "title": "Overview of HTTP - HTTP | MDN",
  "metaDescription": "HTTP is a client-server protocol...",
  "canonicalUrl": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview",
  "language": "en-US",
  "headings": [
    { "level": 1, "text": "Overview of HTTP" }
  ],
  "internalLinks": [
    "https://developer.mozilla.org/en-US/docs/Web/HTTP"
  ],
  "externalLinks": [],
  "cleanedText": "Overview of HTTP HTTP is a client-server protocol...",
  "markdown": "# Overview of HTTP\n\nHTTP is a client-server protocol...",
  "wordCount": 2241,
  "contentHash": "a stable 64-character SHA-256 value",
  "responseTimeMs": 328,
  "fetchedAt": "2026-01-15T12:00:00.000Z",
  "error": null
}
```

The snippet is shortened for readability. Real Markdown and link arrays contain the page's extracted values.

### How much does it cost to crawl website pages?

The Actor uses pay per event pricing:

- one `start` event per run;
- one `item` event for each successfully extracted page containing useful text;
- failure and unsupported-content records have no separate item charge.

The final price depends on your Apify plan tier. At the current BRONZE rates, a run has a **$0.005 start fee plus $0.007104 per useful page**.

| Useful pages | BRONZE example cost |
| ---: | ---: |
| 1 | about 1.21 cents |
| 10 | about 7.60 cents |
| 100 | about 71.54 cents |
| 1,000 | about 710.90 cents |

Optional proxy traffic and Apify platform usage may be billed according to your account and proxy configuration. The Actor does not enable paid proxy traffic on its own.

### Build an LLM or RAG ingestion workflow

A repeatable knowledge-base refresh can:

1. run this Actor on a schedule;
2. fetch the default dataset through the API;
3. discard rows with `error` or empty `cleanedText`;
4. compare `contentHash` with the prior successful run;
5. chunk only new or changed `markdown`;
6. attach `url`, `title`, and `fetchedAt` as source metadata;
7. upsert vectors into your search or RAG store.

The Actor returns page-level documents. It does not create embeddings, choose chunk sizes, or write to a particular vector database, so you retain control over your retrieval pipeline.

### Refresh a search index

Use `cleanedText` as the searchable body and retain:

- `url` as the document key;
- `title` and `metaDescription` for result display;
- `language` for analyzers;
- `canonicalUrl` for duplicate handling;
- `contentHash` for incremental updates;
- `internalLinks` for graph or discovery analysis.

Scheduled runs are independent snapshots. Store the prior dataset or hash map in your own downstream system when you need historical comparisons.

### API usage

Replace `YOUR_TOKEN` with an Apify API token stored securely. Do not commit tokens to source code.

#### cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-content-crawler/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview"}],"maxPages":5,"maxDepth":1}'
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-content-crawler').call({
  startUrls: [{ url: 'https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview' }],
  maxPages: 5,
  maxDepth: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map(({ url, title, contentHash }) => ({ url, title, contentHash })));
```

#### Python

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/website-content-crawler').call(run_input={
    'startUrls': [{'url': 'https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview'}],
    'maxPages': 5,
    'maxDepth': 1,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print([(item['url'], item['title'], item['contentHash']) for item in items])
```

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-content-crawler"
```

**Claude Desktop, Cursor, and VS Code setup:** add this equivalent HTTP MCP server configuration in the client's MCP settings:

```json
{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=automation-lab/website-content-crawler"
    }
  }
}
```

Example prompts:

- "Crawl these five documentation pages and return clean Markdown with source URLs."
- "Create a bounded 20-page content inventory for this public website."
- "Refresh these two public documentation sites and show pages whose content hashes changed from my saved list."

MCP runs the same Actor input contract and returns links to its run and dataset.

### Limits and responsible crawling

- Only public HTTP(S) targets are supported.
- Private, loopback, link-local, reserved, local-name, credential-bearing, and unsafe redirect targets are rejected.
- The Actor does not sign in, solve CAPTCHAs, bypass access controls, or render JavaScript applications.
- Only HTML, XHTML, plain text, and Markdown responses become useful content records.
- Binary documents, media, and archive links are not crawled.
- The crawler follows links found in fetched content; it does not promise complete sitemap coverage.
- Query-heavy calendars, faceted navigation, and duplicate URL forms may require stricter limits.
- Common layout elements are removed heuristically. Use `contentSelector` and `removeSelectors` when a site's structure requires source-specific cleanup.
- The Actor uses bounded transient retries for network failures, HTTP 429, and server errors.

Use conservative limits and concurrency. Review the target site's terms, robots guidance, and acceptable-use rules before crawling.

### Troubleshooting

#### Why did I get only one page?

Check `maxDepth`. A value of `0` intentionally fetches only start URLs. Also inspect `internalLinks`: JavaScript-generated navigation may not exist in the HTTP response, or the page may link to a different hostname.

#### Why is `error` populated?

The record contains the exact bounded failure, such as a timeout, unsupported content type, invalid selector, HTTP error, or network exhaustion. Review `statusCode`, `contentType`, and run logs before retrying.

#### Why is the Markdown empty or noisy?

Use a narrower `contentSelector`, commonly `main` or `article`. Add site-specific cookie banners, menus, or sidebars to `removeSelectors`.

#### Why was my URL rejected?

The Actor intentionally blocks private networks, localhost, reserved IP ranges, credential-bearing URLs, and redirects to those targets. Supply a public HTTP(S) URL.

#### Should I enable a proxy?

Try direct access first. Configure a proxy only when the public site requires it and you are authorized to use it. There is no automatic proxy fallback.

### Legality and privacy

You are responsible for deciding whether you may crawl a target and process its content.

- Crawl public pages you are authorized to access.
- Follow applicable website terms, robots instructions, rate limits, copyright rules, and database rights.
- Do not use the Actor to bypass authentication or technical access controls.
- Avoid collecting personal or sensitive data unless you have a lawful purpose and appropriate safeguards.
- Set page, depth, and concurrency limits that are proportionate to the website.
- Secure exported datasets and delete them when they are no longer needed.

This Actor provides technical extraction tooling, not legal advice.

### Related Automation Lab Actors

- [Multi-page On-page SEO Audit Crawler](https://apify.com/automation-lab/multi-page-on-page-seo-audit) for deterministic titles, indexability, content signals, and SEO issue flags.
- [Sitewide Broken Link Checker](https://apify.com/automation-lab/sitewide-broken-link-checker) for validating internal and external link targets.
- [Multi-Site Article Content Extractor](https://apify.com/automation-lab/multi-site-article-content-extractor) when you already have article URLs and need article-specific fields.
- [Website HTML & Text Change Monitor](https://apify.com/automation-lab/website-html-text-change-monitor) for persisted versioned snapshots and machine-readable additions or removals.

### FAQ

#### Is this a Google website crawler?

No. It crawls public pages from the URLs you supply. It does not imitate Googlebot, submit pages to Google, or access Google Search Console.

#### Can it crawl several websites in one run?

Yes. Add up to 20 start URLs. `maxPages` is shared across the run, and each start URL keeps its own website scope.

#### Does it render JavaScript?

No. The Actor is optimized for static HTML, server-rendered HTML, plain text, and Markdown. Use a browser-focused crawler when essential content exists only after client-side rendering.

#### Does it compare runs automatically?

No. `contentHash` makes comparison efficient, but snapshots are independent. Compare hashes in your database, automation, or scheduled workflow.

#### Are failed pages charged as items?

No. The `item` event is charged only for successfully extracted pages with useful cleaned text. A typed error row can still appear in the dataset so failures remain visible.

#### Can it ignore URL query parameters?

Yes. Query parameters are ignored by default. Enable `followQueryParameters` only when different query strings represent distinct pages.

# Changelog

This Actor's version history is a separate document: https://apify.com/automation-lab/website-content-crawler/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

One or more public website pages where crawling starts. Each site stays within its own hostname by default.

## `maxPages` (type: `integer`):

Maximum total page records across all supplied websites.

## `maxDepth` (type: `integer`):

How many internal-link levels to follow from each start URL. Use 0 for only the supplied pages.

## `maxConcurrency` (type: `integer`):

Maximum pages fetched at once. Lower this for small or sensitive websites.

## `maxLinksPerPage` (type: `integer`):

Maximum internal links accepted from any one page for further crawling.

## `includeSubdomains` (type: `boolean`):

Follow links to subdomains of each supplied website.

## `followQueryParameters` (type: `boolean`):

Treat URLs with different query strings as different pages. Tracking parameters are always removed.

## `contentSelector` (type: `string`):

Extract text and Markdown from the first element matching this CSS selector.

## `removeSelectors` (type: `array`):

CSS selectors removed before clean text and Markdown conversion. Leave unset for navigation, headers, footers, scripts, and other boilerplate defaults.

## `requestTimeoutSecs` (type: `integer`):

Per-page request timeout in seconds.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy or custom proxies. No paid proxy fallback is enabled automatically.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxPages": 20,
  "maxDepth": 2,
  "maxConcurrency": 5,
  "maxLinksPerPage": 20,
  "includeSubdomains": false,
  "followQueryParameters": false,
  "contentSelector": "body",
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

Default dataset items in the page-content overview view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxPages": 20
}' |
apify call automation-lab/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CXYRTEnThX5jSF5ts/builds/zx9GmA9hSzgobeEAa/openapi.json