Website Content Crawler: Pages to Clean Text

Crawls any public website from a start URL, follows internal links up to a set depth, and extracts title, breadcrumbs, clean text body, and word count. Ideal for RAG ingestion and site audits.

• Pay per result row: the platform usage of the run is included, and a run that finds nothing has no result charges.
• No login, no cookies, no account on the crawled sites: public pages only.
• Export to JSON, CSV, Excel, XML or HTML, or read the rows through the Apify API.
• Start it from the Console, on a schedule, through the API, or as a tool for agents over MCP.
• Tested every day on its example input, with the rows checked, and repaired when a check fails.

What you can use it for

• Knowledge bases for search and retrieval: clean text of every page of a documentation site.
• Site audits: word count and breadcrumbs per page across a whole site.
• Content migration: export the text of an old site before a redesign.
• Competitor research: read every page of a rival site in one dataset.

What you get

Every result is one row in the run's dataset. Download it as JSON, CSV, Excel or HTML, or read it through the Apify API. The main fields:

Field	Meaning
url	Page URL
title	Title
h1	H1 Heading
breadcrumbs	Breadcrumbs
description	Description
textContent	Text content
wordCount	Word count
depth	Crawl depth

Example row:

{
  "url": "https://docs.python.org/3/library/json.html",
  "title": "json - JSON encoder and decoder",
  "h1": "json - JSON encoder and decoder",
  "breadcrumbs": [
    "The Python standard library",
    "Internet Data Handling",
    "json - JSON encoder and decoder"
  ],
  "description": null,
  "textContent": "json - JSON encoder and decoder JSON (JavaScript Object Notation), specified by RFC 7159 and by ECMA-404, is a lightweight data interchange format.",
  "wordCount": 1240,
  "depth": 1
}

How to use it

1. Open the Input tab (or call the actor through the API, see below).
2. Fill in Start URLs; the form opens with a working example.
3. Set Maximum results to cap the size and the cost of the run.
4. Click Start. Rows appear on the Output tab while the run works.
5. Download the dataset, or connect it to Google Sheets, Make, Zapier, n8n or a webhook.

Input

Field	Type	Required	Description
startUrls	array	yes	One or more page URLs to begin crawling from. Only links sharing the same origin and path prefix as each start URL are followed.
maxDepth	integer	no	How many link-hops from the start URL to follow. 0 = only the start URL itself; 2 covers most one-level doc sections. Default: 2.
maxPagesPerUrl	integer	no	Hard cap on pages fetched per start URL, independent of the global maxItems limit. Default: 50.
includeUrlPattern	string	no	Optional regex. Only URLs matching this pattern are crawled. Applied after the automatic path-prefix scope filter.
excludeUrlPattern	string	no	Optional regex. URLs matching this pattern are skipped. Example: /changelog//search to skip changelog and search pages.
maxItems	integer	no	The run stops after this many result rows. It also caps what one run can cost. Default: 100.
proxyConfiguration	object	no	Leave as is: the run starts on the cheapest network the site accepts and switches to Apify Proxy automatically when the site blocks it. Pick proxy groups here only to force one specific network.

Example input:

{
  "startUrls": [
    "https://docs.python.org/3/library/"
  ],
  "maxDepth": 1,
  "maxPagesPerUrl": 15,
  "maxItems": 20
}

Pricing

You pay per result row in the dataset (the current price per 1,000 results is on the Pricing tab). A run that finds nothing produces no rows and no result charges. Set Maximum results to cap what a single run can cost.

Limits and notes

• The crawler stays within the same origin and path prefix as each start URL. Use includeUrlPattern or excludeUrlPattern to narrow scope on large sites; combine with maxPagesPerUrl to control cost.
• Set maxDepth=0 to extract only the start page. maxDepth=1 adds all pages linked from the start page. Depths above 3 can produce thousands of pages on large sites.
• Pages behind login, paywalls, or bot walls are not accessible.
• Not affiliated with any documentation platform or website.
• Only publicly visible data is collected. No login, no paywalled content, no personal contact details.
• The actor reads only pages that the site's robots.txt leaves open to automated access.
• Sites change their pages from time to time. The actor is checked automatically and repaired when a check fails.
• Each run sends one short usage record to the developer: the Actor and run IDs, the number of results, whether the run succeeded (and if not, the kind of error, such as "blocked" or "input problem"), whether the account is on a paid Apify plan, how the run was started (for example from the Console, the API or a schedule) and a one-way fingerprint of the account (a hash, not the account ID), so that repeat runs of one account count once. It contains nothing from your input or results.

Run it through the API

Python, with the apify-client package:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("v0iddo/website-content-crawler").call(run_input={"startUrls": ["https://docs.python.org/3/library/"], "maxDepth": 1, "maxPagesPerUrl": 15, "maxItems": 20})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript, with the apify-client package:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('v0iddo/website-content-crawler').call({"startUrls": ["https://docs.python.org/3/library/"], "maxDepth": 1, "maxPagesPerUrl": 15, "maxItems": 20});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

One HTTP call that waits for the run and returns the rows (for runs that finish within 5 minutes):

curl -X POST "https://api.apify.com/v2/acts/v0iddo~website-content-crawler/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.python.org/3/library/"], "maxDepth": 1, "maxPagesPerUrl": 15, "maxItems": 20}'

Use it from an agent (MCP)

Claude, Cursor, VS Code and other MCP clients can call this actor as a tool through the Apify MCP server. Add it to the client's MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=v0iddo/website-content-crawler",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

Then ask, for example: "Crawl this documentation site and give me the clean text of every page."

FAQ

Do I need an account on the crawled sites, cookies or an API key?

No. The actor reads only what every visitor of the crawled sites can see, without logging in. You only need an Apify account to start it.

How many results can one run return?

Up to 5,000 rows per run. Without a limit it stops at 100. Set Maximum results to the number you need; large jobs can also be split into several runs or scheduled.

How am I charged?

Per result row in the dataset, at the price on the Pricing tab for your Apify plan. The platform usage of the run is included in that price. A run that finds nothing produces no rows and no result charges, and your spending limit for a run is always respected.

Why did my run return no results?

The status message of the run says why: nothing matched the input, the input was not a page the actor understands, the page is closed to automated access by robots.txt, or the site refused every network the actor tried. Check the input against the example above and run it again with a small limit.

Can I run it on a schedule?

Yes. Save your input as a task and add a schedule in the Apify Console (daily, hourly or with a cron expression). Each run gets its own dataset, so you keep a history.

Which formats can I export?

JSON, CSV, Excel, XML, HTML table and RSS from the Output tab or the API. Integrations send the rows to Google Sheets, Slack, Make, Zapier, n8n or a webhook.

Is collecting this data allowed?

The actor collects only what the crawled sites publishes to every visitor, without a login, and it reads only pages that robots.txt leaves open. Public data can still be personal data under laws such as the GDPR, and sites have their own terms: make sure your use of the data follows the rules that apply to you.

What happens when the crawled sites changes its pages?

Automated tests run the actor every day on the example input and check the rows, not only that the run finished. When a check fails, the actor is repaired and rebuilt; the build history on this page shows each update.

Something is missing or wrong. Where do I report it?

Open an issue on the Issues tab of this actor with the run link. Requests for a new field are welcome too.

Related actors

• Sitemap URL Extractor
• Website Technology Detector