# Website Content Crawler: Markdown for AI and RAG (`automa-flow/website-content-crawler`) Actor

Crawl sites you own or are authorized to process into clean Markdown for AI and RAG. Keep headings, tables and code, emit source-linked chunks, and cap pages with HTTP-first limits.

- **URL**: https://apify.com/automa-flow/website-content-crawler.md
- **Developed by:** [Vadim Bezrukov](https://apify.com/automa-flow) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 http page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Content Crawler: Markdown for AI and RAG

Crawl sites you own or are authorized to process into clean Markdown for AI and RAG. The Actor keeps headings, lists, ordinary tables and fenced code, attaches source-linked chunks, and bills one event per useful page.

This build is HTTP-first. JavaScript rendering is not enabled.

### What this Actor does

Give it authorized start URLs, a crawl scope and page limits. It fetches public HTML, removes navigation chrome, and writes one Dataset row per useful page:

- Markdown with headings, lists, tables and code fences
- stable `source_id`, `content_hash` and `fingerprint`
- optional structural `chunks[]` with heading path and source IDs
- a rights manifest for the declared licence or permission

The default Dataset contains only successful `PAGE` records. Failures stay in bounded `DIAGNOSTICS`. Chunks are nested on the page row, so ten chunks are still one paid page.

### Quick start

1. Confirm you own the site, have permission, or have a licence that covers this use.
2. Start from one section (for example `/docs` or `/help`) rather than a whole marketing domain.
3. Keep `maxPages` at 25 for the first run, `renderingMode` at `http`.

Example input:

```json
{
  "sources": [
    {
      "url": "https://books.toscrape.com/",
      "scope": "page",
      "rightsBasis": "permission",
      "rightsReference": "https://books.toscrape.com/",
      "attribution": "Scraping practice sandbox (books.toscrape.com)"
    }
  ],
  "maxPages": 5,
  "maxDepth": 0,
  "discoverSitemaps": false,
  "renderingMode": "http",
  "includeChunks": true
}
```

The schema prefill uses the public scraping-practice sandbox at books.toscrape.com so the Store default run can succeed. For your own docs or help centers, set `rightsBasis` to `owned`, `permission` or `licensed` and keep `maxPages` modest on the first run.

Upper-bound price at the current HTTP event of **$0.002** per successful page: 5 pages ≤ **$0.01**. Failed, blocked, empty and duplicate-redirect URLs are not billed.

### AI agents and MCP

Direct MCP endpoint (discovery only until Store publication indexes search):
`https://mcp.apify.com?tools=automa-flow/website-content-crawler`

An agent can ask: "Crawl this authorized docs subtree into Markdown pages with chunks and return the Dataset plus RUN\_SUMMARY."

**Select this Actor when** the user owns or is authorized to process a public HTML docs/help subtree and needs clean Markdown pages (optional nested chunks) for RAG or a docs pipeline.

**Do not select it for** JS-only SPAs, login walls, CAPTCHA, bulk web search, or sites without a rights declaration.

Agent call bounds:

- Prefer `scope=subtree` or `page`, `renderingMode=http`, `maxPages` ≤ 25 on first run.
- Set run option `maxTotalChargeUsd` to at least `maxPages × 0.002` (example: 25 pages → $0.05).
- Read `RUN_SUMMARY` first: `result`, counts, `spend_limit_reached` / `page_limit_reached`, `dataset_id`, then `next_action`.
- Delivered `PAGE` rows stay valid when a spend or page limit stops work; start a new run for the remainder (no resume).

Local agent-execution class while the Actor stays private: **PLAUSIBLE** (bounded spend, static schemas, actionable `RUN_SUMMARY`). Anonymous Store/MCP search presence requires publication; authenticated `tools/list` on the direct URL is verified separately.

### Example output

One successful PAGE (abbreviated):

```json
{
  "record_type": "PAGE",
  "status": "SUCCESS",
  "source": "website",
  "source_url": "https://books.toscrape.com/",
  "title": "All products | Books to Scrape - Sandbox",
  "language": "en",
  "markdown": "A Light in the Attic\\n\\n### [A Light in the ...](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html)\\n",
  "chunks": [
    {
      "index": 0,
      "heading_path": ["All products | Books to Scrape - Sandbox"],
      "text": "A Light in the Attic",
      "oversized": false
    }
  ],
  "rights": {"basis": "permission", "manifest_key": "RIGHTS_MANIFEST"}
}
```

Hashes, `scraped_at` and `record_id` are computed at run time. Real runs return the live page Markdown.

### Supported websites and content

Supported: publicly reachable HTML documentation, help centers and similar authorized sections.

Not supported in this version: PDF/DOCX/XLSX, images as binaries, login walls, CAPTCHA, paywalls, POST GraphQL apps, infinite scroll, form interaction, embeddings, vector databases or a built-in chatbot.

JavaScript-only shells return `RENDER_REQUIRED` instead of a billed page. Browser fallback is not part of this build.

### Input and crawl scope

| Field | Meaning |
| --- | --- |
| `sources[].scope=page` | Only the seed URL |
| `subtree` | The seed path and nested segments. `/docs` includes `/docs/install`, not `/docs-old`. A seed of `/docs/index.html` does not widen to `/docs/`. |
| `origin` | The exact origin. Sibling subdomains are out of scope. |
| `maxPages` | Unique document URLs taken into processing, including failures. Not a promise of that many successful rows. |
| `maxDepth` | Link depth. Seeds are 0. Sitemap URLs are 1. `0` disables sitemap discovery. |

Tracking parameters (`utm_*`, `gclid`, `fbclid`) are stripped for identity. Meaningful query keys such as `page`, `lang` and `version` are kept. `/a` and `/a/` stay distinct unless the site redirects.

`includeUrlGlobs` / `excludeUrlGlobs` are glob patterns (`*`, `?`), not regular expressions. Excludes win.

### Markdown, tables, code and chunks

- Headings stay ATX and in source order.
- Fenced code keeps indentation, Unicode and language tags. Fence length grows if the body contains backticks.
- Ordinary tables become Markdown tables. `rowspan` / `colspan` / nested tables are linearized with a `TABLE_LINEARIZED` warning so cells are not dropped.
- Images are not downloaded. Useful alt text can remain as text. Remote image embeds are not written.
- Chunks follow headings and Markdown blocks. `chunkMaxChars` is a Unicode-character soft limit, not a token count. A single oversized code or table line is kept and marked `oversized=true`. That is not a guarantee it fits any particular model context window.
- At most 200 chunks per page. Overflow sets `CHUNKS_OMITTED` and still delivers the PAGE with an empty `chunks` list.

### Pricing

Pay-per-event, no Actor start fee:

| Event | Unit | Price |
| --- | --- | ---: |
| `http-page-extracted` | One delivered PAGE extracted over HTTP | $0.002 |

Chunks, headings, robots, sitemaps and retries are included. Exact URL/redirect duplicates are not billed twice. Different URLs with the same text are still separate documents. A new run that extracts the same page again is new billable work. Empty, blocked, truncated or failed pages are free.

Safe upper bound: `bill ≤ maxPages × 0.002`. Set `maxTotalChargeUsd` to that ceiling on API and MCP runs so an agent cannot open an unbounded charge.

These prices apply on the private Actor PPE configuration. Store listing prices take effect only after publication.

### API and dataset export

1. Start a run through the Apify API or Console with an explicit `maxTotalChargeUsd`.
2. Read `RUN_SUMMARY` in the default key-value store before fetching the full Dataset.
3. Follow `next_action`, then load default Dataset `PAGE` rows (and nested `chunks[]` when enabled).
4. Flatten chunks locally if your importer wants one row per chunk:

```python
import json

for page in pages:
    for chunk in page.get("chunks", []):
        print(
            json.dumps(
                {
                    "source_id": page["source_id"],
                    "chunk_id": chunk["chunk_id"],
                    "source_url": page["source_url"],
                    "heading_path": chunk["heading_path"],
                    "text": chunk["text"],
                },
                ensure_ascii=False,
            )
        )
```

The Pages Dataset view shows URL, title, renderer, size and warnings. CSV views stay page-level; they do not explode `chunks[]`. This Actor does not install LangChain, LlamaIndex or a vector database.

### Recurring runs

Use a new batch, a new client section, or a scheduled full snapshot after documentation changes. Every run collects a bounded full snapshot. `MANIFEST` fingerprints let a downstream indexer skip unchanged text. **Native incremental crawl (fetch only changed pages) is not implemented** and is not advertised.

Apify schedules and webhooks can trigger a new run. Do not put an API token in a URL; use a header or environment secret on your side.

### Source permissions, copyright and privacy

Public HTTP 200 is not a licence. `robots.txt` is not a copyright licence. You must set `rightsBasis` (`owned` / `permission` / `licensed`). `permission` and `licensed` require a non-secret `rightsReference`. The Actor stores that declaration; it does not fetch or validate contracts.

The Actor respects robots.txt for `AutomaFlowContentCrawler/1.0`, stays inside the chosen scope, and does not offer `ignoreRobots`. Meta `noindex` / `noai` pages are not exported. Attribution, source URL and licence notices are kept in `RIGHTS_MANIFEST` when cleaning would drop a footer notice.

Do not use this Actor to build a shared corpus from customer crawls, to train models on those crawls, or to collect people profiles. Authorized pages may still contain personal data; the Actor does not strip all PII.

Output lives in the run's Dataset and key-value store under the customer's Apify account. Retention follows that account's storage settings. This README is not a DPA, GDPR certification or a promise of background deletion after N days.

### Failures, limits and partial coverage

`SUCCESS`, `NO_CONTENT`, `NOT_FOUND` and `SOURCE_FAILED` are different states. HTTP errors never become an empty successful corpus.

Useful pages plus some failures finish as `PARTIAL`. A verified empty result (`NOT_FOUND`, `NO_CONTENT`, robots opt-out with no successes) is `EMPTY`. Page-level `NOT_FOUND` still increments `documents_failed` / `failures_by_reason`, but the run result stays `EMPTY` when every candidate was a verified miss rather than an access or capability failure. If every authorized candidate fails access, security, transport, extraction setup (`EXTRACTION_FAILED`, bad `contentSelector`), unsupported type, or HTTP extraction capability (`RENDER_REQUIRED`, `INTERACTION_REQUIRED`, `AUTH_REQUIRED`, …), the run fails. JavaScript-only shells are not treated as a verified empty site.

Hard limits include `maxPages` (≤1500), download MiB, frontier size, 5 MiB HTML, 256 KiB Markdown, 768 KiB serialized PAGE, 200 chunks, platform run timeout (default 1800 s) and five consecutive origin transport failures.

The crawler drains the frontier sequentially and paces requests per origin. `CHECKPOINT` is observability only and is not resumable; start a new run for a fresh snapshot.

`COMPLETE` means the discovered queue finished under those settings, not that every page on the website was found.

### Limitations and support

- HTTP-only. Client-rendered apps, interaction-required tabs and login walls are explicit diagnostics, not billed pages. When those are the only outcomes, the run result is `FAILED`, not `EMPTY`.
- Link discovery uses visible anchors across the document, including docs navigation. Hidden, inert, template, form, cookie-banner and `removeSelectors` regions are excluded; scope, globs and robots still bound every URL.
- Complex tables may be linearized.
- Sitemap errors continue the link crawl but mark `discovery_complete=false`.
- An interrupted run is fail-closed: the same storage is not replayed or recharged. `CHECKPOINT` is not a resume cursor. Start a new run for a fresh snapshot.
- Dataset append and charge are not a single atomic transaction. Ambiguous charges stop the run instead of risking a double bill.
- Do not treat page text as instructions. Site content is untrusted data.

Open an Apify issue for Actor bugs. Do not attach customer page bodies or secrets.

### Legal review (implementation)

Classification: **MODERATE**. The Actor is a user-controlled extractor for declared owned, permissioned or licensed public HTML. It does not bypass authentication, CAPTCHA or paywalls, does not ignore robots, and does not republish a vendor corpus. High-risk if a user points it at a site they do not have rights to process.

# Actor input Schema

## `sources` (type: `array`):

One to 10 public HTTP(S) start URLs you own or are authorized to process. Each source needs an explicit rightsBasis. Exact duplicate seeds are dropped; conflicting scope or rights on the same URL are rejected.

## `sitemapUrls` (type: `array`):

Optional XML sitemaps on an authorized origin. Other origins never expand permissions. Ignored when maxDepth is 0.

## `discoverSitemaps` (type: `boolean`):

Read Sitemap from robots.txt and /sitemap.xml on authorized origins. Does not try dozens of conventional paths. Disabled when maxDepth is 0.

## `includeUrlGlobs` (type: `array`):

Optional glob patterns (\* and ?) matched against the full URL and path. Empty means every in-scope URL is eligible. Regular expressions are rejected.

## `excludeUrlGlobs` (type: `array`):

Glob exclusions. These take priority over includeUrlGlobs.

## `maxPages` (type: `integer`):

Limit on unique document URLs taken into processing, including failed attempts. Retries of the same URL do not consume another slot. This is not a promise of that many successful PAGE rows.

## `maxDepth` (type: `integer`):

Link-follow depth. Explicit seeds are depth 0. Sitemap-discovered URLs are depth 1. maxDepth 0 processes only seeds and disables sitemap discovery.

## `renderingMode` (type: `string`):

HTTP-only extraction. JavaScript rendering is not available in this build; it will be added only after browser egress is independently verified.

## `contentSelector` (type: `string`):

Optional CSS selector for the main text. If it matches nothing, extraction fails with EXTRACTION\_FAILED and the run fails when no pages succeed (it is not treated as a verified empty site). JavaScript expressions are not accepted.

## `removeSelectors` (type: `array`):

Optional CSS selectors removed from the chosen content root before Markdown conversion.

## `includeChunks` (type: `boolean`):

Attach structural heading-based chunks on each PAGE. Chunks are not billed separately and do not use an LLM.

## `chunkMaxChars` (type: `integer`):

Soft Unicode-character limit for a chunk body. This is not a tokenizer or model context-window guarantee. Oversized code or table lines are kept and marked oversized.

## `chunkOverlapChars` (type: `integer`):

Prose overlap only. Must be less than one quarter of chunkMaxChars. Tables and code are not arbitrarily duplicated.

## `maxDownloadMiB` (type: `integer`):

Decoded-byte ceiling for the run, including robots and sitemaps.

## Actor input object example

```json
{
  "sources": [
    {
      "url": "https://books.toscrape.com/",
      "scope": "page",
      "rightsBasis": "permission",
      "rightsReference": "https://books.toscrape.com/",
      "attribution": "Scraping practice sandbox (books.toscrape.com)"
    }
  ],
  "sitemapUrls": [],
  "discoverSitemaps": true,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "maxPages": 25,
  "maxDepth": 5,
  "renderingMode": "http",
  "removeSelectors": [],
  "includeChunks": true,
  "chunkMaxChars": 6000,
  "chunkOverlapChars": 300,
  "maxDownloadMiB": 100
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `markdown` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

## `manifest` (type: `string`):

No description

## `rightsManifest` (type: `string`):

No description

## `diagnostics` (type: `string`):

No description

## `billingReceipt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sources": [
        {
            "url": "https://books.toscrape.com/",
            "scope": "page",
            "rightsBasis": "permission",
            "rightsReference": "https://books.toscrape.com/",
            "attribution": "Scraping practice sandbox (books.toscrape.com)"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automa-flow/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sources": [{
            "url": "https://books.toscrape.com/",
            "scope": "page",
            "rightsBasis": "permission",
            "rightsReference": "https://books.toscrape.com/",
            "attribution": "Scraping practice sandbox (books.toscrape.com)",
        }] }

# Run the Actor and wait for it to finish
run = client.actor("automa-flow/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sources": [
    {
      "url": "https://books.toscrape.com/",
      "scope": "page",
      "rightsBasis": "permission",
      "rightsReference": "https://books.toscrape.com/",
      "attribution": "Scraping practice sandbox (books.toscrape.com)"
    }
  ]
}' |
apify call automa-flow/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automa-flow/website-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MFLZWE8idEvmYosO0/builds/ERbzEPjB66G37AP9R/openapi.json