# Website to Markdown Crawler for RAG – Hard Spend Cap (`appetizing_horsefly/capped-content-crawler`) Actor

Crawl a website sitemap-first and get clean Markdown or plain text for every page, for RAG, LLM knowledge bases and content audits. You set a hard page and spend cap before the run, so a crawl never costs more than planned. Static HTML only.

- **URL**: https://apify.com/appetizing_horsefly/capped-content-crawler.md
- **Developed by:** [GM Labs](https://apify.com/appetizing_horsefly) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Capped Website Content Crawler – Sitemap-First, Hard Spend Cap

Crawl any website and get clean **Markdown or plain text** for every page, ready for RAG pipelines, LLM knowledge bases, search indexes and content audits. You set a **hard spend cap** before the run starts, so a crawl can never cost more than you planned.

### Why use it

- **Predictable cost**: price is **$0.50 per 1,000 saved pages** ($0.0005 per page). Set `maxPages` or `maxSpendUsd`; whichever is lower wins. You're billed only for saved pages. Failed, duplicate, thin and robots-blocked pages are free.
- **Sitemap-first discovery**: reads robots.txt sitemaps and `/sitemap.xml` before following links, so it finds deep pages quickly and doesn't wander.
- **Clean content**: strips navigation, headers, footers, cookie banners and scripts, and keeps the main content.
- **Polite**: respects robots.txt by default and stays within the start URL's host and path prefix.
- **Stops loudly**: if no pages are saved, the run fails with a clear reason and nothing is charged.

### Try it free in 1 click

The input form is **prefilled with a small, working example**. Press **Start** without changing anything: the run crawls 5 pages of the Apify Academy docs into clean Markdown. Our test run of exactly this input finished in about 2 seconds with 5 results. At list price that is about $0.00, which fits inside the free monthly credit of Apify's free plan. Then swap in your own values.

Prefilled input:

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 5
}
```

### Sample output

Real rows from the prefilled run on 2026-10-14 (5 of 5 results shown; long fields trimmed). Download the full dataset as JSON, CSV or Excel.

| url | title | wordCount | discoveredVia | markdown |
|---|---|---|---|---|
| https://docs.apify.com/academy | Apify Academy | Academy | Apify Documentation | 168 | sitemap | # Apify Academy Copy for LLM Learn everything about we… |
| https://docs.apify.com/academy/actor-marketi… | Name your Actor | Academy | Apify Documentation | 1220 | sitemap | On this page # Name your Actor Copy for LLM \*\*Apify's… |
| https://docs.apify.com/academy/actor-marketi… | Actor description & SEO description | Academy | Apif… | 1264 | sitemap | On this page # Actor description & SEO description Cop… |
| https://docs.apify.com/academy/actor-marketi… | Actors and emojis | Academy | Apify Documentation | 557 | sitemap | On this page # Actors and emojis Copy for LLM Using em… |
| https://docs.apify.com/academy/actor-marketi… | Importance of Actor URL | Academy | Apify Documentat… | 1132 | sitemap | On this page # Importance of Actor URL Copy for LLM \*\*… |

### Input

| Field | Default | Notes |
|---|---|---|
| `startUrls` | – | Sites or sections to crawl |
| `maxPages` | 100 | Hard page cap |
| `maxSpendUsd` | 0 | Budget in USD, converted to a page cap (0 = off) |
| `maxRunMinutes` | 30 | Time limit; everything saved so far is kept |
| `useSitemaps` / `followLinks` / `maxDepth` | true / true / 5 | Discovery settings |
| `includeUrlPatterns` / `excludeUrlPatterns` | – | Regex filters |
| `respectRobotsTxt` | true | |
| `outputFormat` | markdown | `markdown` or `text` |
| `minWords` | 30 | Thinner pages are skipped and not charged |

```json
{ "startUrls": [{ "url": "https://docs.example.com/" }], "maxSpendUsd": 1, "outputFormat": "markdown" }
```

A $1 budget means at most 2,000 pages.

### Output example

```json
{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "finalUrl": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "canonicalUrl": null,
  "title": "A Light in the Attic | Books to Scrape - Sandbox",
  "description": "...",
  "language": "en-us",
  "lastmod": null,
  "discoveredVia": "link",
  "depth": 1,
  "httpStatus": 200,
  "wordCount": 412,
  "contentHash": "3f2a…",
  "markdown": "# A Light in the Attic\n\n...",
  "crawledAt": "2026-10-10T06:00:20.000Z"
}
```

The run's key-value store also holds a `SUMMARY` record with the page cap, maximum cost, stop reason, and counts of failed, thin, duplicate and robots-blocked pages.

### Use cases

RAG and chatbot knowledge bases, documentation ingestion, SEO and content audits, competitor content monitoring, and building site archives.

### Notes

Static HTML only (no JavaScript rendering). Crawl only sites you are allowed to access.

### Use it as an MCP tool (Claude, Cursor, ChatGPT, any AI agent)

This Actor works as a tool in the [Apify MCP server](https://mcp.apify.com), so an AI agent can call it directly and you pay per result, the same as a normal run. Add this server URL to any MCP client:

```
https://mcp.apify.com/?tools=appetizing_horsefly/capped-content-crawler
```

Sign in with your Apify account (or pass your Apify API token as a Bearer header). The agent reads the input fields above from the Actor's input schema. Example request to the agent: "Crawl up to 50 pages of docs.example.com and return each page as Markdown."

### Support

Questions, a bug, or a data source you need? Use this Actor's **Issues** tab or email **gm-labs@mail.grokbot.com**. Built and maintained by GM Labs.

# Actor input Schema

## `startUrls` (type: `array`):

Sites or sections to crawl. Only pages under the same host and path prefix are crawled.

## `maxPages` (type: `integer`):

The run stops after this many pages are saved. You are never billed for more.

## `maxSpendUsd` (type: `number`):

Hard budget. Converted to a page cap at the per-page price; the lower of the two caps wins. 0 = use maxPages only.

## `maxRunMinutes` (type: `integer`):

The crawl stops cleanly after this time and keeps everything saved so far.

## `useSitemaps` (type: `boolean`):

Discover pages from robots.txt Sitemap: entries and /sitemap.xml before following links.

## `followLinks` (type: `boolean`):

Also follow in-scope links found on pages (needed when a site has no sitemap).

## `maxDepth` (type: `integer`):

Link-following depth from start URLs.

## `includeUrlPatterns` (type: `array`):

Only save pages whose URL matches one of these regexes. Empty = all in scope.

## `excludeUrlPatterns` (type: `array`):

Skip URLs matching any of these regexes.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by robots.txt.

## `outputFormat` (type: `string`):

Format of the main content.

## `concurrency` (type: `integer`):

How many HTTP requests to run in parallel. Lower it for small or slow sites.

## `requestTimeoutSecs` (type: `integer`):

Seconds to wait for each page request before skipping it.

## `minWords` (type: `integer`):

Pages with less main text than this are skipped (not charged).

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 5,
  "maxSpendUsd": 0,
  "maxRunMinutes": 30,
  "useSitemaps": true,
  "followLinks": true,
  "maxDepth": 5,
  "excludeUrlPatterns": [
    "\\.(pdf|zip|png|jpe?g|gif|svg|mp4)$",
    "/(tag|tags|author|page)/"
  ],
  "respectRobotsTxt": true,
  "outputFormat": "markdown",
  "concurrency": 10,
  "requestTimeoutSecs": 20,
  "minWords": 30
}
```

# Actor output Schema

## `results` (type: `string`):

All results in the default dataset

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "maxPages": 5,
    "excludeUrlPatterns": [
        "\\.(pdf|zip|png|jpe?g|gif|svg|mp4)$",
        "/(tag|tags|author|page)/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("appetizing_horsefly/capped-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy"],
    "maxPages": 5,
    "excludeUrlPatterns": [
        "\\.(pdf|zip|png|jpe?g|gif|svg|mp4)$",
        "/(tag|tags|author|page)/",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("appetizing_horsefly/capped-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 5,
  "excludeUrlPatterns": [
    "\\\\.(pdf|zip|png|jpe?g|gif|svg|mp4)$",
    "/(tag|tags|author|page)/"
  ]
}' |
apify call appetizing_horsefly/capped-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,appetizing_horsefly/capped-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MJKDc492EFXEcV8Yh/builds/B2XVS5OF9b2FWblDU/openapi.json