Capped Website Content Crawler – Sitemap-First, Hard Spend Cap

Crawl any website and get clean Markdown or plain text for every page, ready for RAG pipelines, LLM knowledge bases, search indexes and content audits. You set a hard spend cap before the run starts, so a crawl can never cost more than you planned.

Why use it

• Predictable cost: price is $0.50 per 1,000 saved pages ($0.0005 per page). Set maxPages or maxSpendUsd; whichever is lower wins. You're billed only for saved pages. Failed, duplicate, thin and robots-blocked pages are free.
• Sitemap-first discovery: reads robots.txt sitemaps and /sitemap.xml before following links, so it finds deep pages quickly and doesn't wander.
• Clean content: strips navigation, headers, footers, cookie banners and scripts, and keeps the main content.
• Polite: respects robots.txt by default and stays within the start URL's host and path prefix.
• Stops loudly: if no pages are saved, the run fails with a clear reason and nothing is charged.

Try it free in 1 click

The input form is prefilled with a small, working example. Press Start without changing anything: the run crawls 5 pages of the Apify Academy docs into clean Markdown. Our test run of exactly this input finished in about 2 seconds with 5 results. At list price that is about $0.00, which fits inside the free monthly credit of Apify's free plan. Then swap in your own values.

Prefilled input:

{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 5
}

Sample output

Real rows from the prefilled run on 2026-10-14 (5 of 5 results shown; long fields trimmed). Download the full dataset as JSON, CSV or Excel.

url	title	wordCount	discoveredVia	markdown
https://docs.apify.com/academy	Apify Academy | Academy | Apify Documentation	168	sitemap	# Apify Academy Copy for LLM Learn everything about we…
https://docs.apify.com/academy/actor-marketi…	Name your Actor | Academy | Apify Documentation	1220	sitemap	On this page # Name your Actor Copy for LLM **Apify's…
https://docs.apify.com/academy/actor-marketi…	Actor description & SEO description | Academy | Apif…	1264	sitemap	On this page # Actor description & SEO description Cop…
https://docs.apify.com/academy/actor-marketi…	Actors and emojis | Academy | Apify Documentation	557	sitemap	On this page # Actors and emojis Copy for LLM Using em…
https://docs.apify.com/academy/actor-marketi…	Importance of Actor URL | Academy | Apify Documentat…	1132	sitemap	On this page # Importance of Actor URL Copy for LLM **…

Input

Field	Default	Notes
startUrls	–	Sites or sections to crawl
maxPages	100	Hard page cap
maxSpendUsd	0	Budget in USD, converted to a page cap (0 = off)
maxRunMinutes	30	Time limit; everything saved so far is kept
useSitemaps / followLinks / maxDepth	true / true / 5	Discovery settings
includeUrlPatterns / excludeUrlPatterns	–	Regex filters
respectRobotsTxt	true
outputFormat	markdown	markdown or text
minWords	30	Thinner pages are skipped and not charged

{ "startUrls": [{ "url": "https://docs.example.com/" }], "maxSpendUsd": 1, "outputFormat": "markdown" }

A $1 budget means at most 2,000 pages.

Output example

{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "finalUrl": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "canonicalUrl": null,
  "title": "A Light in the Attic | Books to Scrape - Sandbox",
  "description": "...",
  "language": "en-us",
  "lastmod": null,
  "discoveredVia": "link",
  "depth": 1,
  "httpStatus": 200,
  "wordCount": 412,
  "contentHash": "3f2a…",
  "markdown": "# A Light in the Attic\n\n...",
  "crawledAt": "2026-10-10T06:00:20.000Z"
}

The run's key-value store also holds a SUMMARY record with the page cap, maximum cost, stop reason, and counts of failed, thin, duplicate and robots-blocked pages.

Use cases

RAG and chatbot knowledge bases, documentation ingestion, SEO and content audits, competitor content monitoring, and building site archives.

Notes

Static HTML only (no JavaScript rendering). Crawl only sites you are allowed to access.

Use it as an MCP tool (Claude, Cursor, ChatGPT, any AI agent)

This Actor works as a tool in the Apify MCP server, so an AI agent can call it directly and you pay per result, the same as a normal run. Add this server URL to any MCP client:

https://mcp.apify.com/?tools=appetizing_horsefly/capped-content-crawler

Sign in with your Apify account (or pass your Apify API token as a Bearer header). The agent reads the input fields above from the Actor's input schema. Example request to the agent: "Crawl up to 50 pages of docs.example.com and return each page as Markdown."

Support

Questions, a bug, or a data source you need? Use this Actor's Issues tab or email gm-labs@mail.grokbot.com. Built and maintained by GM Labs.