Website to Markdown & Text Crawler with Spend Cap

Crawl a website and get every page as clean Markdown and plain text, ready for AI, RAG, LLM fine-tuning, search indexes or content audits. You set hard limits on money, pages, pages per domain and time, and the crawler stops there. You are never charged for pages that failed, came back empty or were duplicates.

• Paste a URL, press Start, get Markdown + text + title, description and language for each page
• Plain HTTP by default (fast and cheap), optional headless browser for JavaScript sites
• One combined .md or .txt file for the whole site if you want it
• Respects robots.txt and Crawl-delay, crawls politely

Why this one is better

Built around common complaints about website crawlers: runaway bills, runs that never stop, excludes that don't work, and paying for junk pages.

	This Actor	Typical website crawler
Fixed price per page	$1.275 per 1,000 pages (plain HTTP), shown up front	Pay-per-usage crawlers bill compute time, so the cost depends on the site and settings
Hard spend limit	"Max spend (USD)" input + the run's "Maximum cost per run" are both enforced in code; the crawler stops before a page that would go over	Often only the page limit
Hard page limit	"Max pages" defaults to 50, so a forgotten setting can't crawl 100,000 pages	Some crawlers default to millions of pages
Per-domain limit	"Max pages per domain" when you crawl many sites at once	Rarely available
Failed pages	Not charged: 404/500 errors, timeouts, DNS errors, redirect loops	Varies
Empty pages	Not charged (fewer than 10 words by default)	Often charged
Duplicates	Not charged and don't count toward Max pages: same canonical URL, same final URL after redirects, or identical text (content hash), plus tracking parameters like utm_source removed	Often deduplicated by canonical URL only, and duplicates may still count toward the limit
Auto-stop	Stops if no new page could be saved for 2 minutes ("Stop if stuck for"), plus a total "Max run time" (30 min default), and finishes cleanly before the platform timeout	Runs can keep going until the platform timeout
Include / exclude patterns	Checked on every link and on redirect targets, so excluded sections stay excluded. Simple rules: /blog/**, example.com/docs/**, or plain text like /tag/	Usually glob syntax, checked on links found on pages
Browser pages	Opt-in and priced separately, so you know which pages cost more	Often on by default
robots.txt	Respected by default, with its own user agent (WebsiteToTextBot) and Crawl-delay support	Varies

Compared with specific Apify Actors (from their public Store pages, November 2026):

• apify/web-fetch costs $1.50 per 1,000 successful fetches on the Free and Bronze plans; this Actor costs $1.275 per 1,000 pages (15% less) on those plans and less on higher plans. Web Fetch fetches one URL per request and doesn't crawl a site; this Actor follows links for you. Web Fetch's README says it charges for a site's 404 page when it converts it to Markdown; this Actor never charges for 4xx/5xx pages. Web Fetch can get past bot protection; this Actor does not try to (see FAQ).
• apify/website-content-crawler is priced by platform usage (its README estimates about $0.20 per 1,000 pages with raw HTTP and $0.50-$5 per 1,000 with a browser). Its "Max pages" default is 9,999,999 and its README says duplicate pages (same canonical URL) still count toward Max pages, and that Crawl-delay isn't supported yet. This Actor has a fixed per-page price, a hard dollar cap, a per-domain cap, content-hash deduplication and an idle auto-stop. Website Content Crawler has more advanced browser features (adaptive Firefox, scrolling, file downloads) that this Actor doesn't.

How to use it

1. Enter one or more Start URLs, for example https://docs.example.com/.
2. Choose the Crawl scope: pages under that path (default), the whole website, the whole domain with subdomains, or only the URLs you entered.
3. Set Max pages and, if you want, Max spend (USD).
4. Press Start. Results appear in the Output tab as they are saved.

Tip: to convert a list of single pages (no crawling), paste them all and set Crawl scope to Only the start URLs.

What you get

One dataset row per saved page. Every row has the same columns in the same order; empty values are null, counts are numbers and dates are ISO 8601 in UTC. A real row from a test run on 2026-11-04 (text shortened):

{
  "url": "https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting.md",
  "title": "Inspecting web pages with browser DevTools",
  "status": "ok",
  "wordCount": 1248,
  "language": null,
  "description": null,
  "httpStatus": 200,
  "fetchedVia": "http",
  "depth": 1,
  "linksCount": 10,
  "truncated": false,
  "canonicalUrl": null,
  "requestedUrl": "https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting.md",
  "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript?legacy-js-course=%2F",
  "skipReason": null,
  "error": null,
  "contentHash": "8cec9146fa007a347dfa26d6ae3209c6",
  "markdown": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.txt\n> Use this file to\n\n...",
  "text": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.tx\n\n...",
  "scrapedAt": "2026-11-05T04:08:21.940Z"
}

With Include skipped pages on, pages that were not saved get a row too, with status: "skipped", a skipReason and the other columns null (real row, shortened to the filled columns):

{
  "url": "https://example.com/does-not-exist-xyz",
  "status": "skipped",
  "httpStatus": 404,
  "depth": 0,
  "skipReason": "http-error",
  "error": "HTTP 404",
  "scrapedAt": "2026-11-05T04:08:21.383Z"
}

Views. The Output tab opens in Overview (URL, title, words, language, status, HTTP status, fetch method, depth). There are also Markdown, Plain text, Skipped pages and Detailed (every field) views. Pages with the same text, or the same final URL, are saved only once.

Changed in v1.1: crawledAt is now scrapedAt; saved rows now have status: "ok"; saved and skipped rows have the same columns in a fixed order.

Also saved in the run's key-value store:

• combined.md or combined.txt: all pages in one file, if you turned on Also save one combined file. Each page starts with a small header (source: and title:), which RAG tools can use as metadata.
• RUN_SUMMARY: pages saved and charged, total charged, why the crawl stopped, and the number of skipped pages per reason.
• SKIPPED_PAGES: every skipped (free) page with the reason, such as http-error, timeout, duplicate, empty, not-html, robots-txt, too-many-redirects.

Download the data as JSON, CSV, Excel or HTML, or use the API, Make, Zapier, n8n, LangChain, LlamaIndex or the Apify MCP server.

Pricing

Pay per event: you pay only for pages saved to your dataset, plus a tiny Actor start fee. No platform usage charges on top.

Event	Free / Bronze	Silver	Gold and up
Page (plain HTTP)	$0.001275 ($1.275 / 1,000)	$0.0010625	$0.00085
Browser page (only with Browser or Auto crawler type)	$0.0025 ($2.50 / 1,000)	$0.0022	$0.0019
Actor start (at the start of each run, one per GB of memory, minimum one)	$0.0000425	$0.0000425	$0.0000425

A page is charged as either Page or Browser page, never both. Examples: 1,000 pages over plain HTTP cost about $1.28 on the Free plan. A single-page run costs about $0.0013.

Never charged: pages that return an error (4xx/5xx), time out or can't be reached, redirect loops, pages with fewer words than your minimum, duplicates, files that aren't web pages (PDF, images, ZIP...), pages blocked by robots.txt, and pages outside your scope or excluded by your patterns.

Apify free plan: runs are limited to 100 pages each. Any paid Apify plan removes this limit.

Input options

Option	Default	What it does
Start URLs	(required)	Where the crawl starts.
Crawl scope	Pages under the start URL path	Which links to follow.
Max pages	50	Stop after this many saved (charged) pages.
Max spend (USD)	none	Stop before charging more than this.
Max pages per domain	0 (no limit)	Cap per domain when crawling many sites.
Max link depth	20	How many clicks from a start URL.
Max run time	30 min	Stop and save after this long.
Stop if stuck for	120 s	Stop if nothing new could be saved for this long.
Only crawl URLs matching / Never crawl URLs matching	none	Include / exclude patterns (see FAQ).
Crawler type	Plain HTTP	Plain HTTP, Auto, or Browser.
Respect robots.txt	on	Follow robots.txt rules and Crawl-delay.
Max parallel requests	5	Total parallel requests (each site still gets at most one request every 0.25 s).
Page timeout	30 s	Skip pages that load slower than this.
Text format	Markdown and plain text	Which text fields to return.
Also save one combined file	No	One combined.md / combined.txt for the whole crawl.
Skip pages with fewer words than	10	Treat short pages as empty (not saved, not charged).
Content area / Remove elements	automatic	CSS selectors to fine-tune what text is kept.
Keep menus, header and footer	off	Keep navigation text.
Keep image links in Markdown	off	Keep  image links.
List skipped pages in the dataset	off	Add free rows that explain each skipped page.
Max page size	10 MB	Skip bigger pages.

FAQ

How do I make sure I never spend more than a set amount? Set Max spend (USD), or set Maximum cost per run in Run options (or both; the lower one wins). The crawler checks the limit before fetching each page and stops when the next page would go over. Max pages is a second hard stop.

What exactly am I charged for? Only for pages that were downloaded successfully, had real text, were not duplicates, and were saved to your dataset. Everything else is free and is listed in SKIPPED_PAGES with the reason. A tiny "Actor start" fee ($0.0000425 per GB of memory, so one fee at the default 512 MB) is charged when the run starts.

How do include and exclude patterns work?

• https://example.com/blog/** matches everything under /blog/ (** = anything, * = anything except /).
• example.com/blog/** is the same but also matches http:// and www..
• /blog/** is matched against the path only, on any of your start sites.
• /tag/ (no *) means "the URL contains this text".
• {a,b} means "a or b", e.g. /{blog,news}/**. Excluded and non-matching links are never fetched, and a page that redirects into an excluded area is skipped too. Start URLs are always fetched.

Why was a page skipped? Open SKIPPED_PAGES (or turn on List skipped pages in the dataset). Each entry has a reason: http-error (for example 404), timeout, network-error, too-many-redirects, empty, duplicate, not-html, robots-txt, out-of-scope, excluded, too-large.

The page needs JavaScript and comes back empty. Set Crawler type to Auto (plain HTTP first; the browser only for pages that come back empty, look JavaScript-only or return 403) or Browser (every page). Browser pages cost more and need at least 1 GB memory (2 GB+ recommended for speed).

Does it get past logins, paywalls, CAPTCHAs or bot protection? No, on purpose. It only reads pages that are publicly available without logging in, it does not solve CAPTCHAs, and it does not use tricks to get around blocking. If a site blocks it, those pages are skipped and not charged.

Does it collect emails or phone numbers? No. It doesn't extract or list contact details or other personal data. It returns the page text as published. You are responsible for how you use the content and for respecting the site's terms and copyright.

Is it legal to crawl websites? Crawling publicly available pages is generally legal in many countries, but copyright, database rights, terms of service and privacy laws (such as GDPR) still apply to what you do with the data. This Actor respects robots.txt by default. If in doubt, ask a lawyer or the site owner.

Will it stop on its own if something goes wrong? Yes. It stops when it hits Max pages, Max spend, Max run time, or when nothing new could be saved for "Stop if stuck for" seconds. It also finishes cleanly about 25 seconds before the run timeout in Run options, so you keep everything saved so far.

Can I use the output for RAG or a chatbot? Yes. Use the markdown field per page, or turn on Also save one combined file to get one combined.md for the whole site. Each row has url, title, language and contentHash for metadata and change detection.

How fast is it? With plain HTTP, about 2-4 pages per second per site (it waits at least 0.25 s between requests to the same site, longer if robots.txt asks for it). Crawling several sites at once is faster. Our 20-page test crawl finished in about 7 seconds.

Something doesn't work. Open an issue on the Issues tab with the run link. Runs always finish with a clear status message and a RUN_SUMMARY.