Crawl4AI LLM Scraper — watch 500 pages for $1

Point it at a site. It renders every page in a real browser, gives you LLM-ready Markdown, and then — in watch mode — keeps checking those pages for you at $0.002 each. 500 pages watched per dollar. You only pay browser prices for the pages that actually changed.

Powered by the open-source Crawl4AI engine on Apify's managed infrastructure.

• Markdown for AI — every page rendered in a real browser, converted to clean Markdown with a link graph
• Watch mode — $0.002 to check a page that didn't change, browser prices only for the ones that did
• Structured extraction — JSON-LD and OpenGraph pulled into typed entities and flat price / author / rating columns
• AI-ready package — a JSONL corpus, an llms.txt index and a manifest, written straight to the run's key-value store
• Site intelligence — broken links, orphan pages, duplicate content and a token estimate, free with any crawl
• Alerts — a webhook or Slack message the moment a watched page moves

Two modes

Crawl — the classic scrape. Render pages, follow internal links, get clean Markdown with a link graph.

Watch — the cheap one. For each page, a plain HTTP request fetches the HTML, strips the noise (scripts, styles, comments, CSRF nonces, build IDs) and hashes it. If the hash matches last time, the page is reported as unchanged for $0.002 and the browser is never started. Only pages whose fingerprint moved get rendered, diffed, and reported as a change.

That is the whole trick: monitoring is cheap because most pages don't change, and you stop paying render prices to find that out.

1000 pages, daily, 3% of them change
   watch mode:  970 × $0.002  +  30 × $0.05  =  $3.44 / day
   re-scraping: 1000 × $0.005                =  $5.00 / day, and you still have to diff it yourself

Try it free

Runs with maxPages ≤ 3, no watch/alert/incremental/aiPackage features, no residential proxy, and ≤ 4 GB memory are not billed at all. Free runs still tell you what's there — how many JSON-LD types the page exposes, and whether it changed since you last looked — just with the values withheld:

{
  "structured": { "typesFound": 2, "types": ["Product", "Offer"], "locked": true },
  "changeSignal": { "changedSincePrevious": true, "locked": true }
}

Set estimateOnly: true on a full-size run to see the exact bill before you commit to it. It crawls everything and charges nothing.

Input

Field	Type	Default	Description
startUrls	array	—	Pages to scrape or watch: [{"url": "https://..."}]
mode	string	crawl	crawl | watch
maxDepth	int	1	Levels of internal links to follow (0 = start URLs only)
maxPages	int	10	Hard cap on pages per run (max 1000)
outputFormat	string	markdown	markdown | html | text
includeRawHtml	bool	false	Also store the raw unrendered HTML
pageLoadTimeoutMs	int	30000	Max render wait per page
renderConcurrency	int	5	Pages rendered in parallel (auto-reduced to fit memory)
extract	string	none	none | auto | jsonld | og — structured data per page
aiPackage	bool	false	Write the JSONL corpus + llms.txt + manifest to the run's key-value store
siteIntel	bool	false	Add the audit block (broken links, orphans, duplicates, tokens) to the summary
watchId	string	hash of startUrls	State store to compare against; share it across runs to track the same site
changeDetection	string	fast	fast (HTTP first, render on change) | rendered (always render)
changeThreshold	number	0.02	Fraction of lines that must differ to count as a change
skipUnchanged	bool	false	Incremental crawl: don't re-render pages that haven't changed
alertWebhookUrl	string	—	HTTPS endpoint to POST changes to; Slack webhooks auto-format
maxChargeUsd	number	5.00	Hard ceiling on what the run may charge
estimateOnly	bool	false	Run everything, charge nothing, report the estimate
proxyConfiguration	object	auto	Apify proxy groups
verifyProxyRotation	bool	false	Debug: report the exit IP of two probe requests

Output

One dataset item per page, plus a run summary. The Changes dataset view shows just the deltas.

{
  "recordType": "page",
  "url": "https://example.com/pricing",
  "title": "Pricing",
  "markdown": "# Pricing\n...",
  "changeType": "changed",
  "diffRatio": 0.14,
  "charsDelta": 320,
  "previousCrawledAt": "2026-09-03T08:00:00Z",
  "detectedVia": "render",
  "addedLines": ["| Pro | $49/mo |"],
  "removedLines": ["| Pro | $39/mo |"],
  "entityType": "Product",
  "price": 49.0,
  "currency": "USD",
  "structured": { "types": ["Product", "Offer"], "jsonld": [], "og": {}, "canonical": "..." },
  "links": { "internal": ["..."], "external": ["..."] },
  "chars": 12345,
  "depth": 0,
  "crawledAt": "2026-09-04T08:00:00Z"
}

changeType is new, unchanged, changed or removed. Unchanged pages found on the cheap path carry detectedVia: "http" and no content — they cost $0.002 and didn't start a browser.

The final item is the summary:

{
  "recordType": "summary", "mode": "watch", "checked": 500, "unchanged": 486,
  "changed": 12, "new": 2, "removed": 0, "estimatedChargeUsd": 1.582
}

Structured extraction

Set extract and every page also comes back as typed data instead of just prose.

extract	What it reads
none (default)	Nothing is published — you still see which types a page exposes, with the values locked
jsonld	<script type="application/ld+json"> blocks, including @graph wrappers and arrays
og	og: / twitter: meta tags and the canonical URL; og:type stands in for the entity type
auto	Both

"structured": {
  "types": ["Product", "Offer"],
  "jsonld": [ { "@type": "Product", "name": "Widget", "offers": { "price": "19.99" } } ],
  "og": { "title": "Widget", "image": "https://..." },
  "twitter": { "card": "summary_large_image" },
  "canonical": "https://shop.example/widget"
}

The parts a spreadsheet can use are also lifted to the top level, so the Structured data dataset view is a readable table: entityType, price, currency, publishedAt, author, rating. Prices are parsed from JSON-LD offers (including priceSpecification and AggregateOffer.lowPrice), falling back to product:price:amount. A price we cannot parse confidently — a range like "from 10 to 20" — is passed through as a string rather than guessed at, because a silently wrong number in a price column is worse than an unparsed one.

Billing follows the value: structured-extract is charged per page where at least one type was actually found. A page with no structured data costs nothing extra, so pointing this at a whole site doesn't bill you for its empty pages.

Changed in v0.3. In v0.2 the parsed JSON-LD was attached to every paid page whether you asked for it or not. It is now behind extract — leave it at none and you get the locked count, exactly like a free run. Set extract: "auto" to get the values back.

AI-ready package

aiPackage: true writes three files to the run's default key-value store when the crawl ends:

Key	What it is
ai-package.jsonl	The corpus — one JSON object per page: {url, title, markdown, structured, crawledAt}
llms.txt	The link index in llms.txt format: one - [Title](url) per page, with the OpenGraph description where there is one
manifest.json	{actor, version, runId, pageCount, generatedAt, schema: {fields: [...]}} — what the bundle holds and how to read it

Point a RAG pipeline at the run, read ai-package.jsonl, done — no dataset paging, no reshaping. Pages with no content (a watch-mode page that didn't change, so was never rendered) are listed in llms.txt but left out of the corpus; manifest.linkIndexCount and manifest.pageCount tell you both numbers. A corpus over 8 MB is truncated rather than failing the write, and says so in manifest.truncated and in the log.

$0.100, once per run, charged only after the bundle is actually written. A run that scraped nothing writes nothing and charges nothing.

Site intelligence

siteIntel: true adds an audit block to the run summary. It is computed entirely from what the run already fetched — no extra requests, no extra charge:

{
  "recordType": "summary",
  "brokenLinks": [ { "url": "https://site/old", "status": 404, "detectedVia": "render" } ],
  "brokenLinkCount": 3,
  "orphanCount": 7,
  "orphanPages": ["https://site/legacy/page"],
  "duplicateCount": 4,
  "duplicateGroups": [ { "hash": "9f2c...", "urls": ["https://site/a", "https://site/b"], "count": 2 } ],
  "tokenEstimate": 412300,
  "topPages": [ { "url": "https://site/docs", "title": "Docs", "chars": 48210 } ]
}

• brokenLinks — pages the crawl reached that answered 4xx/5xx, or failed to load at all. A 404 page still renders fine, which is exactly how a broken internal link surfaces here.
• orphanCount — URLs this watch knows about that nothing crawled in this run links to. Start URLs are entry points, never orphans. This is only meaningful on a completed crawl with history: a run cut short by maxPages has only seen part of the link graph, and the log says so.
• duplicateCount — pages whose normalized content hashes identically to another page. One original per group is not counted as a duplicate.
• tokenEstimate — total characters ÷ 4. What it would cost an LLM to read this site.
• topPages — the ten largest pages by character count.

Alerts

Set alertWebhookUrl to an HTTPS endpoint and every detected change is POSTed to it:

{ "url": "...", "changeType": "changed", "diffRatio": 0.14, "title": "Pricing", "changedAt": "..." }

A Slack incoming webhook (https://hooks.slack.com/services/...) is detected automatically and sent as Slack blocks instead. Delivery is billed only on a 2xx; timeouts and errors are logged and never fail the run. The URL must be https and must resolve to a public address — loopback, private and link-local targets are rejected, and redirects are not followed.

estimateOnly runs do not send alerts — a dry run shouldn't fill your channel — but they do price the deliveries they would have made, so the estimate stays honest.

Pricing

Pay per event. Apify compute is billed separately as usual.

Event	Price	When
actor-start	$0.010	Once per run. Not charged on free runs.
page-scraped	$0.005	Per page rendered. First 3 per run are free.
page-monitored	$0.002	Per page checked in watch mode with no material change.
change-detected	$0.050	Per page with a material content change, or a page that disappeared.
structured-extract	$0.010	Per page where extract found at least one type. Pages with no structured data are free.
alert-delivered	$0.020	Per successful (2xx) webhook delivery.
ai-package	$0.100	Once per run, after the bundle is written to the key-value store.

maxChargeUsd is enforced locally, in-run: once the ceiling is hit, nothing further is billed and the summary says so. Charges are also de-duplicated per (URL, event) and persisted, so an Apify migration mid-run cannot double-bill you.

Use cases

• Competitor and pricing monitoring — watch a catalogue daily, get a webhook when a price moves
• Docs / policy change tracking — know when a vendor edits their terms, with the added and removed lines
• Incremental RAG ingestion — skipUnchanged: true re-embeds only what actually changed
• Drop-in RAG corpus — aiPackage: true hands your pipeline a JSONL corpus and an llms.txt index instead of a dataset to reshape
• Product / catalogue extraction — extract: "auto" turns a shop crawl into entityType / price / currency / rating columns
• Content audits — siteIntel: true for broken links, orphan pages, duplicate content and the token cost of a site, from a crawl you were running anyway
• RAG / knowledge-base ingestion, LLM training-data prep, AI-agent research pipelines

How change detection works

1. Plain HTTP GET, no browser.
2. Normalize: drop <script>, <style>, comments, nonce/CSRF/data-build-* attributes, collapse whitespace. This is what stops a rotating CSRF token from billing you for a change every hour.
3. SHA-256 the result and compare to the stored fingerprint. Match → unchanged, $0.002, done.
4. Mismatch (or a non-200, or changeDetection: "rendered") → render in the browser and compare line by line. A change counts only if the diff ratio clears changeThreshold.
5. A URL known from a previous run but not seen this time is reported as removed — but only if the crawl actually completed. A run truncated by maxPages or wrecked by failures never reports its unvisited tail as deleted.

Fingerprints live in a named key-value store, c4ai-watch-<watchId>, one record per URL, holding the hash, character count, title, a body snapshot for diffing (up to 200 KB), and the page's internal links. Pages above 200 KB are tracked by hash only: still reported as changed, without the line-level diff. Records not seen for 90 days are dropped.

Proxy and IP rotation

Requests are routed through Apify Proxy using a small pool of sticky sessions, rotated per request. Sessions are bounded rather than unlimited, because Playwright binds a proxy when a browser context is created, not per navigation — an unbounded set of proxy URLs would either be ignored or spawn an unbounded set of contexts.

Verify it rather than trust it. Set verifyProxyRotation: true and the run fetches an IP echo endpoint twice on two different sessions and writes both exit IPs to the dataset:

{ "recordType": "proxyRotationCheck", "firstIp": "1.2.3.4", "secondIp": "5.6.7.8", "rotating": true }

If rotating is false, both requests left through the same IP and rotation on this Crawl4AI version is per-run, not per-request — plan around that instead of assuming.

Notes

• Crawl4AI is MIT-licensed open source; this actor is a managed wrapper around it.
• Structured extraction reads Crawl4AI's result.html — the post-JavaScript DOM with <script> tags intact — and never result.cleaned_html, which strips exactly the ld+json blocks the parser needs. So JSON-LD injected by client-side JavaScript is picked up too.
• For sites with aggressive anti-bot protection, use the residential proxy group in proxyConfiguration.
• Crawl runs also record fingerprints, so you can crawl a site once and flip to mode: "watch" afterwards with no re-scrape.