Website Crawler - Site Pages to Markdown and Text

Crawl one public website within the page, depth and time limits you set. For every page you get clean text, Markdown, links, metadata and JSON-LD, in the shared PageRecord v1 format. Every run also writes a coverage report that says what was found, what was read, what was skipped and why, and whether the crawl stopped at a limit or ran out of pages to discover.

It never claims to have found "all pages" of a site. A crawler can only see pages that are linked, listed in a sitemap or reachable through JavaScript it runs. The report states exactly which of those limits applied.

Pages are read by LeadProof's own page-reading engine, the one behind our Web Page Reader: HTTP first, a headless Chromium only when a page needs it, one browser for the whole crawl. No third-party crawler underneath, so what the report says is what actually happened.

Why this one

Most crawlers hand you pages and let you assume that was the whole site. This one keeps the two questions apart: what it read, and what it could not reach. The coverage report names every limit that applied, counts that add up, and says in one sentence whether the site was exhausted or the crawl stopped early. If you are building a knowledge base, a RAG index or a site audit, that sentence is the difference between "we have the docs" and "we have most of the docs, probably".

How to use it

1. Put the site in Start URLs. One site per run; www and the bare domain count as the same site.
2. Set Max pages to what the job is worth. The crawl stops there and says so.
3. Leave Render mode on auto: pages are read over HTTP, and a headless browser starts only for pages that need JavaScript. Set it to http for a cheaper crawl of a server-rendered site, or browser for a site whose links only appear after rendering.
4. Narrow the crawl with Include URL patterns (for example https://site.com/docs/**) when you want one section, and widen it with Include subdomains when the site spans several.
5. Run it, then read the coverage report (SUMMARY) before you trust the dataset as "the site".

What you get

Output	Where	What it holds
Pages	default dataset	One item per page read (success or failure), plus one item per start URL that could not be read. PageRecord v1 fields plus a crawl object.
Coverage report	key-value record SUMMARY	Counts that add up, stop reasons, outcome, timing, HTTP vs browser reads, browser launches, robots.txt and sitemap diagnostics.
URL inventory	URL_MANIFEST_CSV and URL_MANIFEST (JSON)	Every in-scope URL the crawl learned about and its final state: read, duplicate, skipped (and why) or not attempted.

One page item (run DGaR43TMhBL73yBhL, long fields shortened)

{
  "schemaVersion": "1.0",
  "inputId": "page-440dbfe72ca784d0c6036f7f",
  "status": "succeeded",
  "requestedUrl": "https://leadproof.co/blog/",
  "finalUrl": "https://leadproof.co/blog/",
  "canonicalUrl": "https://leadproof.co/blog/",
  "fetchedAt": "2026-09-19T21:03:35.375Z",
  "httpStatus": 200,
  "contentType": "text/html",
  "title": "LeadProof Blog - Local lead generation notes and playbooks",
  "language": "en",
  "markdown": "# LeadProof Blog\n\nPractical notes on local lead generation, ...",
  "text": "LeadProof Blog\nPractical notes on local lead generation, Goo ...",
  "contentHash": "9f6cbb4a20cf84ec87554af14e9964c95a4deec6d70fde96e6cf988ea10a9eb3",
  "links": [
    {
      "url": "https://leadproof.co/",
      "text": "LeadProof",
      "rel": []
    }
  ],
  "metadata": {
    "description": "LeadProof notes on local lead generation, Google Maps sourcing, mailbox verification, CRM  ...",
    "robots": null,
    "siteName": null,
    "sources": {
      "title": "title",
      "language": "html[lang]",
      "canonicalUrl": "link[rel=canonical]",
      "description": "meta[name=description]"
    },
    "...": "..."
  },
  "structuredData": {
    "jsonLd": [
      "..."
    ],
    "jsonLdErrors": []
  },
  "warnings": [],
  "error": null,
  "diagnostics": {
    "renderModeRequested": "auto",
    "renderModeUsed": "http",
    "fallbackReason": null,
    "attempts": [
      "..."
    ]
  },
  "crawl": {
    "state": "extracted",
    "depth": 1,
    "discoveredVia": "sitemap",
    "referrerUrl": "https://leadproof.co/sitemap.xml",
    "duplicateOf": null,
    "billable": true,
    "renderMode": "http",
    "readMs": 24
  }
}

status describes retrieval (succeeded, partial, failed, skipped). crawl.state says what the page means for the crawl:

crawl.state	Meaning	Billable
extracted	Read completely, has text, first page with this content	yes
duplicate_content	Same normalized text as duplicateOf (for example / and /index.html)	no
duplicate_final_url	Redirected to a page that was already delivered	no
partial	Content returned but cut by a size limit	no
empty	Read, but no text (often a JavaScript-only page on an HTTP read)	no
failed	HTTP error, timeout, network error or blocked page; error says which	no
redirected_out_of_scope	A redirect, meta refresh or script navigation pointed to another site; it was refused before the request	no
blocked_by_robots_redirect	A redirect pointed to a URL robots.txt disallows; refused before the request	no
unsupported_content	Not HTML (PDF, image, ...)	no
skipped_robots / duplicate_start_url	A start URL that robots.txt keeps out, or that repeats another start URL	no

Input

Field	Default	Limits	Notes
startUrls	required	1 - 100	[{ "url": "https://example.com/", "id": "optional-row-id" }]. The crawl stays on these hosts.
maxPages	50	1 - 10,000	Page reads, failures included.
maxDepth	5	0 - 50	Link hops from a start URL. Sitemap pages count as depth 1. 0 = start URLs only.
includeSubdomains	false		Also crawl subdomains of each start host. Parent and sibling domains never.
useSitemaps	true		robots.txt Sitemap: lines, else /sitemap.xml; indexes followed 3 levels deep.
respectRobotsTxt	true		RFC 9309 rules and Crawl-delay. Turn off only for sites you may crawl regardless.
includeUrlPatterns / excludeUrlPatterns	[]	50 patterns, 300 chars	Globs on the full URL: ** any characters, * any except /. Start URLs are always read.
renderMode	auto	auto, http, browser	auto: HTTP first, the browser only for JavaScript shells, near-empty pages and bot challenges. http: never a browser (cheapest). browser: every page (slowest).
maxConcurrency	5	1 - 20	Pages loading at once.
maxRequestsPerMinute	60	1 - 600	Per host, for every request. A longer robots.txt Crawl-delay wins.
maxCrawlSeconds	3600	30 - 86,400	Wall-clock budget. Also stops early enough before the run's own timeout.
pageTimeoutSeconds	30	5 - 120	One page read.
maxPageSizeKb	5000	100 - 20,000	Bigger pages are cut and marked partial.
maxUrlsPerPattern	200	1 - 10,000	Trap guard, see below.
maxDiscoveredUrls	5000	100 - 20,000	Distinct in-scope URLs tracked; more are counted, not tracked.
maxSitemapUrls / maxSitemapFiles	5000 / 20	0 - 20,000 / 1 - 200
includeHtml	false		Adds raw HTML (up to 1 MB per page).

Example:

{
  "startUrls": [{ "url": "https://books.toscrape.com/" }],
  "maxPages": 40,
  "maxDepth": 3,
  "maxConcurrency": 4,
  "maxRequestsPerMinute": 120,
  "excludeUrlPatterns": ["https://books.toscrape.com/catalogue/category/**"]
}

How the crawl decides what to read

• Scope. Only the start hosts. www.example.com and example.com are the same site; an explicit non-default port is another site. With includeSubdomains, blog.example.com is included when you start at example.com. Offsite links are counted in the report and never requested.
• Redirects. The engine asks the crawler before every redirect hop, meta refresh and browser navigation away from the page (in the browser the redirect chain is walked hop by hop before Chromium opens the page). A target outside the scope, or one robots.txt disallows, is refused before any request is sent. Tested against fixture servers' request logs in HTTP and browser mode, and live with httpbin.org redirecting to example.com.
• Order and depth. Breadth-first. A page's depth is its shortest link distance from a start URL, also when pages load in parallel.
• URL normalization. Fragments, default ports, tracking parameters (utm_*, gclid, fbclid, ...) and session IDs are removed. Every other query parameter is kept exactly as written: ?id=1 and ?id=2 are different pages. Parameter order does not create duplicates. Paths keep their case and trailing slash.
• Duplicates. URLs are deduplicated before any request. Pages whose text is identical after normalization are delivered but flagged duplicate_content and not billed. A URL that redirects to an already delivered page is duplicate_final_url; a redirect target is not fetched a second time.
• Canonical links are hints. A page whose rel=canonical points elsewhere is still delivered; the canonical URL is crawled too. Distinct pages are never dropped because of a canonical tag.
• Traps. URLs that differ only in numbers, dates, IDs or query values share a pattern (/calendar/{n}/{n}, /shop?color&size), and each pattern gets at most maxUrlsPerPattern URLs. Paths with a segment repeated 3 times, more than 25 segments or more than 8 query parameters are skipped.
• robots.txt (RFC 9309). 2xx: rules and Crawl-delay apply. 4xx: everything is allowed. 5xx, 429, DNS or connection errors: the site is not crawled (the start URL row says which error). Page-level

  <meta name="robots" content="nofollow">
  stops link discovery from that page.
• Sitemaps. Nested indexes up to 3 levels, at most maxSitemapFiles files and maxSitemapUrls URLs, 20 MB per download and 50 MB after decompression (gzip bombs stop at the cap), cycles ignored, only in-scope URLs used.
• Downloads. Links to PDFs, images, archives, media and similar files are not fetched.

Limits, stopping and honest coverage

The crawl stops scheduling when it reaches maxPages, maxCrawlSeconds, the run's own timeout, your spending limit, or when nothing is left to read. Pages already loading finish and are stored. SUMMARY.crawlOutcome is one of:

Outcome	Meaning
exhausted_discovered_urls	Every in-scope URL the crawl discovered was read. Unlinked pages may still exist.
finished_within_limits	Nothing left in the queue, but some URLs were skipped by depth, pattern, tracking or sitemap limits (limitSkips).
stopped_at_limit	Stopped at stoppedBy with notAttempted URLs left. The site was not exhausted.
interrupted	Aborted or migrated; resume continues from the checkpoint.
no_pages_read	Nothing could be read; the start URL rows say why. The run is marked failed.

How the counts add up (SUMMARY.reconciliation): every tracked in-scope URL has exactly one state, and the states sum to urlsTracked. Dataset items = one per page read attempt + one per start URL that was not read. billablePages = items with crawl.billable. Offsite links are counted, not tracked.

Interruptions and resume

The crawl checkpoints its frontier every 10 seconds and on every platform persistState. Pages already in the dataset are the source of truth: after a crash, migration or resurrection, the run re-reads the dataset, skips every stored page, re-derives the links found after the last checkpoint and continues. No page is stored or charged twice. A different input on the same storage is refused instead of mixed. Graceful abort writes an interrupted report.

Verified on the platform with the engine build: an auto-mode crawl of a JavaScript site aborted after 4 pages and resurrected ended with 10 items, 10 unique URLs, crawl time carried over (lVD26I51zAM5Bn5b6).

What you pay for

Price	Event	Charged for
$0.002 per page ($2.00 per 1,000)	page-extracted	One page that was read completely and has text nobody else in this run already returned.
$0.002 per GB of the run	apify-actor-start	Starting the crawl: robots.txt, sitemaps and the first requests. Apify charges this once per GB, so the default 2 GB run pays $0.004 and a 1 GB run $0.002. It is charged on every run, including a run that extracts nothing.

Everything else is free: pages that failed, were empty, were blocked by robots.txt, were duplicates of a page you already got, were cut short by a size limit (partial), or were discovered and never read because a limit stopped the crawl. You pay for content you can use, plus the run fee.

So the smallest possible bill is the run fee on its own: point the crawler at a site that blocks every request and you are charged $0.004 at the default memory and nothing else.

Example run	Cost
The prefilled example: 10 pages of books.toscrape.com at 2 GB	10 x $0.002 + $0.004 = $0.024
A 500-page documentation site at 2 GB	500 x $0.002 + $0.004 = $1.004
A site that blocks every request, so nothing is extracted	$0.004, the run fee alone, at the default 2 GB

A crawl of the same site costs the same whether its pages come over HTTP or need the browser: the price is per page of content, not per technique.

No double charge. A page is stored first and charged with its own idempotency key, so a retry cannot bill twice. If the platform migrates the run or you resurrect it, pages already in the dataset are not read or charged again; only the run fee is charged again, because the run starts a second time. The crawl also stops starting pages once your spending limit no longer covers the ones already in flight.

Limitations

• Auto mode renders a page only when its HTTP read looks like a JavaScript shell, has almost no text or is a bot challenge. A page with plenty of server-side text but extra links added by JavaScript is read over HTTP, so those links are not seen; use renderMode: "browser" for such sites.
• maxRequestsPerMinute spaces page reads. A browser read also loads the page's own subresources (images, fonts and media are blocked by the engine; scripts and styles are not).
• One site per start host. No login, no form submission, no CAPTCHA solving, no attempt to get around blocks: a blocked page is reported as failed.
• PDF and other downloads are not extracted.
• Ports 80 and 443 only; private, loopback, link-local and cloud metadata addresses are refused at connect time, including after redirects and DNS changes.
• robots.txt is followed by default. With respectRobotsTxt: false you are responsible for having permission.
• Page text is data from third parties. It is stored as is and never executed or interpreted as instructions.

External dependencies and secrets

No secrets and no paid third-party services. Runs use Apify platform compute and storage only. The page reader (actors/web-page-reader/lib, LeadProof) and Chromium from the Apify Playwright base image are compiled into the image from the same repository commit. Open-source libraries are listed in THIRD_PARTY_NOTICES.md.

Fair use

The crawler follows robots.txt by default, spaces its requests per host (maxRequestsPerMinute, and any Crawl-delay the site asks for), and identifies itself honestly in its user agent. It does not log in, solve CAPTCHAs, rotate identities or work around a block: a page that refuses us is reported as failed. Turning respectRobotsTxt off is your decision and your responsibility, and crawling a site you do not own may be limited by its terms of use or by local law. Page text is third-party data: it is stored as it was served and never executed or followed as instructions.

Other LeadProof Actors

• Web Page Reader - the same page-reading engine for a list of URLs you already have.
• Search & Read - country-pinned search plus the page reads, in one run.
• Business Enrichment - match a business name to its website, email and phone.
• Google Maps Scraper and Google Maps Verified Leads - local business data, with verified emails.

Built by LeadProof.

Development

The Actor links the page reader library ("@leadproof/page-reader": "file:../web-page-reader/lib"), so it needs actors/web-page-reader/lib in the same checkout. npm test compiles that library first and runs 56 tests on the real engine, none skipped. Deployment, builds and the verified build number: DEPLOYMENT.md.