Website Content Crawler for LLM — Markdown, browser if needed

What you get back, one record per page:

{
  "url": "https://docs.example.com/guide/install",
  "title": "Installation",
  "markdown": "# Installation\n\nRun `pip install example`...\n\n| Option | Default |\n|---|---|\n...",
  "charCount": 4812,
  "fetchedWith": "http",
  "tiersTried": ["http"],
  "httpStatus": 200
}

markdown is the page body with links and tables kept and navigation, cookie banners and footers removed — ready to embed in a vector database without a second cleaning step.

Why this one is cheaper to run

Most crawlers in this category start a headless browser for every page. This one tries the cheap way first and only escalates when a page actually forces it:

	what it does	when it is used
http	plain request	most pages
tls-fingerprint	same request with a real browser's TLS handshake	pages behind Cloudflare-style checks
browser	Chromium renders the page	pages that build their content in JavaScript

Every record tells you which one paid for it, in fetchedWith. You are not asked to take the saving on trust.

What we measured, and how

100 pages, two cohorts chosen by external rankings rather than by us — the front page of Hacker News, and the documentation sites of the most-downloaded PyPI packages. Both paths ran against the same URLs with the same extractor, so the difference comes from how the HTML was obtained and from nothing else.

	documentation (40)	general web (60)
plain HTTP is enough	100%	75%
adding the TLS fingerprint	—	83%
adding a real browser	—	93%
no method reaches it (paywall, hard block)	0%	7%
time versus browser-only	5.3× faster	10.6× faster

Three findings worth stating plainly, because they shaped the code:

• Four pages that looked like they needed JavaScript were in fact returning 403 Enable JavaScript and cookies to continue. They gave up their content after the TLS handshake changed — no rendering. On w3.org that was 14 505 characters where Chromium extracted 140. A browser is not a cure for a block; it is an expensive thing that sometimes works.
• On one page a browser was much worse: 13 792 characters over HTTP versus 58 rendered. This Actor therefore returns the best result it obtained, not the last one it tried.
• 7% of the general web returns nothing to any method. Those pages are reported as skipped, with the reason, and are not charged.

If you only crawl documentation, set How hard to try to http and the run will never touch a browser — on the pages we measured it never needed to.

Publishers who opted out of AI use are skipped, and you are told why

robots.txt says whether a page may be fetched. It says nothing about whether its text may be used for text and data mining — and that is exactly what a Markdown corpus is for. Under the EU copyright directive a publisher may reserve that right in a machine-readable form, and a growing number do.

This Actor reads those signals from the response it is already fetching — the TDM-Reservation header, X-Robots-Tag: noai, and the matching <meta> tags in the document head — and when it finds one, the page is not converted, not delivered and not charged. The run status names the page and the exact signal, so a reserved page never looks like a failure or like a silent gap in your corpus.

Nothing is guessed from the wording of a page. Only a reservation expressed for machines counts, because only that one is unambiguous.

Pricing

You are billed per delivered page. A page that returns no extractable content is not delivered and not charged, and a run that delivers nothing fails with the reason rather than finishing green and empty.

Input

field	meaning
startUrls	pages to convert
maxPages	hard cap for the whole run — also your cost ceiling
crawlDepth	0 = only your URLs, 1–2 = follow links (right for a docs site)
sameDomainOnly	keep a crawl from wandering off the site
maxTier	how far it may escalate: http, tls-fingerprint, browser

Limits, stated up front

• Pages behind a login or a hard paywall are not reachable and are reported as such.
• crawlDepth above 2 mostly discovers navigation pages; it costs more and adds little.
• The run stops at maxPages and says so in the run status when more pages were queued.
• A page body is read up to 8 MB and no further. The largest page in our 100-page sample was 2.1 MB, so this cuts nothing real — it exists so that a server answering with an endless stream cannot take the run down after you have already paid for earlier pages. When the ceiling is hit, the page is reported with that reason instead of being escalated to a more expensive method.

If it worked for you

Ratings are the one thing we cannot build ourselves, and on Apify they feed the quality score that decides whether anyone finds this Actor at all. If it did the job, a rating on the Store page takes a few seconds and genuinely changes what happens to it.

If it did not do the job, the issue tab is more useful to us than a low rating with no detail — tell us the URL that failed and what you expected, and it becomes a test case in the next release. Every defect listed in the changelog got there that way, or from our own gate.