Web Page Reader - Clean Text, Markdown and Links

Give it a list of public URLs and get one clean record per URL: readable text, Markdown, every link on the page, title, language, canonical URL, metadata and JSON-LD.

Static pages and JavaScript pages both work. Every page is fetched over plain HTTP first, which is fast and cheap. A headless Chromium is started only when the evidence says the HTTP result is not the page: an empty JavaScript app shell, almost no text, a bot-challenge interstitial, or a certificate chain that only a browser can complete. You can also force http (never start a browser) or browser (render every page).

Failures come back as records too, with a reason you can branch on, so nothing disappears silently.

Built for lead research, RAG pipelines and change monitoring, where you need the content of pages you already know about rather than a crawl of a whole site. The same engine runs inside LeadProof's Website Crawler.

Try it

The prefilled input reads one static page and one JavaScript-rendered page:

{
  "urls": [
    { "url": "https://example.com", "id": "static-page" },
    { "url": "https://quotes.toscrape.com/js/", "id": "javascript-page" }
  ],
  "renderMode": "auto"
}

The second page has no text in its HTML: its quotes are written by JavaScript. The reader notices, renders it in Chromium and returns the quotes. diagnostics keeps both attempts (67 characters over HTTP, 1,433 after rendering).

What you get per URL

A real record from that run, shortened:

{
  "schemaVersion": "1.0",
  "inputId": "static-page",
  "status": "succeeded",
  "requestedUrl": "https://example.com/",
  "finalUrl": "https://example.com/",
  "canonicalUrl": null,
  "fetchedAt": "2026-09-19T21:44:51.547Z",
  "httpStatus": 200,
  "contentType": "text/html",
  "title": "Example Domain",
  "language": "en",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "text": "Example Domain\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\nLearn more",
  "contentHash": "8c1e8564424fdb68b8b7bdff3e16173a2e3599e9b71620637251486c5c4d5ed6",
  "links": [{ "url": "https://iana.org/domains/example", "text": "Learn more", "rel": [] }],
  "metadata": { "description": null, "sources": { "title": "title", "language": "html[lang]" }, "openGraph": {} },
  "structuredData": { "jsonLd": [], "jsonLdErrors": [] },
  "warnings": [],
  "error": null,
  "diagnostics": { "renderModeUsed": "http", "fallbackReason": null, "contentScopeUsed": "main", "durationMs": 55 }
}

Field	What it is
inputId	Your row id, or a stable generated one. Repeated URLs keep separate ids.
status	succeeded, partial, failed or skipped (see below).
requestedUrl, finalUrl, canonicalUrl	What you asked for, where redirects ended, what the page declares as canonical.
httpStatus, contentType, fetchedAt	HTTP facts of the document that was used, time in UTC.
title, language	<title>, and the language the page declares (never guessed).
markdown, text	Main content with navigation, cookie banners, scripts and forms removed. Markdown keeps headings, lists, tables and links.
contentHash	SHA-256 of the normalized text, for de-duplication and change detection.
links	Every unique link on the page, including navigation: url, text, rel. mailto: and tel: are kept.
metadata	Description, author, site name, dates, robots, Open Graph, Twitter, hreflang alternates, and sources saying where each value came from.
structuredData	Parsed JSON-LD blocks, plus the blocks that could not be parsed.
warnings, error	What was cut or widened, and why a page failed (code, message, retryable).
diagnostics	Render mode used, why the browser was needed, and time, bytes and requests of each attempt.
html	Only when you turn on Include HTML, size-capped.

The full contract is the JSON Schema in lib/contracts/page-record.v1.schema.json, with an example record for every case in lib/contracts/golden/.

Input settings

Setting	Default	Range	Notes
urls	required	1 - 5,000	Strings or {"url", "id"} objects. A bare domain gets https://.
renderMode	auto	auto, http, browser	See the top of this page.
contentScope	auto	auto, main, page	page keeps header and footer text (addresses, phone numbers); main is strict.
includeHtml / maxHtmlKb	off / 1,000	up to 5,000 KB	Raw HTML in each record.
maxLinks	1,000	0 - 5,000	Links kept per page.
maxContentChars	1,000,000	1,000 - 5,000,000	Cap on text and on Markdown per page.
maxConcurrency	5	1 - 20	Pages at the same time.
maxBrowserConcurrency	2	1 - 5	Browser tabs at the same time.
maxConcurrencyPerHost / hostDelayMs	2 / 250 ms	1 - 8 / 0 - 10,000	Politeness per host name.
pageTimeoutSecs	45	5 - 180	Whole page, HTTP and browser attempts together.
maxPageSizeKb	5,000	10 - 20,000	Larger documents are cut and returned as partial.
maxRedirects	5	0 - 10	Redirect hops, meta refresh included.
minTextLength	100	0 - 5,000	Auto mode: less text than this sends the page to the browser.
browserFallbackOnChallenge	on		Auto mode: one browser try on a bot-challenge page.
removeSelectors	none	50	Extra CSS selectors to drop before extraction.
acceptLanguage	en-US,en;q=0.9		Sent to sites.

Status and what you pay for

Status	Meaning	Charged
succeeded	The document was read completely.	Yes, when text was extracted
partial	Content returned, but a limit was hit (size, time, request budget) or the browser fallback failed. The warnings say which.	Yes, when text was extracted
failed	No usable content: blocked address, DNS, TLS, timeout, HTTP error, bot challenge, unsupported file type. error.retryable says whether trying later can help.	No
skipped	Not attempted: the run was aborted or reached your spending limit.	No

Price: $0.002 per page read ($2.00 per 1,000), charged as the page-read event, plus Apify's standard apify-actor-start event. Apify charges that start event once per GB of the run's memory, so a run at the default 2 GB records two of them ($0.0001 per run), and a 1 GB run records one ($0.00005).

What that means in practice:

• One charge per unique URL. List the same URL five times and it is fetched once and charged once. The other four rows still get their own record, marked with diagnostics.dedupedFrom, for free.
• Empty, failed and skipped pages are free. A 404, a blocked page, an unsupported file type, a page with no text, and every row that a spending limit or an abort left unread: no charge.
• No double charge when a run restarts. If the platform migrates the run, or you resurrect it, rows already in the dataset are neither read nor charged again.
• Examples, at the default 2 GB:
  • The prefilled example above (2 URLs, both readable): 2 x $0.002 + 2 x $0.00005 = $0.00410. The same run at 1 GB would be $0.00405, because only one start event is charged.
  • 100 URLs of which 80 return text: 80 x $0.002 + 2 x $0.00005 = $0.16010.

Stops, retries and restarts

• You abort the run, or it reaches your spending limit: every row not yet started gets a skipped record, and pages in flight are cut and recorded as failed with error.code: aborted, so the dataset keeps exactly one record per row. The run also writes a RETRY_INPUT record to its key-value store: your settings plus exactly the rows that were never read, with the same inputIds. Start a new run with it to finish the work.
• Resurrecting a stopped run does not re-read those rows. A dataset is append-only, so reading them again would give a row a second record. A resurrected run reads only rows with no record and points to RETRY_INPUT for the rest.
• The platform migrates the run: nothing was written for rows that were not started, so the restarted run reads them all.

Limits and what it does not do

• Public web pages only. Private, loopback, link-local and cloud-metadata addresses are refused, including through redirects, DNS tricks and every request a page makes in the browser.
• HTML, XHTML and plain text only. PDFs, images, JSON, feeds and file downloads return unsupported_content_type.
• No logins, no cookies you supply, no CAPTCHA solving. A page behind a bot wall returns blocked_by_site. A site that answers with a non-standard status (LinkedIn sends 999 to clients it blocks) returns nonstandard_http_status.
• No proxies: pages are read from Apify's datacenter addresses, so some sites block them.
• Frames and shadow DOM content are not read. Page language is what the page declares, not detected from the text.
• robots.txt is not consulted for these single-page reads, because you supply the URLs. LeadProof's Website Crawler, which discovers pages itself, does respect it.
• Page text is data, not instructions. If you pass it to an LLM, treat it as untrusted input.

Please use it responsibly: keep concurrency and the per-host settings reasonable, and respect the terms of the sites you read.

Measured performance (2 GB memory, September 2026)

Run	Pages	Result	Time
20 public sites (Kenya, Israel, UK, global)	20	16 succeeded, 2 partial, 2 failed (a bot wall, a real 404); browser needed on 3	38 s
Mixed smoke test	12	8 succeeded, 4 failed as designed (404, metadata IP, a redirect to it, expired TLS)	7 s
Browser mode, including a 15 MB page	4	2 succeeded, 1 partial (cut at 5 MB), 1 blocked redirect	20 s

For developers

lib/ is @leadproof/page-reader, a TypeScript library other Actors run in process: one PageReader, one shared browser, no Actor run per page. API, options, security model and the install recipe are in lib/README.md. Third-party licenses: THIRD_PARTY_LICENSES.md.