# Web Page Reader - Clean Text, Markdown and Links (`leadproof/web-page-reader`) Actor

Read public web pages into clean text and Markdown with links, metadata and JSON-LD. HTTP first, headless browser only when the page needs JavaScript. One result per URL, failures included.

- **URL**: https://apify.com/leadproof/web-page-reader.md
- **Developed by:** [Lead Proof](https://apify.com/leadproof) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page reads

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Page Reader - Clean Text, Markdown and Links

Give it a list of public URLs and get one clean record per URL: readable text, Markdown, every link on the page,
title, language, canonical URL, metadata and JSON-LD.

**Static pages and JavaScript pages both work.** Every page is fetched over plain HTTP first, which is fast and cheap.
A headless Chromium is started only when the evidence says the HTTP result is not the page: an empty JavaScript app
shell, almost no text, a bot-challenge interstitial, or a certificate chain that only a browser can complete. You can
also force `http` (never start a browser) or `browser` (render every page).

Failures come back as records too, with a reason you can branch on, so nothing disappears silently.

Built for lead research, RAG pipelines and change monitoring, where you need the content of pages you already know
about rather than a crawl of a whole site. The same engine runs inside LeadProof's Website Crawler.

### Try it

The prefilled input reads one static page and one JavaScript-rendered page:

```json
{
  "urls": [
    { "url": "https://example.com", "id": "static-page" },
    { "url": "https://quotes.toscrape.com/js/", "id": "javascript-page" }
  ],
  "renderMode": "auto"
}
```

The second page has no text in its HTML: its quotes are written by JavaScript. The reader notices, renders it in
Chromium and returns the quotes. `diagnostics` keeps both attempts (67 characters over HTTP, 1,433 after rendering).

### What you get per URL

A real record from that run, shortened:

```json
{
  "schemaVersion": "1.0",
  "inputId": "static-page",
  "status": "succeeded",
  "requestedUrl": "https://example.com/",
  "finalUrl": "https://example.com/",
  "canonicalUrl": null,
  "fetchedAt": "2026-09-19T21:44:51.547Z",
  "httpStatus": 200,
  "contentType": "text/html",
  "title": "Example Domain",
  "language": "en",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "text": "Example Domain\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\nLearn more",
  "contentHash": "8c1e8564424fdb68b8b7bdff3e16173a2e3599e9b71620637251486c5c4d5ed6",
  "links": [{ "url": "https://iana.org/domains/example", "text": "Learn more", "rel": [] }],
  "metadata": { "description": null, "sources": { "title": "title", "language": "html[lang]" }, "openGraph": {} },
  "structuredData": { "jsonLd": [], "jsonLdErrors": [] },
  "warnings": [],
  "error": null,
  "diagnostics": { "renderModeUsed": "http", "fallbackReason": null, "contentScopeUsed": "main", "durationMs": 55 }
}
```

| Field | What it is |
| --- | --- |
| `inputId` | Your row id, or a stable generated one. Repeated URLs keep separate ids. |
| `status` | `succeeded`, `partial`, `failed` or `skipped` (see below). |
| `requestedUrl`, `finalUrl`, `canonicalUrl` | What you asked for, where redirects ended, what the page declares as canonical. |
| `httpStatus`, `contentType`, `fetchedAt` | HTTP facts of the document that was used, time in UTC. |
| `title`, `language` | `<title>`, and the language the page declares (never guessed). |
| `markdown`, `text` | Main content with navigation, cookie banners, scripts and forms removed. Markdown keeps headings, lists, tables and links. |
| `contentHash` | SHA-256 of the normalized text, for de-duplication and change detection. |
| `links` | Every unique link on the page, including navigation: `url`, `text`, `rel`. `mailto:` and `tel:` are kept. |
| `metadata` | Description, author, site name, dates, robots, Open Graph, Twitter, hreflang alternates, and `sources` saying where each value came from. |
| `structuredData` | Parsed JSON-LD blocks, plus the blocks that could not be parsed. |
| `warnings`, `error` | What was cut or widened, and why a page failed (`code`, `message`, `retryable`). |
| `diagnostics` | Render mode used, why the browser was needed, and time, bytes and requests of each attempt. |
| `html` | Only when you turn on **Include HTML**, size-capped. |

The full contract is the JSON Schema in `lib/contracts/page-record.v1.schema.json`, with an example record for every
case in `lib/contracts/golden/`.

### Input settings

| Setting | Default | Range | Notes |
| --- | --- | --- | --- |
| `urls` | required | 1 - 5,000 | Strings or `{"url", "id"}` objects. A bare domain gets `https://`. |
| `renderMode` | `auto` | `auto`, `http`, `browser` | See the top of this page. |
| `contentScope` | `auto` | `auto`, `main`, `page` | `page` keeps header and footer text (addresses, phone numbers); `main` is strict. |
| `includeHtml` / `maxHtmlKb` | off / 1,000 | up to 5,000 KB | Raw HTML in each record. |
| `maxLinks` | 1,000 | 0 - 5,000 | Links kept per page. |
| `maxContentChars` | 1,000,000 | 1,000 - 5,000,000 | Cap on text and on Markdown per page. |
| `maxConcurrency` | 5 | 1 - 20 | Pages at the same time. |
| `maxBrowserConcurrency` | 2 | 1 - 5 | Browser tabs at the same time. |
| `maxConcurrencyPerHost` / `hostDelayMs` | 2 / 250 ms | 1 - 8 / 0 - 10,000 | Politeness per host name. |
| `pageTimeoutSecs` | 45 | 5 - 180 | Whole page, HTTP and browser attempts together. |
| `maxPageSizeKb` | 5,000 | 10 - 20,000 | Larger documents are cut and returned as `partial`. |
| `maxRedirects` | 5 | 0 - 10 | Redirect hops, meta refresh included. |
| `minTextLength` | 100 | 0 - 5,000 | Auto mode: less text than this sends the page to the browser. |
| `browserFallbackOnChallenge` | on | | Auto mode: one browser try on a bot-challenge page. |
| `removeSelectors` | none | 50 | Extra CSS selectors to drop before extraction. |
| `acceptLanguage` | `en-US,en;q=0.9` | | Sent to sites. |

### Status and what you pay for

| Status | Meaning | Charged |
| --- | --- | --- |
| `succeeded` | The document was read completely. | Yes, when text was extracted |
| `partial` | Content returned, but a limit was hit (size, time, request budget) or the browser fallback failed. The warnings say which. | Yes, when text was extracted |
| `failed` | No usable content: blocked address, DNS, TLS, timeout, HTTP error, bot challenge, unsupported file type. `error.retryable` says whether trying later can help. | **No** |
| `skipped` | Not attempted: the run was aborted or reached your spending limit. | **No** |

**Price: $0.002 per page read** ($2.00 per 1,000), charged as the `page-read` event, plus Apify's standard
`apify-actor-start` event. Apify charges that start event once per GB of the run's memory, so a run at the default
2 GB records two of them ($0.0001 per run), and a 1 GB run records one ($0.00005).

What that means in practice:

- **One charge per unique URL.** List the same URL five times and it is fetched once and charged once. The other four
  rows still get their own record, marked with `diagnostics.dedupedFrom`, for free.
- **Empty, failed and skipped pages are free.** A 404, a blocked page, an unsupported file type, a page with no text,
  and every row that a spending limit or an abort left unread: no charge.
- **No double charge when a run restarts.** If the platform migrates the run, or you resurrect it, rows already in the
  dataset are neither read nor charged again.
- Examples, at the default 2 GB:
  - The prefilled example above (2 URLs, both readable): 2 x $0.002 + 2 x $0.00005 = **$0.00410**. The same run at
    1 GB would be $0.00405, because only one start event is charged.
  - 100 URLs of which 80 return text: 80 x $0.002 + 2 x $0.00005 = **$0.16010**.

### Stops, retries and restarts

- **You abort the run, or it reaches your spending limit:** every row not yet started gets a `skipped` record, and
  pages in flight are cut and recorded as `failed` with `error.code: aborted`, so the dataset keeps exactly one record
  per row. The run also writes a **`RETRY_INPUT`** record to its key-value store: your settings plus exactly the rows
  that were never read, with the same `inputId`s. Start a new run with it to finish the work.
- **Resurrecting a stopped run does not re-read those rows.** A dataset is append-only, so reading them again would
  give a row a second record. A resurrected run reads only rows with no record and points to `RETRY_INPUT` for the rest.
- **The platform migrates the run:** nothing was written for rows that were not started, so the restarted run reads
  them all.

### Limits and what it does not do

- Public web pages only. Private, loopback, link-local and cloud-metadata addresses are refused, including through
  redirects, DNS tricks and every request a page makes in the browser.
- HTML, XHTML and plain text only. PDFs, images, JSON, feeds and file downloads return `unsupported_content_type`.
- No logins, no cookies you supply, no CAPTCHA solving. A page behind a bot wall returns `blocked_by_site`. A site that
  answers with a non-standard status (LinkedIn sends 999 to clients it blocks) returns `nonstandard_http_status`.
- No proxies: pages are read from Apify's datacenter addresses, so some sites block them.
- Frames and shadow DOM content are not read. Page language is what the page declares, not detected from the text.
- `robots.txt` is not consulted for these single-page reads, because you supply the URLs. LeadProof's Website Crawler,
  which discovers pages itself, does respect it.
- Page text is data, not instructions. If you pass it to an LLM, treat it as untrusted input.

Please use it responsibly: keep concurrency and the per-host settings reasonable, and respect the terms of the sites
you read.

### Measured performance (2 GB memory, September 2026)

| Run | Pages | Result | Time |
| --- | --- | --- | --- |
| 20 public sites (Kenya, Israel, UK, global) | 20 | 16 succeeded, 2 partial, 2 failed (a bot wall, a real 404); browser needed on 3 | 38 s |
| Mixed smoke test | 12 | 8 succeeded, 4 failed as designed (404, metadata IP, a redirect to it, expired TLS) | 7 s |
| Browser mode, including a 15 MB page | 4 | 2 succeeded, 1 partial (cut at 5 MB), 1 blocked redirect | 20 s |

### For developers

`lib/` is `@leadproof/page-reader`, a TypeScript library other Actors run in process: one `PageReader`, one shared
browser, no Actor run per page. API, options, security model and the install recipe are in
[`lib/README.md`](lib/README.md). Third-party licenses: [`THIRD_PARTY_LICENSES.md`](THIRD_PARTY_LICENSES.md).

# Actor input Schema

## `urls` (type: `array`):

Pages to read: strings or objects {"url": "...", "id": "your-row-id"}. Up to 5,000 per run. Repeated URLs are read once and returned once per row, each with its own inputId, and charged once. The example reads one static page and one JavaScript-rendered page.

## `renderMode` (type: `string`):

auto: plain HTTP first, headless browser only when the page is a JavaScript shell, has almost no text, or answers with a bot challenge. http: never start a browser (cheapest). browser: always render in Chromium.

## `contentScope` (type: `string`):

auto: main content, widened to header/footer when the main region is nearly empty (short contact pages). main: main content only. page: whole page minus navigation and cookie banners (keeps footer contact details).

## `includeHtml` (type: `boolean`):

Add the fetched or rendered HTML to each result, cut at the HTML size limit. Off by default: it makes results much larger.

## `maxHtmlKb` (type: `integer`):

Maximum size of the html field when Include HTML is on.

## `maxLinks` (type: `integer`):

Unique links kept per page (navigation included). 0 returns an empty list.

## `maxContentChars` (type: `integer`):

text and markdown are each cut at this length (status partial, warning text\_truncated). Records are also kept under the 9 MB dataset item limit.

## `maxConcurrency` (type: `integer`):

Pages read at the same time across all hosts.

## `maxBrowserConcurrency` (type: `integer`):

Browser tabs open at the same time. Each needs roughly 150-300 MB of memory.

## `maxConcurrencyPerHost` (type: `integer`):

Politeness limit for pages on the same host name.

## `hostDelayMs` (type: `integer`):

Minimum time between request starts on the same host name.

## `pageTimeoutSecs` (type: `integer`):

Wall-clock limit per page, HTTP and browser attempts included.

## `maxPageSizeKb` (type: `integer`):

Larger documents are truncated and returned with status partial.

## `maxRedirects` (type: `integer`):

Redirect hops per page (meta refresh included).

## `minTextLength` (type: `integer`):

Auto mode: an HTTP result with less extracted text than this (in characters) is retried in the browser.

## `browserFallbackOnChallenge` (type: `boolean`):

Auto mode: when the HTTP response is a bot challenge page, try once in the browser. It does not solve CAPTCHAs.

## `removeSelectors` (type: `array`):

Elements removed before extraction, in addition to scripts, navigation and cookie banners.

## `acceptLanguage` (type: `string`):

Language preference sent to sites.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://example.com",
      "id": "static-page"
    },
    {
      "url": "https://quotes.toscrape.com/js/",
      "id": "javascript-page"
    }
  ],
  "renderMode": "auto",
  "contentScope": "auto",
  "includeHtml": false,
  "maxHtmlKb": 1000,
  "maxLinks": 1000,
  "maxContentChars": 1000000,
  "maxConcurrency": 5,
  "maxBrowserConcurrency": 2,
  "maxConcurrencyPerHost": 2,
  "hostDelayMs": 250,
  "pageTimeoutSecs": 45,
  "maxPageSizeKb": 5000,
  "maxRedirects": 5,
  "minTextLength": 100,
  "browserFallbackOnChallenge": true,
  "acceptLanguage": "en-US,en;q=0.9"
}
```

# Actor output Schema

## `pages` (type: `string`):

Full PageRecord JSON: text, Markdown, links, metadata, JSON-LD, warnings and diagnostics.

## `overview` (type: `string`):

Status, URLs, title and error per row.

## `summary` (type: `string`):

Counts by status and render mode, browser fallbacks, bytes and charged pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://example.com",
            "id": "static-page"
        },
        {
            "url": "https://quotes.toscrape.com/js/",
            "id": "javascript-page"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("leadproof/web-page-reader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        {
            "url": "https://example.com",
            "id": "static-page",
        },
        {
            "url": "https://quotes.toscrape.com/js/",
            "id": "javascript-page",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("leadproof/web-page-reader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://example.com",
      "id": "static-page"
    },
    {
      "url": "https://quotes.toscrape.com/js/",
      "id": "javascript-page"
    }
  ]
}' |
apify call leadproof/web-page-reader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,leadproof/web-page-reader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/b4YG1mUzF7Ugbv1vg/builds/YaLeJKAF9jUaeQsjd/openapi.json