# Web Page Text Scraper – Clean Text & Links (`simple.actors/page-text-reader`) Actor

Scrape any web page as clean text: the full visible prose, with scripts, navigation, headers, footers, booking widgets and cookie banners stripped out. Discover mode also returns every same-site link with its anchor text. JavaScript is rendered only when a page needs it.

- **URL**: https://apify.com/simple.actors/page-text-reader.md
- **Developed by:** [Simple Actors](https://apify.com/simple.actors) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

$2.00 / 1,000 page reads

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Web Page Text Scraper – Clean Text & Links

Scrape any web page as clean, readable text. It returns what is actually on the
page — the visible prose, with the scripts, navigation, headers, footers,
booking widgets and cookie banners taken out — and, when you ask for it, every
link to the same site with the words that were clickable. Built for feeding LLMs
and RAG pipelines, content and SEO auditing, site research and text mining.

It runs the page's JavaScript when the page needs it, so a site that builds
itself in the browser reads the same as one that does not.

Two things make it different from a general crawler: it reads the pages you
name and nothing else, and **a page that could not be read never comes back
looking like a page with nothing on it** — it arrives as an error row with no
`text` field at all, rather than as a page whose text is empty. See
[Failures](#failures) — that distinction is the point of this Actor.

### Features

- **Clean text extraction** — scripts, styles, navigation, headers, footers,
  cookie banners, booking and newsletter forms, carousel controls and repeated
  lines all removed.
- **Full text, never summarised** — not shortened, not rewritten; a name in the
  middle of a paragraph survives.
- **Sentence boundaries preserved** — a full stop at every block boundary, so
  `Chef: Ana. Owner: Bo.` never becomes the one person "Ana Owner".
- **Same-site link discovery with anchor text** — every internal link and the
  words that were clickable, in page order. External links excluded on purpose.
- **JavaScript rendered only when needed** — `auto` opens a browser only when
  the served HTML turns out not to hold the page; `always` and `never` force it.
- **Redirects followed and reported** — `loadedUrl` is where the page was really
  read from, including a redirect the page's own JavaScript performed.
- **Raw HTML on request** — the rendered DOM where a browser was used, the
  served HTML otherwise, for running your own extractor.
- **Unreadable is never confused with empty** — an error row carries no `text`
  key at all, so "could not read" and "nothing on the page" stay distinguishable.
- **404 and 410 are answers, not failures** — returned as ordinary page rows
  with their status code.
- **Bare hosts accepted** — `example.com` works as well as a full URL.
- **Deduplicated** — the same page is never read twice in one run, and
  `www.example.com` and `example.com` are the same page.
- **Runs stay green.** Page problems are rows; only a bug fails a run.

### Input

```json
{
  "urls": ["https://www.whiteoakny.com/"],  // plain list; accepts a bare host too
  "mode": "discover",        // "discover" = text + same-site links, "text" = text alone
  "renderJs": "auto",        // "auto" | "always" | "never"
  "maxTextLength": 50000,    // safety net, not a trim
  "includeHtml": false,      // also return the page's raw markup as `html`
  "proxy": { "useApifyProxy": true }
}
```

| Input | Type | Default | What it is for |
|---|---|---|---|
| `startUrls` | array | — | The addresses to read, in the URL editor. |
| `urls` | array | — | The same, as a plain list. Both are read; either alone is enough. Accepts a bare host like `example.com`. |
| `mode` | string | `discover` | `discover` adds the same-site links; `text` returns the text alone. |
| `renderJs` | string | `auto` | When to open a browser: `auto`, `always`, `never`. |
| `maxTextLength` | integer | `50000` | Safety net on a single page's text (500–50000). |
| `includeHtml` | boolean | `false` | Also return the page's raw markup beside the text. |
| `proxy` | object | Apify datacenter | Switch to residential if a site answers with a challenge page. |

### Two modes: discover, then read

#### Discover — the page, plus where else to go

Give it an entry page, usually a site's home page.

```json
{
  "urls": ["https://www.whiteoakny.com/"],
  "mode": "discover"
}
```

```json
[
  {
    "type": "page",
    "url": "https://www.whiteoakny.com/",
    "loadedUrl": "https://www.whiteoakny.com/",
    "statusCode": 200,
    "title": "White Oak Tavern: Rustic & Old-School American Tavern in NYC",
    "text": "Named after the tall white oak trees of Kentucky… Our dishes are a representation of forgotten cooking skills and time-honored techniques… Executive Chef Jorge Madriz has created a menu that fuses the traditional with the contemporary…",
    "textLength": 1145,
    "textTruncated": false,
    "rendered": false,
    "links": [
      { "url": "https://www.whiteoakny.com/menus/", "anchor": "Menus" },
      { "url": "https://www.whiteoakny.com/who-we-are/", "anchor": "Who We Are" },
      { "url": "https://www.whiteoakny.com/the-oak-ledger-blog/", "anchor": "Blog: The Oak Ledger" }
    ],
    "retrievedAt": "2026-08-24T09:12:00.000Z"
  }
]
```

**The anchor text is the useful part.** Addresses are not reliable — `/about`
is a 404 on four of the five sites this was first built against, while their
"about" pages live at `/who-we-are/`, `/our-story` and `/team`. The words next
to the link are what tell you which page is worth reading.

Links to other sites are left out on purpose. A page links to its parent
company, its booking platform and its delivery partner, and none of those pages
are about the site you asked about.

#### Text — read the pages you picked

```json
{
  "urls": [
    "https://www.whiteoakny.com/who-we-are/",
    "https://www.whiteoakny.com/team"
  ],
  "mode": "text"
}
```

```json
[
  {
    "type": "page",
    "url": "https://www.whiteoakny.com/who-we-are/",
    "loadedUrl": "https://www.whiteoakny.com/who-we-are/",
    "statusCode": 200,
    "title": "Who We Are",
    "text": "Our Story. Chef de Cuisine…",
    "textLength": 2310,
    "textTruncated": false,
    "rendered": false,
    "retrievedAt": "2026-08-24T09:12:04.000Z"
  },
  {
    "type": "page",
    "url": "https://www.whiteoakny.com/team",
    "loadedUrl": "https://www.whiteoakny.com/team/",
    "statusCode": 404,
    "retrievedAt": "2026-08-24T09:12:04.000Z"
  }
]
```

The rows have the same shape in both modes. Only `links` is added by discover.

### Output

| Field | Present | Meaning |
|---|---|---|
| `type` | always | `page` or `error` |
| `url` | always | the address asked for |
| `loadedUrl` | always | where the page was finally read from, after redirects — including one done by the page's own JavaScript |
| `statusCode` | always | the HTTP status the page answered with |
| `title` | read pages | the page's own `<title>` |
| `text` | read pages | the visible prose, in full |
| `textLength` | read pages | how many characters that is |
| `textTruncated` | read pages | true only if the page was longer than the limit |
| `rendered` | read pages | true if a browser was needed to see the page |
| `html`, `htmlTruncated` | when asked for | the page's own markup, for running your own extractor on |
| `links` | discover | same-site links, `{ url, anchor }`, in the order they appear |
| `error`, `errorDescription` | error rows | why the page could not be read |
| `retrievedAt` | always | when it was read, ISO 8601, UTC |

#### What `text` is

The page's visible prose, in full. It is not summarised, not shortened and not
rewritten — a name is as often in the middle of a paragraph as in a heading, so
nothing is left out on the grounds of looking unimportant. A typical page runs
1,200 to 7,000 characters, and the 50,000-character limit is a safety net
rather than a policy: a page that exceeds it is cut at a word and marked
`textTruncated: true`, so a shortened answer can never pass for a complete one.

Taken out, because none of it is the page saying anything:

- `<script>` and `<style>`, so a JavaScript variable can never be read as a name
- navigation, headers and footers — including the ones built without the
  semantic tags, which most page builders use
- cookie and consent banners
- booking and newsletter forms, whose time and date pickers otherwise arrive as
  a wall of "11:00 PM 10:30 PM 10:00 PM…"
- carousel controls, accessibility-widget instructions and "Skip to main
  content"
- a line that has already been said once

A full stop is placed at every block boundary. `<h3>Chef: Ana</h3><p>Owner:
Bo</p>` becomes `Chef: Ana. Owner: Bo.` rather than `Chef: Ana Owner: Bo`,
where "Ana Owner" reads as one person's name.

If you would rather run your own extractor, turn on **Also return the raw
HTML** and every row carries the page's markup as `html` beside the text — the
rendered DOM where a browser was used, the served HTML otherwise.

### How to use

#### From Apify Console

1. Open the Actor and click **Try for free** / **Start**.
2. Paste the addresses into **Pages to read (plain list)** — a bare host such as
   `example.com` works.
3. Leave **Mode** on *Discover* the first time: you get the page's text plus
   every same-site link with its anchor text, which tells you which pages are
   worth reading next.
4. Feed those links back in with **Mode** set to *Text* to read the pages you
   picked.
5. Leave **Run the page's JavaScript** on *Auto* — a browser is opened only when
   the served HTML turns out not to hold the page.
6. Click **Start**, then open the **Dataset** tab and export as JSON, CSV or
   Excel.

**Skip rows that have no `text` key** — those are pages that could not be read,
not pages that were empty.

#### From the API

```bash
curl -s "https://api.apify.com/v2/acts/simple.actors~page-text-reader/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["https://www.whiteoakny.com/"], "mode": "discover"}'
```

The two-step pattern — discover, then read what looks useful — with the
JavaScript client:

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const actor = client.actor('simple.actors/page-text-reader');

// 1. Discover: what pages does this site have, and what are they called?
const found = await actor.call({ urls: ['https://www.whiteoakny.com/'], mode: 'discover' });
const { items: [home] } = await client.dataset(found.defaultDatasetId).listItems();

// 2. Read the ones whose anchor text looks relevant.
const wanted = home.links
    .filter((l) => /about|story|team|who we are/i.test(l.anchor))
    .map((l) => l.url);
const read = await actor.call({ urls: wanted, mode: 'text' });
const { items } = await client.dataset(read.defaultDatasetId).listItems();

for (const row of items) {
    if (!('text' in row)) continue;   // unread, not empty — key on presence
    console.log(row.title, row.textLength);
}
```

### Use cases

- **Feed an LLM or RAG pipeline** — clean prose with the boilerplate already
  removed, so you are not paying tokens for cookie banners and nav menus.
- **Content and SEO auditing** — pull the real on-page copy plus the internal
  link graph with anchor text for a set of pages.
- **Site research and data collection** — discover a small site's pages by their
  anchor text, then read only the ones that matter.
- **Text mining and NLP corpora** — build a clean text dataset from a list of
  URLs, with sentence boundaries intact.
- **Monitoring page copy for changes** — read the same pages on a schedule and
  diff `text`, without markup churn creating false positives.
- **Checking whether a site needs a browser at all** — `rendered` on every row
  tells you which pages built themselves in the browser.

### Usage notes

#### What it costs

**$0.002 per page read — $2.00 per 1,000 pages.** One price, nothing else: no
per-run fee, no charge for links, no charge for size.

A page is charged when it is delivered. That includes a page that answered
**404**, because "this page is not there" is a real answer about the site and
costs the same to find out.

**A page that could not be read is never charged.** Refusals, challenges,
timeouts, unusable addresses — those rows carry an `error` and cost nothing, so
a night where a site was down bills only for the sites that answered.

| what you asked for | what it costs |
|---|---|
| one page | $0.002 |
| a site's home page plus one interior page | $0.004 |
| 150 venues, one discover call each | $0.30 |
| 1,000 pages | $2.00 |

#### Failures

**An empty `text` on a 200 means one thing only: the page loaded and genuinely
had nothing readable on it.** Some pages really are like that — a home page
that is one photograph and a menu of links has no prose, and saying so is a
real answer.

So a page that could *not* be read is never reported that way:

- it comes back as a row with `type: "error"`, an `error` code and an
  `errorDescription`, carrying **no `text` field at all**.

**The rows are the failure channel, not the run status.** A run succeeds even
when some — or all — of its pages could not be read, so do not read a green run
as "every page was fine". Its status message counts them (`8 of 10 page(s)
read; 2 could not be read`), and each one is in the dataset.

What this gives you is the distinction that matters: `text: ""` means the page
loaded and had nothing on it, and a missing `text` means it was never read. A
night of refused requests can never arrive looking like a night of empty pages.
If your pipeline acts on emptiness — writing a site off, putting it on a
cooldown — key that decision on whether the row **has** a `text` field, and skip
the rows that do not.

A **404** or **410** is not a failure. The page is not there, which is an
answer, and it arrives as an ordinary `page` row carrying its status code and
no text.

| `error` | what happened |
|---|---|
| `http_error` | the site answered with a status that is not a page |
| `challenge` | it answered with a bot check or a refusal instead of the page |
| `read_failed` | it could not be reached at all |
| `render_failed` | its scripts had to be run and the browser could not |
| `empty_page` | it answered with a document holding nothing at all — no heading, no word. Far more often a page that never finished arriving than a page with nothing to say, so it is reported as unread |
| `needs_browser` | its scripts had to be run and this run was told not to |
| `bad_input` | the address was not usable |

#### The browser

Most pages are read over plain HTTP, which is fast and cheap. A browser is
opened only when the served HTML turns out not to hold the page — measured
across twenty-six real sites, that was two of them: one served 51KB of markup
with no readable text, the other 145 bytes of
`window.location.href="/lander"`. Rendering the other twenty-four added
accessibility notices and carousel labels, and not one sentence of the sites'
own writing.

`rendered` on each row says which route answered. You can force the choice with
**Run the page's JavaScript**: `always` renders everything (slower and dearer),
`never` refuses the browser entirely — and a page that needed one then comes
back as a `needs_browser` error row rather than as a page with no text on it.

#### Notes

- Every request goes through Apify Proxy, and each retry lands on a different
  address, which is what clears most one-off refusals.
- The same page is never read twice in one run, and a link to `www.example.com`
  and one to `example.com` are the same page.
- Pages are read several at a time, so a batch costs about what its slowest
  page costs rather than the sum of all of them.
- Unofficial. It reads public pages as a browser would; it does not log in,
  solve challenges or ignore a site's refusal.

### FAQ

**Is scraping web page text legal?**
This Actor reads public pages as a browser would — it does not log in, solve
challenges, or ignore a site's refusal. What you may do with the text depends on
the site: check the platform's ToS and the site's own terms and robots policy
before collecting or republishing content.

**Do I need a browser or Puppeteer setup?**
No. `renderJs: "auto"` is the default and reads most pages over plain HTTP,
opening a browser only when the served HTML turns out not to hold the page.
Across twenty-six real sites, that was two of them.

**Does it crawl a whole site or follow links automatically?**
No — it reads the pages you name and nothing else. *Discover* mode hands you the
same-site links with their anchor text so **you** choose what to read next; feed
those URLs back in a second run with `mode: "text"`. That is the intended
pattern, and it keeps you from paying for pages you did not want.

**How do I tell an empty page from a page that failed?**
By whether the row has a `text` key. `text: ""` means the page loaded and
genuinely had nothing readable on it. A row with **no** `text` field at all was
never read, and carries `type: "error"` with an `error` code instead. If your
pipeline acts on emptiness, key on the presence of `text`, not its value.

**Why did my run succeed when pages failed?**
Page problems are rows, not run failures — a run succeeds even if every page in
it failed. The status message counts them (`8 of 10 page(s) read; 2 could not be
read`), and each one is in the dataset. Treat the error rows as the failure
channel.

**Is a 404 an error?**
No. The page not being there is a real answer about the site, so it arrives as
an ordinary `page` row with its status code and no text.

**Can I get the raw HTML instead of cleaned text?**
Yes — turn on `includeHtml` and every row carries `html` beside the text: the
rendered DOM where a browser was used, the served HTML otherwise.

**Will long pages be cut off?**
Only past 50,000 characters, which is a safety net rather than a policy — a
typical page runs 1,200 to 7,000. A page that does exceed it is cut at a word
boundary and marked `textTruncated: true`, so a shortened answer can never pass
for a complete one.

# Actor input Schema

## `startUrls` (type: `array`):

The addresses to read, one per entry. In discover mode these are entry pages — usually a site's home page — and each row comes back with the page's text plus the same-site links found on it. In text mode they are the specific pages you want read.

## `urls` (type: `array`):

The same thing as "Pages to read", as a plain list of addresses — easier to paste from your own records, and it accepts a bare host such as example.com. Both inputs are read; giving either one is enough.

## `mode` (type: `string`):

"discover" reads the page and lists every link to the same site, each with the words that were clickable — use it first, to find out which of a site's pages is worth reading. "text" reads the pages you name and returns their text alone. The rows have the same shape either way; only "links" is added by discover.

## `renderJs` (type: `string`):

"auto" reads the served HTML and only opens a browser when that HTML turns out not to hold the page — the right setting for almost everyone, and the cheapest. "always" opens a browser for every page: slower and dearer, worth it only if you know your sites build themselves in the browser. "never" refuses the browser entirely, so a site that needs one comes back as an error row rather than as an empty page.

## `maxTextLength` (type: `integer`):

A safety net, not a trim. Text is returned in full — a typical page runs 1,200 to 7,000 characters, far below this — and a page that does exceed it is cut at a word and marked with textTruncated: true, so a shortened answer can never pass for a complete one.

## `includeHtml` (type: `boolean`):

Adds the page's own markup to each row as `html`, beside the cleaned text — for a caller who would rather run their own extractor. Off by default: the markup is often a hundred times the size of the text, which makes every row slow to read back for anyone who did not ask for it.

## `proxy` (type: `object`):

Defaults to Apify's datacenter proxy, which is included in every plan and is enough for almost every site. Switch to residential if a site answers you with a challenge page — every retry already lands on a different address, so a challenge that survives that is usually about the address pool rather than the request.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.whiteoakny.com/"
    }
  ],
  "urls": [
    "example.com"
  ],
  "mode": "discover",
  "renderJs": "auto",
  "maxTextLength": 50000,
  "includeHtml": false,
  "proxy": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

One row per page: what was asked for, what it answered, its title and its text.

## `links` (type: `string`):

The same-site links each page carried, with their anchor text.

## `raw` (type: `string`):

Every field of every row, exactly as pushed.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.whiteoakny.com/"
        }
    ],
    "urls": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("simple.actors/page-text-reader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.whiteoakny.com/" }],
    "urls": [],
}

# Run the Actor and wait for it to finish
run = client.actor("simple.actors/page-text-reader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.whiteoakny.com/"
    }
  ],
  "urls": []
}' |
apify call simple.actors/page-text-reader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,simple.actors/page-text-reader"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XHxhsQnhe6nEzaZj0/builds/5W2FFwIgFJIooTIZ0/openapi.json