# LLM Web Page Research Browser (`automation-lab/llm-web-page-research-browser`) Actor

Convert public web pages into cleaned Markdown, compact chunks, metadata, links, and citations for LLM research and RAG ingestion.

- **URL**: https://apify.com/automation-lab/llm-web-page-research-browser.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.92 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## LLM Web Page Research Browser

Turn anonymous public web pages into clean Markdown, plain text, compact chunks, metadata, links, and citations for LLM research and retrieval-augmented generation (RAG).

This Actor provides bounded **LLM web scraping** for supplied URLs. It does not search the web, log in to websites, or pretend every JavaScript application can be read without a browser.

### What does LLM Web Page Research Browser do?

For every successfully fetched page, the Actor:

- follows up to five public HTTP redirects;
- isolates readable article or main-page content;
- converts that content to Markdown and plain text;
- extracts title, description, author, publisher, language, and publication date when exposed;
- extracts and deduplicates same-origin links;
- splits Markdown into overlapping, retrieval-ready chunks;
- attaches a source URL, title, and retrieval timestamp to the page and every chunk;
- reports word, character, and approximate token counts.

One dataset item represents one successfully processed page. Failed pages are logged and are not charged as items.

### Who is this Actor for?

- **RAG engineers** preparing source documents for a vector database.
- **AI agent builders** who need cited context from known URLs.
- **Researchers and analysts** collecting readable source material for grounded synthesis.
- **Documentation teams** transforming public guides into stable ingestion records.
- **Compliance teams** preparing public policy pages for internal search, while retaining citations.

Choose this Actor when you already know the source URLs. It is not a web search API.

### Why use it for LLM web scraping?

Raw HTML is noisy and expensive to send to an LLM. This Actor removes navigation and common page chrome, normalizes content, and returns both complete page fields and compact chunks.

The output is deterministic enough for repeated ingestion:

- bounded URL and page counts;
- bounded download and content sizes;
- configurable chunks and overlap;
- stable page-level records;
- citations carried into every chunk;
- no hidden browser or residential-proxy fallback.

### Supported pages

The Actor supports anonymously reachable public HTTP(S) HTML pages that expose useful server-rendered content.

Typical sources include:

- documentation pages;
- public articles and reference pages;
- public policy and government pages;
- blogs and knowledge bases;
- public product or company information pages.

The Actor rejects local and private network destinations, credential-bearing URLs, non-HTTP protocols, and non-HTML responses.

### Getting started

1. Open the Actor in Apify Console.
2. Add one or more public page URLs to **Public page URLs**.
3. Keep the default chunk size and overlap for a first run.
4. Set **Maximum pages** to bound the run.
5. Click **Start**.
6. Open the default dataset to inspect page records and cited chunks.
7. Export JSON or connect the dataset to your ingestion workflow.

A practical first input is:

```json
{
  "startUrls": [
    { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }
  ],
  "maxPages": 1,
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "maxContentChars": 50000,
  "maxLinks": 100,
  "includeMarkdown": true,
  "includeText": true
}
```

### Input parameters

| Field | Type | Default | Meaning |
|---|---:|---:|---|
| `startUrls` | array | required | Public HTTP(S) pages to process. |
| `maxPages` | integer | `10` | Maximum unique supplied URLs processed, up to 50. |
| `maxContentChars` | integer | `50000` | Maximum cleaned content characters per page. |
| `chunkSize` | integer | `2000` | Target maximum characters per chunk. |
| `chunkOverlap` | integer | `200` | Repeated characters between chunks; must be smaller than `chunkSize`. |
| `maxLinks` | integer | `100` | Maximum same-origin links retained per page. |
| `includeMarkdown` | boolean | `true` | Include complete cleaned Markdown. |
| `includeText` | boolean | `true` | Include complete cleaned plain text. |
| `requestTimeoutSecs` | integer | `30` | Timeout per request, from 5 to 120 seconds. |
| `maxRetries` | integer | `2` | Retries for network errors, timeouts, HTTP 429, and temporary 5xx errors. |

At most 50 URLs are accepted in one input. Each response is capped at 2.5 MB before parsing.

### Output fields

| Field | Description |
|---|---|
| `url` | Final source URL after public redirects. |
| `requestedUrl` | URL supplied by the user. |
| `title` | Readable page or article title. |
| `description` | Metadata description, when exposed. |
| `author` | Author or byline, when exposed. |
| `siteName` | Publisher or site name, when exposed. |
| `language` | Declared or extracted language. |
| `publishedAt` | Source publication time, when exposed. |
| `retrievedAt` | UTC retrieval timestamp. |
| `statusCode` | Successful HTTP status. |
| `contentType` | Source response media type. |
| `markdown` | Cleaned Markdown, or `null` when disabled. |
| `text` | Cleaned plain text, or `null` when disabled. |
| `charCount` | Markdown character count. |
| `wordCount` | Approximate word count. |
| `tokenEstimate` | Approximate tokens at four characters per token. |
| `truncated` | Whether content reached `maxContentChars`. |
| `links` | Deduplicated same-origin URL and anchor-text pairs. |
| `chunks` | Compact overlapping Markdown chunks with citations. |
| `citation` | Page-level source URL, title, and retrieval time. |

Metadata fields are nullable because not every publisher exposes them.

### Example output

The following shortened shape comes from the current Wikipedia RAG example:

```json
{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "requestedUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation - Wikipedia",
  "language": "en",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "wordCount": 2580,
  "tokenEstimate": 7500,
  "truncated": true,
  "chunks": [
    {
      "id": "en.wikipedia.org-001",
      "text": "Retrieval-augmented generation (RAG) is a technique...",
      "charCount": 1772,
      "wordCount": 169,
      "citation": {
        "sourceUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
        "title": "Retrieval-augmented generation - Wikipedia",
        "retrievedAt": "2026-08-24T20:10:00.000Z"
      }
    }
  ]
}
```

The full record also includes Markdown, text, metadata, and extracted links.

### Chunking and citations

Chunks are created from cleaned Markdown rather than raw HTML.

The splitter prefers a paragraph or sentence boundary near the target size. Neighboring chunks repeat the requested overlap so downstream retrieval does not lose context at a boundary.

Every chunk includes:

- a stable-in-record sequence ID;
- chunk text;
- character and word counts;
- final source URL;
- page title;
- retrieval timestamp.

Store all citation fields with the embedding so generated answers can point back to the source.

### How much does it cost to prepare web pages for RAG?

The Actor uses pay-per-event pricing:

- **Start:** $0.005 once per run.
- **Page item:** tiered by your Apify plan; the current BRONZE price is $0.003196 per successful page.

At BRONZE pricing:

| Successful pages | BRONZE calculation | Estimated Actor charge |
|---:|---|---:|
| 1 | `0.005 + 1 × 0.003196` | 0.008196 USD |
| 10 | `0.005 + 10 × 0.003196` | 0.03696 USD |
| 50 | `0.005 + 50 × 0.003196` | 0.1648 USD |

Failed, rejected, duplicate, or skipped pages do not emit an item charge. Platform compute and transfer usage follow your Apify account terms. Prices shown here will be kept synchronized with active Actor pricing.

### RAG ingestion workflow

A common workflow is:

1. Keep canonical source URLs in your application.
2. Run this Actor on new or changed pages.
3. Read `chunks` from the default dataset.
4. Generate one embedding per chunk.
5. Store `citation` beside the vector.
6. Retrieve relevant chunks for a question.
7. Include the source URL and title in the generated answer.
8. Rerun on a schedule when source freshness matters.

The Actor does not detect page changes itself. Your scheduler or pipeline decides when to re-ingest.

### Agent research workflow

An autonomous agent can call the Actor after it has selected known source URLs.

Useful prompts include:

- “Prepare these three public policy pages as cited chunks.”
- “Extract the readable documentation and list same-origin follow-up links.”
- “Return Markdown and token estimates for these public sources.”

The Actor does not accept a search query and does not discover arbitrary sites from the open web.

### Run through the Apify API with cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~llm-web-page-research-browser/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url":"https://docs.apify.com/platform/actors"}],
    "maxPages": 1,
    "chunkSize": 1500,
    "chunkOverlap": 150
  }'
```

Poll the returned run or use the synchronous dataset-items endpoint when appropriate for your payload size.

### JavaScript API example

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/llm-web-page-research-browser').call({
  startUrls: [{ url: 'https://docs.apify.com/platform/actors' }],
  maxPages: 1,
  chunkSize: 1500,
  chunkOverlap: 150,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].chunks);
```

### Python API example

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/llm-web-page-research-browser').call(run_input={
    'startUrls': [{'url': 'https://docs.apify.com/platform/actors'}],
    'maxPages': 1,
    'chunkSize': 1500,
    'chunkOverlap': 150,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items[0]['chunks'])
```

### Use with MCP

Add the Apify MCP server to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/llm-web-page-research-browser"
```

#### Claude Desktop, Cursor, and VS Code setup

Claude Desktop, Cursor, and VS Code can use this equivalent MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/llm-web-page-research-browser"
    }
  }
}
```

Example prompt ideas for MCP usage:

> Run LLM Web Page Research Browser on the Apify Actors documentation URL with 1,500-character chunks, then summarize the result with citations.

> Prepare the supplied European Commission policy page as Markdown chunks and preserve every source citation.

> Extract same-origin follow-up links from this public documentation page without crawling them.

### Tips for reliable output

- Supply canonical article or documentation URLs instead of home pages.
- Use smaller `maxContentChars` when downstream token budgets are strict.
- Keep overlap around 10% of chunk size for general-purpose semantic retrieval.
- Disable full text or Markdown when only chunks are needed.
- Inspect `truncated` before assuming a record contains the complete source.
- Use a new run for a substantially different source batch so datasets remain easy to audit.

### Limits and failure behavior

This v1 deliberately excludes:

- web search and query-to-URL discovery;
- login-gated or private pages;
- universal browser rendering;
- PDFs, images, feeds, and other non-HTML media;
- automatic residential proxy fallback;
- crawling extracted links as additional pages;
- change detection, alerts, or legal conclusions.

A page can fail because of access controls, a JavaScript-only shell, an unsupported content type, a timeout, an oversized response, or no readable content.

If some pages succeed, the Actor stores those pages and logs failures. If every page fails, the run fails with a non-zero status instead of returning a misleading empty success.

### Legality

Only process pages you are authorized to access. Follow applicable site terms, robots guidance, copyright rules, privacy law, and contractual restrictions. Output is source preparation, not legal advice.

### Security and responsible use

Only process pages you are authorized to access. Follow applicable site terms, robots guidance, copyright rules, privacy law, and contractual restrictions.

The Actor blocks obvious local, loopback, link-local, and private-network destinations before requests and redirects. Do not use it to probe internal services or bypass access controls.

Extracted public text may still contain personal data or copyrighted material. Apply an appropriate retention and downstream-use policy.

### Troubleshooting

#### Why did the run say no pages were processed?

Check each logged URL. Confirm it is public HTML, loads without an account, and exposes readable server-rendered content. A browser-only shell is outside v1 scope.

#### Why is content shorter than the page I see?

Readable-content extraction removes menus and page chrome. Also check `truncated` and increase `maxContentChars` within the schema limit when appropriate.

#### Why are some links missing?

Only same-origin HTTP(S) links are returned, fragments are removed, duplicates are collapsed, and `maxLinks` caps the list.

#### Why is metadata null?

Authors, dates, descriptions, and language depend on source markup. The Actor returns `null` rather than inventing missing metadata.

#### Can I set overlap larger than chunk size?

No. `chunkOverlap` must be smaller than `chunkSize`; invalid combinations fail before fetching pages.

### FAQ

#### Does this Actor search the web?

No. It converts supplied public URLs. Pair it with your own URL discovery step when search is needed.

#### Does it run a browser?

No. Direct HTTP keeps runs bounded and economical. JavaScript-only rendering is not promised in v1.

#### Does it follow extracted links?

No. Links are output for inspection or a later explicitly controlled run.

#### Are citations generated by an LLM?

No. Citations are deterministic source URL, title, and retrieval-time fields attached to the extracted page and chunks.

#### Can I schedule recurring ingestion?

Yes. Use an Apify schedule and send each dataset to your vector-store or ETL integration. This Actor does not compare versions or send alerts.

#### Is this legal advice?

No. Even when processing legal or policy sources, output is source preparation rather than legal analysis.

### Related Automation Lab Actors

- [Schema-Guided Web Data to Excel](https://apify.com/automation-lab/schema-guided-web-data-to-excel) for extracting repeated fields into spreadsheet-ready rows.
- [Web Image Downloader](https://apify.com/automation-lab/web-image-downloader) when the required output is page-linked image files and metadata rather than research text.

Use LLM Web Page Research Browser when cited chunks and reusable research context are the primary output.

# Actor input Schema

## `startUrls` (type: `array`):

Anonymous public HTTP(S) pages to process. Login-gated pages, private networks, file URLs, and browser-only applications are not supported.

## `maxPages` (type: `integer`):

Maximum number of unique supplied URLs to process in this run.

## `maxContentChars` (type: `integer`):

Truncate cleaned page content after this many characters to keep RAG payloads bounded.

## `chunkSize` (type: `integer`):

Target maximum characters in each compact Markdown chunk.

## `chunkOverlap` (type: `integer`):

Characters repeated between neighboring chunks. Must be smaller than chunk size.

## `maxLinks` (type: `integer`):

Maximum normalized same-origin links extracted from each source page.

## `includeMarkdown` (type: `boolean`):

Include the complete cleaned Markdown field in each page record.

## `includeText` (type: `boolean`):

Include a plain-text representation in addition to chunks.

## `requestTimeoutSecs` (type: `integer`):

Maximum seconds for each HTTP request.

## `maxRetries` (type: `integer`):

Retries for timeouts, network failures, HTTP 429, and temporary server errors.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ],
  "maxPages": 10,
  "maxContentChars": 50000,
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "maxLinks": 100,
  "includeMarkdown": true,
  "includeText": true,
  "requestTimeoutSecs": 30,
  "maxRetries": 2
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing one record for every successfully processed page.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/llm-web-page-research-browser").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }] }

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/llm-web-page-research-browser").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ]
}' |
apify call automation-lab/llm-web-page-research-browser --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/llm-web-page-research-browser"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nxVmtd8KzzcyTigOJ/builds/E87GaWa0dGwszcmLm/openapi.json