LLM Web Page Research Browser

Turn anonymous public web pages into clean Markdown, plain text, compact chunks, metadata, links, and citations for LLM research and retrieval-augmented generation (RAG).

This Actor provides bounded LLM web scraping for supplied URLs. It does not search the web, log in to websites, or pretend every JavaScript application can be read without a browser.

What does LLM Web Page Research Browser do?

For every successfully fetched page, the Actor:

• follows up to five public HTTP redirects;
• isolates readable article or main-page content;
• converts that content to Markdown and plain text;
• extracts title, description, author, publisher, language, and publication date when exposed;
• extracts and deduplicates same-origin links;
• splits Markdown into overlapping, retrieval-ready chunks;
• attaches a source URL, title, and retrieval timestamp to the page and every chunk;
• reports word, character, and approximate token counts.

One dataset item represents one successfully processed page. Failed pages are logged and are not charged as items.

Who is this Actor for?

• RAG engineers preparing source documents for a vector database.
• AI agent builders who need cited context from known URLs.
• Researchers and analysts collecting readable source material for grounded synthesis.
• Documentation teams transforming public guides into stable ingestion records.
• Compliance teams preparing public policy pages for internal search, while retaining citations.

Choose this Actor when you already know the source URLs. It is not a web search API.

Why use it for LLM web scraping?

Raw HTML is noisy and expensive to send to an LLM. This Actor removes navigation and common page chrome, normalizes content, and returns both complete page fields and compact chunks.

The output is deterministic enough for repeated ingestion:

• bounded URL and page counts;
• bounded download and content sizes;
• configurable chunks and overlap;
• stable page-level records;
• citations carried into every chunk;
• no hidden browser or residential-proxy fallback.

Supported pages

The Actor supports anonymously reachable public HTTP(S) HTML pages that expose useful server-rendered content.

Typical sources include:

• documentation pages;
• public articles and reference pages;
• public policy and government pages;
• blogs and knowledge bases;
• public product or company information pages.

The Actor rejects local and private network destinations, credential-bearing URLs, non-HTTP protocols, and non-HTML responses.

Getting started

1. Open the Actor in Apify Console.
2. Add one or more public page URLs to Public page URLs.
3. Keep the default chunk size and overlap for a first run.
4. Set Maximum pages to bound the run.
5. Click Start.
6. Open the default dataset to inspect page records and cited chunks.
7. Export JSON or connect the dataset to your ingestion workflow.

A practical first input is:

{
  "startUrls": [
    { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" }
  ],
  "maxPages": 1,
  "chunkSize": 2000,
  "chunkOverlap": 200,
  "maxContentChars": 50000,
  "maxLinks": 100,
  "includeMarkdown": true,
  "includeText": true
}

Input parameters

Field	Type	Default	Meaning
startUrls	array	required	Public HTTP(S) pages to process.
maxPages	integer	10	Maximum unique supplied URLs processed, up to 50.
maxContentChars	integer	50000	Maximum cleaned content characters per page.
chunkSize	integer	2000	Target maximum characters per chunk.
chunkOverlap	integer	200	Repeated characters between chunks; must be smaller than chunkSize.
maxLinks	integer	100	Maximum same-origin links retained per page.
includeMarkdown	boolean	true	Include complete cleaned Markdown.
includeText	boolean	true	Include complete cleaned plain text.
requestTimeoutSecs	integer	30	Timeout per request, from 5 to 120 seconds.
maxRetries	integer	2	Retries for network errors, timeouts, HTTP 429, and temporary 5xx errors.

At most 50 URLs are accepted in one input. Each response is capped at 2.5 MB before parsing.

Output fields

Field	Description
url	Final source URL after public redirects.
requestedUrl	URL supplied by the user.
title	Readable page or article title.
description	Metadata description, when exposed.
author	Author or byline, when exposed.
siteName	Publisher or site name, when exposed.
language	Declared or extracted language.
publishedAt	Source publication time, when exposed.
retrievedAt	UTC retrieval timestamp.
statusCode	Successful HTTP status.
contentType	Source response media type.
markdown	Cleaned Markdown, or null when disabled.
text	Cleaned plain text, or null when disabled.
charCount	Markdown character count.
wordCount	Approximate word count.
tokenEstimate	Approximate tokens at four characters per token.
truncated	Whether content reached maxContentChars.
links	Deduplicated same-origin URL and anchor-text pairs.
chunks	Compact overlapping Markdown chunks with citations.
citation	Page-level source URL, title, and retrieval time.

Metadata fields are nullable because not every publisher exposes them.

Example output

The following shortened shape comes from the current Wikipedia RAG example:

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "requestedUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation - Wikipedia",
  "language": "en",
  "statusCode": 200,
  "contentType": "text/html; charset=UTF-8",
  "wordCount": 2580,
  "tokenEstimate": 7500,
  "truncated": true,
  "chunks": [
    {
      "id": "en.wikipedia.org-001",
      "text": "Retrieval-augmented generation (RAG) is a technique...",
      "charCount": 1772,
      "wordCount": 169,
      "citation": {
        "sourceUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
        "title": "Retrieval-augmented generation - Wikipedia",
        "retrievedAt": "2026-08-24T20:10:00.000Z"
      }
    }
  ]
}

The full record also includes Markdown, text, metadata, and extracted links.

Chunking and citations

Chunks are created from cleaned Markdown rather than raw HTML.

The splitter prefers a paragraph or sentence boundary near the target size. Neighboring chunks repeat the requested overlap so downstream retrieval does not lose context at a boundary.

Every chunk includes:

• a stable-in-record sequence ID;
• chunk text;
• character and word counts;
• final source URL;
• page title;
• retrieval timestamp.

Store all citation fields with the embedding so generated answers can point back to the source.

How much does it cost to prepare web pages for RAG?

The Actor uses pay-per-event pricing:

• Start: $0.005 once per run.
• Page item: tiered by your Apify plan; the current BRONZE price is $0.003196 per successful page.

At BRONZE pricing:

Successful pages	BRONZE calculation	Estimated Actor charge
1	0.005 + 1 × 0.003196	0.008196 USD
10	0.005 + 10 × 0.003196	0.03696 USD
50	0.005 + 50 × 0.003196	0.1648 USD

Failed, rejected, duplicate, or skipped pages do not emit an item charge. Platform compute and transfer usage follow your Apify account terms. Prices shown here will be kept synchronized with active Actor pricing.

RAG ingestion workflow

A common workflow is:

1. Keep canonical source URLs in your application.
2. Run this Actor on new or changed pages.
3. Read chunks from the default dataset.
4. Generate one embedding per chunk.
5. Store citation beside the vector.
6. Retrieve relevant chunks for a question.
7. Include the source URL and title in the generated answer.
8. Rerun on a schedule when source freshness matters.

The Actor does not detect page changes itself. Your scheduler or pipeline decides when to re-ingest.

Agent research workflow

An autonomous agent can call the Actor after it has selected known source URLs.

Useful prompts include:

• “Prepare these three public policy pages as cited chunks.”
• “Extract the readable documentation and list same-origin follow-up links.”
• “Return Markdown and token estimates for these public sources.”

The Actor does not accept a search query and does not discover arbitrary sites from the open web.

Run through the Apify API with cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~llm-web-page-research-browser/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url":"https://docs.apify.com/platform/actors"}],
    "maxPages": 1,
    "chunkSize": 1500,
    "chunkOverlap": 150
  }'

Poll the returned run or use the synchronous dataset-items endpoint when appropriate for your payload size.

JavaScript API example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/llm-web-page-research-browser').call({
  startUrls: [{ url: 'https://docs.apify.com/platform/actors' }],
  maxPages: 1,
  chunkSize: 1500,
  chunkOverlap: 150,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].chunks);

Python API example

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/llm-web-page-research-browser').call(run_input={
    'startUrls': [{'url': 'https://docs.apify.com/platform/actors'}],
    'maxPages': 1,
    'chunkSize': 1500,
    'chunkOverlap': 150,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items[0]['chunks'])

Use with MCP

Add the Apify MCP server to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/llm-web-page-research-browser"

Claude Desktop, Cursor, and VS Code setup

Claude Desktop, Cursor, and VS Code can use this equivalent MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/llm-web-page-research-browser"
    }
  }
}

Example prompt ideas for MCP usage:

Run LLM Web Page Research Browser on the Apify Actors documentation URL with 1,500-character chunks, then summarize the result with citations.

Prepare the supplied European Commission policy page as Markdown chunks and preserve every source citation.

Extract same-origin follow-up links from this public documentation page without crawling them.

Tips for reliable output

• Supply canonical article or documentation URLs instead of home pages.
• Use smaller maxContentChars when downstream token budgets are strict.
• Keep overlap around 10% of chunk size for general-purpose semantic retrieval.
• Disable full text or Markdown when only chunks are needed.
• Inspect truncated before assuming a record contains the complete source.
• Use a new run for a substantially different source batch so datasets remain easy to audit.

Limits and failure behavior

This v1 deliberately excludes:

• web search and query-to-URL discovery;
• login-gated or private pages;
• universal browser rendering;
• PDFs, images, feeds, and other non-HTML media;
• automatic residential proxy fallback;
• crawling extracted links as additional pages;
• change detection, alerts, or legal conclusions.

A page can fail because of access controls, a JavaScript-only shell, an unsupported content type, a timeout, an oversized response, or no readable content.

If some pages succeed, the Actor stores those pages and logs failures. If every page fails, the run fails with a non-zero status instead of returning a misleading empty success.

Legality

Only process pages you are authorized to access. Follow applicable site terms, robots guidance, copyright rules, privacy law, and contractual restrictions. Output is source preparation, not legal advice.

Security and responsible use

Only process pages you are authorized to access. Follow applicable site terms, robots guidance, copyright rules, privacy law, and contractual restrictions.

The Actor blocks obvious local, loopback, link-local, and private-network destinations before requests and redirects. Do not use it to probe internal services or bypass access controls.

Extracted public text may still contain personal data or copyrighted material. Apply an appropriate retention and downstream-use policy.

Troubleshooting

Why did the run say no pages were processed?

Check each logged URL. Confirm it is public HTML, loads without an account, and exposes readable server-rendered content. A browser-only shell is outside v1 scope.

Why is content shorter than the page I see?

Readable-content extraction removes menus and page chrome. Also check truncated and increase maxContentChars within the schema limit when appropriate.

Why are some links missing?

Only same-origin HTTP(S) links are returned, fragments are removed, duplicates are collapsed, and maxLinks caps the list.

Why is metadata null?

Authors, dates, descriptions, and language depend on source markup. The Actor returns null rather than inventing missing metadata.

Can I set overlap larger than chunk size?

No. chunkOverlap must be smaller than chunkSize; invalid combinations fail before fetching pages.

FAQ

Does this Actor search the web?

No. It converts supplied public URLs. Pair it with your own URL discovery step when search is needed.

Does it run a browser?

No. Direct HTTP keeps runs bounded and economical. JavaScript-only rendering is not promised in v1.

Does it follow extracted links?

No. Links are output for inspection or a later explicitly controlled run.

Are citations generated by an LLM?

No. Citations are deterministic source URL, title, and retrieval-time fields attached to the extracted page and chunks.

Can I schedule recurring ingestion?

Yes. Use an Apify schedule and send each dataset to your vector-store or ETL integration. This Actor does not compare versions or send alerts.

Is this legal advice?

No. Even when processing legal or policy sources, output is source preparation rather than legal analysis.

Related Automation Lab Actors

• Schema-Guided Web Data to Excel for extracting repeated fields into spreadsheet-ready rows.
• Web Image Downloader when the required output is page-linked image files and metadata rather than research text.

Use LLM Web Page Research Browser when cited chunks and reusable research context are the primary output.