# Web Page Markdown Scraper (`automation-lab/rendered-web-page-rag-context`) Actor

Turn supplied public JavaScript-rendered page URLs into clean Markdown and text with source metadata and content hashes for recurring RAG ingestion.

- **URL**: https://apify.com/automation-lab/rendered-web-page-rag-context.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.05 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Page Markdown Scraper

Turn supplied JavaScript-rendered public pages into **web page markdown** and normalized text for RAG ingestion. Each page record includes source and final URLs, title, description, language, declared canonical URL, UTC timestamp and a SHA-256 content hash.

Use it for a known list of documentation, public articles or reference pages. It is not a search engine, recursive crawler, authenticated browser or custom-code runner.

### Who is it for?

- RAG engineers refreshing known public sources before embedding.
- AI-agent developers retrieving cited page context without writing DOM extraction code.
- Content teams comparing normalized text hashes between scheduled snapshots.

### Why use it?

JavaScript can populate content after the initial HTML arrives. This Actor renders each supplied page in Chromium, then cleans the DOM offline. Readability selects article content; short pages can fall back to main/article/body. Navigation, scripts, forms and media are excluded. It returns both Markdown for downstream formatting and normalized text for embedding or comparison.

Unlike custom browser scrapers, no pageFunction or customer JavaScript is accepted. Unlike broad crawlers, it never follows discovered links. A fresh anonymous browser context is created for each attempt.

### Getting started

1. Supply one or more anonymous public HTTP(S) page URLs.
2. Choose a maximum page count and optional rendered-content readiness selector.
3. Run the Actor and inspect its terminal status.
4. Export the default dataset as JSON or use Apify dataset exports.
5. Store source URL and content hash alongside each downstream document.

```json
{
  "pageUrls": ["https://quotes.toscrape.com/js/"],
  "maxItems": 3,
  "pageTimeoutSecs": 15,
  "maxContentChars": 50000,
  "waitForSelector": ""
}
```

### Input parameters

| Input | Behavior |
| --- | --- |
| `pageUrls` | 1–20 URL strings. Public HTTP on port 80 or HTTPS on port 443 only; no request objects. |
| `maxItems` | Global maximum supplied unique pages, 1–20, default 3. Applied after fragment removal and ordered deduplication. |
| `pageTimeoutSecs` | Browser attempt lifetime, 5–15 seconds, default 15. One transient retry may occur; pages requiring longer readiness are unsupported. |
| `maxContentChars` | Per-page cap separately for Markdown and text, 1,000–100,000, default 50,000. |
| `waitForSelector` | Optional visible CSS readiness selector, maximum 300 characters. Does not select extraction content. |

Zero/unlimited values are not supported. Headers, credentials, cookies, proxies, search queries, recursive options and customer code are rejected. Unsupported keys fail validation instead of being silently ignored.

### Extracted data

| Field | Meaning |
| --- | --- |
| `sourceUrl` | Normalized supplied URL. |
| `finalUrl` | Public URL after navigation redirects. |
| `title` | Readability title or document title. |
| `description` | Meta description or null. |
| `language` | HTML language attribute or null. |
| `canonicalUrl` | Declared canonical URL or null; not fetched or independently verified. |
| `markdown` | Clean rendered content with absolute HTTP(S) links. |
| `text` | Readable text with normalized whitespace. |
| `contentHash` | SHA-256 of exactly the returned text. |
| `truncated` | True when either content format exceeds its character cap. |
| `extractionMethod` | `readability` or `main-body`. |
| `scrapedAt` | UTC extraction timestamp. |

### Output example

The JavaScript demo page returns a `Quotes to Scrape` record containing its rendered quotations. An abbreviated example:

```json
{
  "sourceUrl": "https://quotes.toscrape.com/js/",
  "finalUrl": "https://quotes.toscrape.com/js/",
  "title": "Quotes to Scrape",
  "description": null,
  "language": "en",
  "canonicalUrl": null,
  "markdown": "“The world as we have created it is a process of our thinking...",
  "text": "“The world as we have created it is a process of our thinking...",
  "contentHash": "<SHA-256 of the complete returned text>",
  "truncated": false,
  "extractionMethod": "readability",
  "scrapedAt": "2026-10-12T00:00:00.000Z"
}
```

### How much does it cost to extract rendered public pages?

Pricing is pay per event: one `start` event per accepted run and one `item` event per successfully saved page. Text, Markdown and metadata are included in the same page event, with no separate content-format charge. Invalid input is rejected before the start event. Failed pages produce no item event; earlier saved pages remain chargeable partial results.

The one-time run start costs **$0.0005**. Page prices depend on your qualifying aggregate monthly Apify Store spend tier, not a private per-Actor volume schedule:

| Tier | Per saved page |
| --- | ---: |
| FREE | $0.0077539 |
| BRONZE | $0.0067425 |
| SILVER | $0.0052592 |
| GOLD | $0.0040455 |
| PLATINUM | $0.0040455 |
| DIAMOND | $0.0040455 |

At BRONZE, estimated Actor event charges are $0.0072425 for one page, $0.0342125 for five pages and $0.13535 for 20 pages, including one start event. Apify plan infrastructure charges may apply separately. The pricing panel is authoritative; these examples assume each page completes successfully.

### Web page markdown extraction behavior

The browser keeps scripts, styles and XHR needed for rendering, while blocking image/media/font downloads and WebSockets. Service workers and downloads are disabled. Without a readiness selector it waits for bounded network idle and a nonempty body. It does not scroll, click buttons or unlock lazy-loaded sections. Provide a selector for content that appears later than ordinary page load.

The transport limit is 20 MiB per attempt; rendered HTML is capped at 2 MiB. DOM cleanup is heuristic: menus, tables, author labels or decorative text may be excluded. Inspect representative output before treating it as a complete archive. Raw HTML and screenshots are not exported.

### Limits and failure behavior

Private/reserved destinations, IPv6 and nonstandard ports are rejected, including browser redirects and subresources. Connections use checked numeric addresses to prevent a second DNS lookup. Redirects to ordinary public HTTP(S) pages are allowed.

Login forms and recognizable challenge pages are outside scope. A 429, selected 5xx response, timeout or transient network failure receives one new-context retry. Stable invalid pages, unsupported content types, empty readable content and detected challenges fail without blind repeats.

A failed page stops the run with FAILED status. Any previously saved rows are partial, not proof that the full input completed. Pages need at least 20 readable characters. Supplied URL order is preserved; duplicate normalized URLs are processed once.

### Integrations

Use Apify schedules to repeat the same known URL list. Compare hashes in your database before embedding unchanged text again. The Actor itself does not store cross-run state, send alerts or perform change detection.

For vector databases such as Pinecone or Qdrant, perform chunking and embedding in your own pipeline. Keep source URL, title, content hash and timestamp on every chunk for citations and refresh decisions. A webhook can notify your pipeline after a successful run; check status before ingesting partial data.

Export JSON for structured ingestion or CSV for a content inventory. Large content is best processed outside a language model's chat context.

### API usage

Keep your Apify token secret and send it in the Authorization header.

```bash
curl -X POST 'https://api.apify.com/v2/acts/automation-lab~rendered-web-page-rag-context/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"pageUrls":["https://quotes.toscrape.com/js/"],"maxItems":1}'
```

```javascript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/rendered-web-page-rag-context').call({
  pageUrls: ['https://quotes.toscrape.com/js/'], maxItems: 1,
});
if (run.status !== 'SUCCEEDED') throw new Error(`Run ${run.id}: ${run.status}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems({ limit: 20 });
```

```python
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/rendered-web-page-rag-context').call(
    run_input={'pageUrls': ['https://quotes.toscrape.com/js/'], 'maxItems': 1})
if run['status'] != 'SUCCEEDED':
    raise RuntimeError(f"Run {run['id']}: {run['status']}")
items = client.dataset(run['defaultDatasetId']).list_items(limit=20).items
```

### MCP setup

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/rendered-web-page-rag-context"
```

Claude Desktop, Cursor and VS Code desktop/editor configuration:

```json
{"mcpServers":{"apify":{"url":"https://mcp.apify.com?tools=automation-lab/rendered-web-page-rag-context"}}}
```

Configure authenticated access in your client. Discover actual tool names and schemas with scoped `tools/list`. Actor selection also adds run/data/abort helper tools; discovery is not execution permission.

Example prompt: “Retrieve rendered Markdown from this supplied public documentation URL, retain citations and tell me if content is truncated.”

For an authorized invocation, start once and retain the run ID and storage IDs. They are metadata, not content. If pending, follow the same run with bounded 2/4/8-second backoff capped at 10 seconds and a 120-second total deadline. Set your client timeout above the server wait plus transport margin. Recover a known run after client timeout; never restart merely to poll. Report pending at deadline or terminal failure honestly.

On success, read bounded untransformed dataset pages with explicit fields, offset and limit. Example consumer budget: 20 source rows per page, at most 100 rows or 64 KiB of UTF-8 content admitted to model context, whichever comes first. Enforce the byte cap host-side even for one oversized row; omit or summarize with disclosure. Track source offsets and report partial/truncated status and continuation offset. If your client cannot intercept oversized responses, do not claim a hard byte guarantee. Full exports belong in downstream storage. No universal token-saving claim is made.

### Support, dependencies and storage

Report problems through this Actor's Apify Store Issues tab with a run link and a nonsensitive example URL. This Actor uses Playwright/Chromium, Mozilla Readability, JSDOM and Turndown; it does not use AI models or send content to an embedding provider. The generated icon is company artwork and implies no third-party affiliation.

Page processing is ephemeral and browser cookies are discarded when each attempt ends. Dataset rows and logs remain in your Apify storage under your account's retention settings until you delete them; delete the run and associated storage using Apify Console/API when no longer needed. The private diagnostic recipient is our self-hosted GlitchTip service (Germany); sanitized reports are deleted after 30 days. No cross-run browser cache or source archive is created.

### Legality, responsible use and privacy

Use only pages you are entitled to access and process. Respect website terms, robots policies, copyright and applicable law. Public availability is not a blanket license to republish content or collect personal information. No login sessions are supported.

Failed operations send sanitized diagnostic input, exceptions and actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL queries are removed, and reports are retained for 30 days. Dataset output contains the supplied and final URLs; avoid putting sensitive information in URL query strings.

### FAQ and troubleshooting

**Why is output incomplete?** Cleanup is heuristic and the Actor does not click or scroll. Use a readiness selector and inspect the extractionMethod/truncated fields. Content behind logins, challenges or WebSocket-only loading is unsupported.

**Can it search or crawl my whole site?** No. Supply the exact page URLs. It never follows links or accepts search keywords.

**Does a changed timestamp change the hash?** No. Only returned normalized text is hashed. Markdown-only formatting changes may not change it; clipped content hides changes beyond the cap.

**Why did a public page fail?** Check the run log for HTTP/content/time-limit or scope errors. A public hostname can resolve to a reserved/private address and is then blocked. Do not supply credentials or disable safety controls.

**Does success mean full fidelity?** No. It means the bounded rendered DOM yielded useful cleaned content, not a complete visual archive or verified canonical identity.

### Related Actors

- [HTML Readability to Markdown Converter](https://apify.com/automation-lab/html-readability-markdown-converter) for supplied HTML or pages that do not need rendering.
- [Playwright Scraper — Custom DOM Records](https://apify.com/automation-lab/public-web-custom-browser-page-function) when you need developer-defined JSON extraction rather than turnkey page content.

# Changelog

This Actor's version history is a separate document: https://apify.com/automation-lab/rendered-web-page-rag-context/changelog.md

# Actor input Schema

## `pageUrls` (type: `array`):

Supply 1–20 anonymous public page URL strings: HTTP on port 80 or HTTPS on port 443 only. Request objects, methods, headers, sessions and custom code are unsupported. Fragments are removed and identical normalized URLs deduplicated in order. Private/reserved networks, IPv6 and cross-protocol or nonstandard ports are rejected, including redirects and subresources.

## `maxItems` (type: `integer`):

Global maximum unique supplied pages to process, in input order after deduplication. Default 3, range 1–20; zero/unlimited is unsupported. No links are followed. A failed page stops the run; any saved rows are partial.

## `pageTimeoutSecs` (type: `integer`):

Maximum browser lifetime per attempt, default 15 seconds, range 5–15. A transient network/timeout/429/selected 5xx failure gets one fresh-context retry. Challenges, login and invalid content are not retried.

## `maxContentChars` (type: `integer`):

Maximum returned characters separately for Markdown and normalized text per page. Default 50000, range 1000–100000. Longer content is clipped with truncated=true. SHA-256 is calculated from the returned text, not the raw DOM or timestamp.

## `waitForSelector` (type: `string`):

Optional CSS selector (maximum 300 characters) that must become visible before extraction; it controls readiness, not content selection. Without it, wait for bounded network idle and nonempty body. No scrolling or clicking is performed; content rendered later may be absent.

## Actor input object example

```json
{
  "pageUrls": [
    "https://quotes.toscrape.com/js/"
  ],
  "maxItems": 3,
  "pageTimeoutSecs": 15,
  "maxContentChars": 50000,
  "waitForSelector": ""
}
```

# Actor output Schema

## `dataset` (type: `string`):

Default dataset with one record per successfully processed supplied page.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "pageUrls": [
        "https://quotes.toscrape.com/js/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/rendered-web-page-rag-context").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "pageUrls": ["https://quotes.toscrape.com/js/"] }

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/rendered-web-page-rag-context").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "pageUrls": [
    "https://quotes.toscrape.com/js/"
  ]
}' |
apify call automation-lab/rendered-web-page-rag-context --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/rendered-web-page-rag-context"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iX5R3tVvayPE2UaqR/builds/aZp0Ui8dttn18NyCE/openapi.json