Web Page Markdown Scraper

Turn supplied JavaScript-rendered public pages into web page markdown and normalized text for RAG ingestion. Each page record includes source and final URLs, title, description, language, declared canonical URL, UTC timestamp and a SHA-256 content hash.

Use it for a known list of documentation, public articles or reference pages. It is not a search engine, recursive crawler, authenticated browser or custom-code runner.

Who is it for?

• RAG engineers refreshing known public sources before embedding.
• AI-agent developers retrieving cited page context without writing DOM extraction code.
• Content teams comparing normalized text hashes between scheduled snapshots.

Why use it?

JavaScript can populate content after the initial HTML arrives. This Actor renders each supplied page in Chromium, then cleans the DOM offline. Readability selects article content; short pages can fall back to main/article/body. Navigation, scripts, forms and media are excluded. It returns both Markdown for downstream formatting and normalized text for embedding or comparison.

Unlike custom browser scrapers, no pageFunction or customer JavaScript is accepted. Unlike broad crawlers, it never follows discovered links. A fresh anonymous browser context is created for each attempt.

Getting started

1. Supply one or more anonymous public HTTP(S) page URLs.
2. Choose a maximum page count and optional rendered-content readiness selector.
3. Run the Actor and inspect its terminal status.
4. Export the default dataset as JSON or use Apify dataset exports.
5. Store source URL and content hash alongside each downstream document.

{
  "pageUrls": ["https://quotes.toscrape.com/js/"],
  "maxItems": 3,
  "pageTimeoutSecs": 15,
  "maxContentChars": 50000,
  "waitForSelector": ""
}

Input parameters

Input	Behavior
pageUrls	1–20 URL strings. Public HTTP on port 80 or HTTPS on port 443 only; no request objects.
maxItems	Global maximum supplied unique pages, 1–20, default 3. Applied after fragment removal and ordered deduplication.
pageTimeoutSecs	Browser attempt lifetime, 5–15 seconds, default 15. One transient retry may occur; pages requiring longer readiness are unsupported.
maxContentChars	Per-page cap separately for Markdown and text, 1,000–100,000, default 50,000.
waitForSelector	Optional visible CSS readiness selector, maximum 300 characters. Does not select extraction content.

Zero/unlimited values are not supported. Headers, credentials, cookies, proxies, search queries, recursive options and customer code are rejected. Unsupported keys fail validation instead of being silently ignored.

Extracted data

Field	Meaning
sourceUrl	Normalized supplied URL.
finalUrl	Public URL after navigation redirects.
title	Readability title or document title.
description	Meta description or null.
language	HTML language attribute or null.
canonicalUrl	Declared canonical URL or null; not fetched or independently verified.
markdown	Clean rendered content with absolute HTTP(S) links.
text	Readable text with normalized whitespace.
contentHash	SHA-256 of exactly the returned text.
truncated	True when either content format exceeds its character cap.
extractionMethod	readability or main-body.
scrapedAt	UTC extraction timestamp.

Output example

The JavaScript demo page returns a Quotes to Scrape record containing its rendered quotations. An abbreviated example:

{
  "sourceUrl": "https://quotes.toscrape.com/js/",
  "finalUrl": "https://quotes.toscrape.com/js/",
  "title": "Quotes to Scrape",
  "description": null,
  "language": "en",
  "canonicalUrl": null,
  "markdown": "“The world as we have created it is a process of our thinking...",
  "text": "“The world as we have created it is a process of our thinking...",
  "contentHash": "<SHA-256 of the complete returned text>",
  "truncated": false,
  "extractionMethod": "readability",
  "scrapedAt": "2026-10-12T00:00:00.000Z"
}

How much does it cost to extract rendered public pages?

Pricing is pay per event: one start event per accepted run and one item event per successfully saved page. Text, Markdown and metadata are included in the same page event, with no separate content-format charge. Invalid input is rejected before the start event. Failed pages produce no item event; earlier saved pages remain chargeable partial results.

The one-time run start costs $0.0005. Page prices depend on your qualifying aggregate monthly Apify Store spend tier, not a private per-Actor volume schedule:

Tier	Per saved page
FREE	$0.0077539
BRONZE	$0.0067425
SILVER	$0.0052592
GOLD	$0.0040455
PLATINUM	$0.0040455
DIAMOND	$0.0040455

At BRONZE, estimated Actor event charges are $0.0072425 for one page, $0.0342125 for five pages and $0.13535 for 20 pages, including one start event. Apify plan infrastructure charges may apply separately. The pricing panel is authoritative; these examples assume each page completes successfully.

Web page markdown extraction behavior

The browser keeps scripts, styles and XHR needed for rendering, while blocking image/media/font downloads and WebSockets. Service workers and downloads are disabled. Without a readiness selector it waits for bounded network idle and a nonempty body. It does not scroll, click buttons or unlock lazy-loaded sections. Provide a selector for content that appears later than ordinary page load.

The transport limit is 20 MiB per attempt; rendered HTML is capped at 2 MiB. DOM cleanup is heuristic: menus, tables, author labels or decorative text may be excluded. Inspect representative output before treating it as a complete archive. Raw HTML and screenshots are not exported.

Limits and failure behavior

Private/reserved destinations, IPv6 and nonstandard ports are rejected, including browser redirects and subresources. Connections use checked numeric addresses to prevent a second DNS lookup. Redirects to ordinary public HTTP(S) pages are allowed.

Login forms and recognizable challenge pages are outside scope. A 429, selected 5xx response, timeout or transient network failure receives one new-context retry. Stable invalid pages, unsupported content types, empty readable content and detected challenges fail without blind repeats.

A failed page stops the run with FAILED status. Any previously saved rows are partial, not proof that the full input completed. Pages need at least 20 readable characters. Supplied URL order is preserved; duplicate normalized URLs are processed once.

Integrations

Use Apify schedules to repeat the same known URL list. Compare hashes in your database before embedding unchanged text again. The Actor itself does not store cross-run state, send alerts or perform change detection.

For vector databases such as Pinecone or Qdrant, perform chunking and embedding in your own pipeline. Keep source URL, title, content hash and timestamp on every chunk for citations and refresh decisions. A webhook can notify your pipeline after a successful run; check status before ingesting partial data.

Export JSON for structured ingestion or CSV for a content inventory. Large content is best processed outside a language model's chat context.

API usage

Keep your Apify token secret and send it in the Authorization header.

curl -X POST 'https://api.apify.com/v2/acts/automation-lab~rendered-web-page-rag-context/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"pageUrls":["https://quotes.toscrape.com/js/"],"maxItems":1}'

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/rendered-web-page-rag-context').call({
  pageUrls: ['https://quotes.toscrape.com/js/'], maxItems: 1,
});
if (run.status !== 'SUCCEEDED') throw new Error(`Run ${run.id}: ${run.status}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems({ limit: 20 });

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/rendered-web-page-rag-context').call(
    run_input={'pageUrls': ['https://quotes.toscrape.com/js/'], 'maxItems': 1})
if run['status'] != 'SUCCEEDED':
    raise RuntimeError(f"Run {run['id']}: {run['status']}")
items = client.dataset(run['defaultDatasetId']).list_items(limit=20).items

MCP setup

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/rendered-web-page-rag-context"

Claude Desktop, Cursor and VS Code desktop/editor configuration:

{"mcpServers":{"apify":{"url":"https://mcp.apify.com?tools=automation-lab/rendered-web-page-rag-context"}}}

Configure authenticated access in your client. Discover actual tool names and schemas with scoped tools/list. Actor selection also adds run/data/abort helper tools; discovery is not execution permission.

Example prompt: “Retrieve rendered Markdown from this supplied public documentation URL, retain citations and tell me if content is truncated.”

For an authorized invocation, start once and retain the run ID and storage IDs. They are metadata, not content. If pending, follow the same run with bounded 2/4/8-second backoff capped at 10 seconds and a 120-second total deadline. Set your client timeout above the server wait plus transport margin. Recover a known run after client timeout; never restart merely to poll. Report pending at deadline or terminal failure honestly.

On success, read bounded untransformed dataset pages with explicit fields, offset and limit. Example consumer budget: 20 source rows per page, at most 100 rows or 64 KiB of UTF-8 content admitted to model context, whichever comes first. Enforce the byte cap host-side even for one oversized row; omit or summarize with disclosure. Track source offsets and report partial/truncated status and continuation offset. If your client cannot intercept oversized responses, do not claim a hard byte guarantee. Full exports belong in downstream storage. No universal token-saving claim is made.

Support, dependencies and storage

Report problems through this Actor's Apify Store Issues tab with a run link and a nonsensitive example URL. This Actor uses Playwright/Chromium, Mozilla Readability, JSDOM and Turndown; it does not use AI models or send content to an embedding provider. The generated icon is company artwork and implies no third-party affiliation.

Page processing is ephemeral and browser cookies are discarded when each attempt ends. Dataset rows and logs remain in your Apify storage under your account's retention settings until you delete them; delete the run and associated storage using Apify Console/API when no longer needed. The private diagnostic recipient is our self-hosted GlitchTip service (Germany); sanitized reports are deleted after 30 days. No cross-run browser cache or source archive is created.

Legality, responsible use and privacy

Use only pages you are entitled to access and process. Respect website terms, robots policies, copyright and applicable law. Public availability is not a blanket license to republish content or collect personal information. No login sessions are supported.

Failed operations send sanitized diagnostic input, exceptions and actor/build/run IDs to our private GlitchTip service for repair. Secret fields and URL queries are removed, and reports are retained for 30 days. Dataset output contains the supplied and final URLs; avoid putting sensitive information in URL query strings.

FAQ and troubleshooting

Why is output incomplete? Cleanup is heuristic and the Actor does not click or scroll. Use a readiness selector and inspect the extractionMethod/truncated fields. Content behind logins, challenges or WebSocket-only loading is unsupported.

Can it search or crawl my whole site? No. Supply the exact page URLs. It never follows links or accepts search keywords.

Does a changed timestamp change the hash? No. Only returned normalized text is hashed. Markdown-only formatting changes may not change it; clipped content hides changes beyond the cap.

Why did a public page fail? Check the run log for HTTP/content/time-limit or scope errors. A public hostname can resolve to a reserved/private address and is then blocked. Do not supply credentials or disable safety controls.

Does success mean full fidelity? No. It means the bounded rendered DOM yielded useful cleaned content, not a complete visual archive or verified canonical identity.

Related Actors

• HTML Readability to Markdown Converter for supplied HTML or pages that do not need rendering.
• Playwright Scraper — Custom DOM Records when you need developer-defined JSON extraction rather than turnkey page content.