Public Webpage HTML Downloader

Download raw response HTML or browser-rendered DOM files from batches of anonymous public webpage URLs. Each successful page is saved as an individual .HTML file, while the default dataset provides its final URL, HTTP status, content type, byte size, retrieval mode, title, timestamp, and download link.

This Actor is designed for repeatable webpage HTML export: archival jobs, downstream parsers, migration audits, evidence capture, and scheduled data pipelines. It handles a webpage HTML only—rather than images, stylesheets, videos, or a full offline website bundle—so results stay focused and integration-friendly.

What does Public Webpage HTML Downloader do?

For every supplied URL, the Actor:

1. validates that the target is a public HTTP or HTTPS address;
2. downloads the server response in raw mode or opens it in Chromium in rendered mode;
3. follows only redirects that remain on public addresses;
4. enforces a configurable timeout and HTML size ceiling;
5. saves successful HTML to the run's key-value store;
6. writes retrieval metadata or a bounded error to the default dataset.

Unlike a generic scraper template, the output is ready to archive or pass directly to another parser. Duplicate input URLs are processed once, private-network targets are rejected, and an error for one URL can be isolated without losing the rest of a batch.

Who is it for?

• Data engineers staging HTML before parsing or enrichment.
• Archivists and researchers preserving public page snapshots with retrieval metadata.
• SEO and migration teams collecting source documents before link, metadata, or template audits.
• Automation builders passing stored HTML links into Make, Zapier, n8n, Python, or Apify workflows.
• Developers reproducing public-page responses and comparing raw HTML with the browser DOM.

Use a source-specific scraper when you need normalized products, reviews, profiles, or other domain entities instead of HTML files.

Raw HTML or browser-rendered HTML?

Mode	Best for	What is stored	Resource profile
raw	Server-rendered pages, APIs returning HTML, efficient bulk archives	Exact decoded HTML response after redirects	Fastest and cheapest
rendered	JavaScript-driven pages whose useful DOM appears after load	Chromium DOM from page.content()	More compute-intensive

Rendered mode blocks images, media, and fonts to reduce transfer and runtime. It does not automatically switch to a proxy, residential IP, login, CAPTCHA solver, or another paid route. A protected page returns an explicit error rather than creating an unexpected cost.

Input parameters

Field	Type	Default	Description
startUrls	array	required	Public HTTP(S) webpage URLs. Request-list objects and plain strings are accepted.
retrievalMode	raw or rendered	raw	Select server response HTML or a Chromium-rendered DOM.
maxItems	integer	20	Maximum unique URLs processed, from 1 to 1,000.
requestTimeoutSecs	integer	30	Per-URL timeout, from 5 to 120 seconds.
renderWaitSecs	number	0	Rendered mode only: extra wait after DOMContentLoaded, from 0 to 30 seconds.
maxHtmlBytes	integer	5000000	Per-page safety ceiling, from 10 KB to 20 MB.
continueOnError	boolean	true	Save an uncharged error row and continue, or fail on the first URL error.

Localhost, private IP ranges, URL credentials, and non-HTTP protocols are rejected. The Actor accepts supplied URLs only; it does not crawl links discovered within a page.

How to download webpage HTML

1. Open the Actor in Apify Console.
2. Add one or more anonymously reachable public webpages under Public webpage URLs.
3. Keep Raw HTTP response for normal HTML pages, or choose Browser-rendered DOM for JavaScript content.
4. Keep the default 5 MB limit unless you know larger documents are required.
5. Click Start.
6. Open Downloaded webpage metadata for status and file links.
7. Open Stored HTML files when you need all files in the run's key-value store.

A useful first run is:

{
  "startUrls": [
    { "url": "https://en.wikipedia.org/wiki/Web_scraping" },
    { "url": "https://news.ycombinator.com/" }
  ],
  "retrievalMode": "raw",
  "maxItems": 2
}

Output data

The default dataset contains one row per processed unique URL. A representative successful record is:

{
  "sourceUrl": "https://news.ycombinator.com/",
  "finalUrl": "https://news.ycombinator.com/",
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "byteSize": 34502,
  "retrievalMode": "raw",
  "retrievedAt": "2026-08-21T14:10:00.000Z",
  "title": "Hacker News",
  "htmlKey": "PAGE_0002_0f63a2a5a562.HTML",
  "htmlUrl": "https://api.apify.com/v2/key-value-stores/STORE_ID/records/PAGE_0002_0f63a2a5a562.HTML",
  "status": "success",
  "errorType": null,
  "errorMessage": null
}

htmlKey and htmlUrl are null for errors. statusCode, finalUrl, and content metadata may also be null when no valid HTTP response was received. Error messages are bounded to 500 characters and are intended for diagnosis, not as stored response bodies.

How are HTML files named and stored?

Successful files use keys such as PAGE_0001_9e0c729bb0e8.HTML. The sequence preserves input order and the short hash keeps names stable enough to identify the source without embedding a long URL. Files live in each run's default key-value store, so separate runs do not mix archives.

Dataset exports contain metadata, not the full HTML body. This keeps CSV, Excel, JSON, and integration payloads manageable while preserving direct file access through htmlUrl. Storage retention follows your Apify account and storage settings.

How much does it cost to download public webpage HTML?

Pay-per-event pricing includes a $0.001 run-start charge and one Webpage downloaded event for every successfully stored HTML file. Error rows are not charged as webpage downloads. The per-page rate decreases by Apify subscription tier:

Tier	Price per successful page
FREE	$0.00092
BRONZE	$0.00080
SILVER	$0.000624
GOLD	$0.00048
PLATINUM	$0.00032
DIAMOND	$0.000224

Examples at the BRONZE rate:

• 1 successful page: about $0.0018 including run start.
• 10 successful pages: about $0.009 including run start.
• 100 successful pages: about $0.081 including run start.

Compute-heavy rendered runs still use the same event curve; the price is based on measured safe operation. Your exact total depends on successful files and your active Apify tier.

Batch archival and downstream workflows

Common patterns include:

1. Scheduled archive — run daily with a stable URL list, then copy htmlUrl, retrievedAt, and statusCode into an archive index.
2. Parser staging — download pages once and let several extraction jobs consume the same immutable run files.
3. Raw-versus-rendered audit — run the same URLs in each mode and compare byteSize, title, and stored DOM.
4. Website migration evidence — preserve selected old pages before a deployment and selected new pages afterward.
5. Failure queue — filter status = error, inspect errorType, and retry only the affected public URLs with an adjusted timeout or mode.

The Actor does not calculate changes between runs. For versioned diffs, use Website HTML & Text Change Monitor.

Run with the Apify API using cURL

Replace APIFY_TOKEN with your token:

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~public-webpage-html-downloader/runs?token=APIFY_TOKEN&waitForFinish=120" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url":"https://news.ycombinator.com/"}],
    "retrievalMode": "raw",
    "maxItems": 1
  }'

Use the returned defaultDatasetId for metadata and defaultKeyValueStoreId for HTML files. For asynchronous production jobs, omit waitForFinish and poll the run endpoint.

Run with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/public-webpage-html-downloader').call({
  startUrls: [
    { url: 'https://en.wikipedia.org/wiki/Web_scraping' },
    { url: 'https://news.ycombinator.com/' },
  ],
  retrievalMode: 'raw',
  maxItems: 2,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map(({ sourceUrl, htmlUrl, status }) => ({ sourceUrl, htmlUrl, status })));

Download a successful file from its htmlUrl, or use client.keyValueStore(run.defaultKeyValueStoreId).getRecord(htmlKey).

Run with Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/public-webpage-html-downloader').call(run_input={
    'startUrls': [{'url': 'https://docs.npmjs.com/about-npm'}],
    'retrievalMode': 'raw',
    'maxItems': 1,
})

for row in client.dataset(run['defaultDatasetId']).iterate_items():
    print(row['status'], row.get('htmlUrl'))

Keep tokens in environment variables or a secret manager, never in source control.

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/public-webpage-html-downloader"

Claude Desktop, Cursor, and VS Code setup

Claude Desktop, Cursor, and VS Code can use this MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/public-webpage-html-downloader"
    }
  }
}

Example prompts:

• "Download raw HTML for these three public documentation URLs and return the stored file links."
• "Capture the browser-rendered DOM of this public JavaScript page and report its final URL and byte size."
• "Run the HTML archive and list only URLs with bounded retrieval errors."

Reliability, limits, and error behavior

The Actor intentionally uses bounded behavior:

• up to 1,000 unique URLs per run;
• up to 20 MB of HTML per page;
• up to 120 seconds per URL;
• up to 30 seconds of extra render wait;
• no recursive link crawling;
• no automatic proxy or login fallback;
• sequential processing to favor stability and bounded browser memory.

continueOnError: true saves errors alongside successes. Set it to false when a pipeline should fail fast. HTTP error pages that contain HTML can still be stored successfully with their actual status code, which is useful for auditing 404 or 503 documents. Non-HTML responses are rejected.

Troubleshooting

The stored page is an empty JavaScript shell. Use retrievalMode: "rendered" and, if necessary, add a small renderWaitSecs value. The Actor waits for DOMContentLoaded before the optional delay.

The page times out. Confirm it is anonymously reachable, then raise requestTimeoutSecs within the 120-second limit. Do not repeatedly retry a login wall or CAPTCHA.

I received PRIVATE_ADDRESS. The hostname resolved to localhost, a private network, link-local space, or another non-public address. This safety boundary cannot be disabled.

I received HTML_TOO_LARGE. Raise maxHtmlBytes only if you trust the public source and need the entire document. The maximum accepted value is 20 MB.

The dataset has an error row but no HTML file. Files are created only after a valid HTML result passes all checks. Use errorType and the run log to diagnose that URL.

Responsible and legal use

Download only public pages you are authorized to access and process. Respect website terms, robots guidance where applicable, copyright, database rights, rate limits, and privacy laws. Do not use the Actor to bypass authentication, paywalls, access controls, or technical protections.

HTML can contain personal data, copyrighted material, scripts, and unsafe markup. Treat stored files as untrusted input: sanitize before displaying them, and avoid executing downloaded scripts in privileged environments. You are responsible for retention and deletion policies for your run storage.

Related Automation Lab Actors

• Website HTML & Text Change Monitor — create versioned snapshots and machine-readable page changes.
• Webpage Text Extractor — return readable text instead of raw HTML files.
• HTML Table to Excel Exporter — turn native webpage tables into XLSX worksheets.

These products provide normalized or comparative outputs. Choose Public Webpage HTML Downloader when the reusable HTML document itself is the required artifact.

FAQ

Does it download a complete website with CSS, images, and JavaScript files?

No. It stores one HTML document per supplied URL. It does not create an offline mirror or rewrite asset links.

Can it download pages behind a login?

No. The supported scope is anonymously reachable public webpages. There are no credential or cookie inputs.

Does rendered mode capture the visual page?

It captures the post-JavaScript DOM as HTML, not a screenshot, PDF, video, or network archive.

Are failed URLs charged as downloaded webpages?

No. A bounded error row is saved for diagnosis, but the per-page event is charged only after HTML has been stored successfully. The one-time run-start event still applies.

Can I schedule recurring archives?

Yes. Save an Actor Task with a stable URL list and attach an Apify schedule. Each run gets its own dataset and key-value store identifiers for downstream indexing.

Why is raw mode the default?

Raw mode is faster, cheaper, and closer to the response a downstream parser receives. Use rendered mode only when JavaScript-generated DOM content is required.