Website Source Code Downloader

Download source files from authorized public websites with explicit safety and volume limits. This website source code downloader crawls same-site pages, discovers referenced assets, stores the retrieved bytes, and exports a machine-readable file manifest.

It archives server-returned HTML, CSS, JavaScript, images, fonts, audio, and video. Every stored file includes its original URL, final URL, archive path, source relationship, HTTP metadata, byte size, SHA-256 digest, and key-value-store reference.

What does Website Source Code Downloader do?

The Actor starts from one or more public HTTP(S) pages. It follows page links on the same host up to your chosen depth and page count. It then downloads the asset types you selected from HTML and CSS references.

The result has two connected parts:

1. A dataset manifest with one typed record per attempted resource.
2. The successfully retrieved source bytes in the run key-value store.

Use archivePath to reconstruct a host-and-path directory tree. Use sha256 to detect duplicate or changed bytes in later authorized runs. Use discoveredFrom and relation to trace why a file belongs to the archive.

Who is this Actor for?

• Developers preparing an authorized static-site migration.
• Agencies inventorying client-owned public website assets.
• Archivists creating bounded snapshots of openly available material.
• QA engineers checking which files a deployment exposes.
• Security and performance teams reviewing public front-end dependencies.
• Data engineers feeding source-file manifests into storage or change pipelines.

This Actor is not a cloning service for server behavior. It retrieves only anonymously reachable responses returned to its HTTP client.

Why use this website source code downloader?

A single “save page” request rarely captures the relationships around a website. This Actor adds bounded crawling and a consistent record for each file.

Key differentiators include:

• Separate maxPages, maxDepth, and maxAssets controls.
• Per-file and cumulative byte ceilings.
• An elapsed run-time ceiling.
• Selectable HTML, CSS, JavaScript, image, font, and media types.
• CSS url(...) and @import discovery.
• Same-site page crawling with optional subdomains.
• Optional referenced CDN assets without following external page links.
• Public-network validation before requests and every redirect.
• File hashes and deterministic archive paths.
• Uncharged typed error rows when continuation is enabled.

What files and metadata are extracted?

Field	Meaning
url	Resource URL requested by the Actor.
finalUrl	Final URL after validated redirects.
rootUrl	Start URL that owns this crawl branch.
resourceType	html, css, javascript, image, font, or media.
statusCode	Successful HTTP response status.
mimeType	Validated response media type.
byteSize	Stored response size in bytes.
sha256	SHA-256 digest of the exact stored bytes.
archivePath	Host-and-path destination for reconstructing a file tree.
storageKey	File key in the run key-value store.
storageUrl	Direct URL for the stored bytes.
discoveredFrom	Page or stylesheet that referenced the resource.
relation	Discovery relationship such as link, script, or css-url.
depth	Same-site page-link depth from the start page.
retrievedAt	ISO 8601 retrieval timestamp.
status	stored or error.
errorType	Stable failure category for error rows.
errorMessage	Bounded diagnostic detail for error rows.

Nullable fields are null when a retrieval fails before that value exists. Error rows make partial archives auditable and do not incur a per-file event.

How to download website source code

1. Open the Actor in Apify Console.
2. Add one to 20 authorized public start URLs.
3. Choose the maximum page count and link depth.
4. Choose the file types and asset count.
5. Set byte and time budgets appropriate for the site.
6. Leave external assets enabled if CDN-hosted files belong in the archive.
7. Run the Actor.
8. Export the dataset manifest or open each storageUrl.

Start with a small crawl such as one page, depth zero, and ten assets. Increase bounds only after checking the site's structure and your authorization.

Input parameters

Input	Type	Default	Description
startUrls	URL array	required	One to 20 public HTTP(S) pages.
maxPages	integer	10	HTML pages attempted across all roots.
maxDepth	integer	1	Same-site link hops; 0 means supplied pages only.
maxAssets	integer	100	Non-page assets attempted after discovery.
assetTypes	string array	all six types	File classes to store.
maxFileBytes	integer	5000000	Maximum bytes accepted for one file.
maxTotalBytes	integer	50000000	Maximum cumulative stored bytes.
maxRunSecs	integer	240	Elapsed scheduling limit.
requestTimeoutSecs	integer	30	Per-request timeout.
includeExternalAssets	boolean	true	Include referenced CDN/third-party assets.
includeSubdomains	boolean	false	Follow page links onto subdomains.
continueOnError	boolean	true	Continue and emit error records.
proxyConfiguration	object	direct	Optional Apify or custom proxy configuration.

Page discovery always stays on the start host unless subdomains are enabled. External assets never cause external HTML pages to be crawled.

Example input

{
  "startUrls": [{ "url": "https://www.python.org/about/" }],
  "maxPages": 3,
  "maxDepth": 1,
  "maxAssets": 12,
  "assetTypes": ["html", "css", "javascript", "image", "font"],
  "maxFileBytes": 5000000,
  "maxTotalBytes": 15000000,
  "maxRunSecs": 120,
  "includeExternalAssets": true,
  "continueOnError": true
}

This exact workflow is included as a public Task example and is tested locally.

Example output

{
  "url": "https://www.python.org/about/",
  "finalUrl": "https://www.python.org/about/",
  "rootUrl": "https://www.python.org/about/",
  "resourceType": "html",
  "statusCode": 200,
  "mimeType": "text/html",
  "byteSize": 53912,
  "sha256": "1599d6b9d1d8f67a153f2d5804c5bde5e3c2b1854e2ed66e99036c7877acdc70",
  "archivePath": "www.python.org/about/index.html",
  "storageKey": "FILE_09b8a7c6d5e4f321_index.html",
  "storageUrl": "https://api.apify.com/v2/key-value-stores/STORE_ID/records/FILE_09b8a7c6d5e4f321_index.html",
  "discoveredFrom": null,
  "relation": "start-url",
  "depth": 0,
  "retrievedAt": "2026-01-15T12:00:00.000Z",
  "status": "stored",
  "errorType": null,
  "errorMessage": null
}

Values vary as websites deploy new content. The schema and field types remain stable for integrations.

How files are organized

The dataset is the archive manifest. The key-value store contains bytes under keys beginning with FILE_.

archivePath combines the final hostname and normalized URL path. Directory-like HTML URLs receive index.html. A short query hash distinguishes URL variants that share a path.

The Actor does not currently produce a ZIP or rewrite links inside documents. A downstream workflow can download the files and place each one at archivePath.

How much does it cost to download website source files?

The Actor uses pay per event pricing. There is one small start event and one file event for every successfully stored file. Error rows are not charged as files.

At the BRONZE rate of $0.00064 per stored file plus a $0.001 start:

Stored files	Example charge
1	$0.00164
10	$0.00740
100	$0.06500
1,000	$0.64100

Your active Apify plan tier determines the exact file price shown in Console. Proxy traffic and platform usage may be billed separately according to your Apify plan. Final limits, redirects, duplicates, and failed resources affect the actual count.

Limits and stopping behavior

The Actor stops scheduling pages at maxPages and assets at maxAssets. It stops adding files before maxTotalBytes would be exceeded. Responses over maxFileBytes become error rows. No new retrieval starts after maxRunSecs.

Each request has one bounded retry for transient 408, 429, and selected 5xx responses. Redirects are limited to ten and every destination is validated again. Malformed inputs and private-network destinations fail closed.

continueOnError: false makes the run fail at the first resource error. With the default true, independent resources continue and failures remain visible.

Reliability tips

• Begin with one start URL and depth zero.
• Raise maxPages only after inspecting internal navigation.
• Disable media for faster, smaller source-code inventories.
• Keep external assets enabled for sites that use a CDN.
• Disable external assets for a strict same-host archive.
• Use a proxy only when the authorized target requires that route.
• Avoid very high limits on sites with calendar or faceted URL spaces.
• Compare hashes rather than timestamps when detecting byte-level changes.

A site can return different files by geography, headers, cookies, or deployment time. This Actor records what its anonymous HTTP client received during the run.

Website migration workflow

For an authorized migration:

1. Run a conservative inventory with HTML, CSS, JavaScript, images, and fonts.
2. Review errors and increase only the necessary byte bounds.
3. Export dataset rows as JSON or CSV.
4. Download each successful storageUrl.
5. Write the bytes to the corresponding archivePath.
6. Use discoveredFrom to investigate missing dependencies.
7. Re-run near cutover and compare sha256 values.

This captures public front-end files. It does not recover databases, server code, private APIs, forms, or authentication.

Change-detection workflow

Schedule bounded runs with the same inputs. Join records by final URL or archive path. Compare sha256, byteSize, and status between runs.

A changed hash means the retrieved bytes changed. It does not explain semantic meaning, so downstream review may still be needed. The Actor itself does not send alerts or retain cross-run state.

Integrations

Dataset rows work with Apify integrations and webhooks. Common destinations include:

• Google Sheets for compact manifests.
• Cloud storage for long-term authorized archives.
• Webhooks that start a file-copy or hash-comparison job.
• Make or Zapier for low-code review workflows.
• Python or Node.js pipelines that reconstruct archivePath trees.

Large binary files remain in the key-value store rather than being embedded in rows. This keeps dataset exports useful and avoids base64 inflation.

Run with the Apify API using cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-source-code-downloader/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://www.python.org/about/"}],
    "maxPages": 1,
    "maxDepth": 0,
    "maxAssets": 10
  }'

Poll the returned run ID, then read its default dataset and key-value store. Keep API tokens in a secret manager rather than source code.

Run with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-source-code-downloader').call({
  startUrls: [{ url: 'https://www.python.org/about/' }],
  maxPages: 1,
  maxDepth: 0,
  maxAssets: 10,
  assetTypes: ['html', 'css', 'javascript'],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Use each successful row's storageUrl to retrieve its source bytes.

Run with Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/website-source-code-downloader").call(run_input={
    "startUrls": [{"url": "https://www.w3.org/standards/"}],
    "maxPages": 3,
    "maxDepth": 1,
    "maxAssets": 10,
    "assetTypes": ["html", "css", "image"],
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["status"], item["archivePath"])

Store the token outside notebooks and repositories.

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-source-code-downloader"

Claude Desktop

Add this remote server to your Claude Desktop MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-source-code-downloader"
    }
  }
}

Cursor

In Cursor MCP settings, add the same apify server name and remote URL shown above.

VS Code

In VS Code MCP settings, register the same HTTP URL as the apify server.

Example prompts:

• “Archive one Python.org page and ten code assets, with no link crawling.”
• “Build a five-page source-file manifest for this authorized public site.”
• “Return only stored CSS and image records with their archive paths and hashes.”

Always include realistic limits in autonomous-agent prompts.

Responsible and legal use

Use this Actor only for websites and files you are authorized to retrieve. Follow applicable law, contracts, terms, copyright rules, and website policies. Do not use it to bypass authentication, paywalls, access controls, or technical blocks.

The Actor rejects local and private-network destinations to reduce SSRF risk. That safeguard does not determine whether you have permission to archive a public URL. You are responsible for selecting targets, limits, retention, and downstream use.

Troubleshooting

Why do I see UNEXPECTED_CONTENT_TYPE?

A URL advertised as one file type returned another media type. This can happen with error pages, bot challenges, login redirects, or stale asset links. Inspect the URL and choose a route the site permits; do not simply raise limits.

Why are some CDN files absent?

Confirm includeExternalAssets is true and that the file type is selected. Assets created only by client-side JavaScript are outside this HTTP-only workflow. The Actor archives server-returned HTML and statically referenced dependencies.

Why did crawling stop early?

Check maxPages, maxAssets, maxTotalBytes, and maxRunSecs. The run log reports attempted pages, attempted assets, stored files, failures, and bytes. Duplicate URL-and-type pairs are intentionally fetched only once.

Can it download authenticated pages?

No. The supported contract is anonymously reachable public HTTP(S) content. URLs containing credentials are rejected.

Frequently asked questions

Does it create a working offline clone?

It creates source bytes and a reconstruction manifest. It does not rewrite links, emulate a server, copy databases, or guarantee interactive behavior.

Does it render JavaScript?

No. It downloads JavaScript files referenced by server-returned HTML but does not run a browser. This keeps the workflow faster and more predictable for static source archives.

Are failed downloads charged?

No file event is charged for an error row. The one-time start event still applies to a run that initializes successfully.

Can I crawl multiple sites in one run?

Yes, up to 20 start URLs. Each start URL owns a separate same-site page branch, while limits apply across the run.

How long are files available?

They follow the retention rules of your Apify account and run storage. Copy files to long-term storage when the archive must be retained.

Related automation-lab Actors

• Public Webpage HTML Downloader for supplied-page raw or browser-rendered HTML without site crawling.
• Website Image Extractor for image URLs, srcset variants, alt text, dimensions, and page context.
• Website HTML & Text Change Monitor for versioned page snapshots and text-level differences.
• Broken Link Checker for validating site links rather than storing source bytes.

Choose the narrower Actor when you do not need a complete bounded file manifest.