# Website Source Code Downloader (`automation-lab/website-source-code-downloader`) Actor

Archive HTML, CSS, JavaScript, images, fonts, and media from authorized public websites with bounded crawling and file-level metadata.

- **URL**: https://apify.com/automation-lab/website-source-code-downloader.md
- **Developed by:** [Automation Lab](https://apify.com/automation-lab) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.38 / 1,000 file extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Source Code Downloader

Download source files from authorized public websites with explicit safety and volume limits.
This website source code downloader crawls same-site pages, discovers referenced assets,
stores the retrieved bytes, and exports a machine-readable file manifest.

It archives server-returned HTML, CSS, JavaScript, images, fonts, audio, and video.
Every stored file includes its original URL, final URL, archive path, source relationship,
HTTP metadata, byte size, SHA-256 digest, and key-value-store reference.

### What does Website Source Code Downloader do?

The Actor starts from one or more public HTTP(S) pages.
It follows page links on the same host up to your chosen depth and page count.
It then downloads the asset types you selected from HTML and CSS references.

The result has two connected parts:

1. A dataset manifest with one typed record per attempted resource.
2. The successfully retrieved source bytes in the run key-value store.

Use `archivePath` to reconstruct a host-and-path directory tree.
Use `sha256` to detect duplicate or changed bytes in later authorized runs.
Use `discoveredFrom` and `relation` to trace why a file belongs to the archive.

### Who is this Actor for?

- Developers preparing an authorized static-site migration.
- Agencies inventorying client-owned public website assets.
- Archivists creating bounded snapshots of openly available material.
- QA engineers checking which files a deployment exposes.
- Security and performance teams reviewing public front-end dependencies.
- Data engineers feeding source-file manifests into storage or change pipelines.

This Actor is not a cloning service for server behavior.
It retrieves only anonymously reachable responses returned to its HTTP client.

### Why use this website source code downloader?

A single “save page” request rarely captures the relationships around a website.
This Actor adds bounded crawling and a consistent record for each file.

Key differentiators include:

- Separate `maxPages`, `maxDepth`, and `maxAssets` controls.
- Per-file and cumulative byte ceilings.
- An elapsed run-time ceiling.
- Selectable HTML, CSS, JavaScript, image, font, and media types.
- CSS `url(...)` and `@import` discovery.
- Same-site page crawling with optional subdomains.
- Optional referenced CDN assets without following external page links.
- Public-network validation before requests and every redirect.
- File hashes and deterministic archive paths.
- Uncharged typed error rows when continuation is enabled.

### What files and metadata are extracted?

| Field | Meaning |
| --- | --- |
| `url` | Resource URL requested by the Actor. |
| `finalUrl` | Final URL after validated redirects. |
| `rootUrl` | Start URL that owns this crawl branch. |
| `resourceType` | `html`, `css`, `javascript`, `image`, `font`, or `media`. |
| `statusCode` | Successful HTTP response status. |
| `mimeType` | Validated response media type. |
| `byteSize` | Stored response size in bytes. |
| `sha256` | SHA-256 digest of the exact stored bytes. |
| `archivePath` | Host-and-path destination for reconstructing a file tree. |
| `storageKey` | File key in the run key-value store. |
| `storageUrl` | Direct URL for the stored bytes. |
| `discoveredFrom` | Page or stylesheet that referenced the resource. |
| `relation` | Discovery relationship such as `link`, `script`, or `css-url`. |
| `depth` | Same-site page-link depth from the start page. |
| `retrievedAt` | ISO 8601 retrieval timestamp. |
| `status` | `stored` or `error`. |
| `errorType` | Stable failure category for error rows. |
| `errorMessage` | Bounded diagnostic detail for error rows. |

Nullable fields are `null` when a retrieval fails before that value exists.
Error rows make partial archives auditable and do not incur a per-file event.

### How to download website source code

1. Open the Actor in Apify Console.
2. Add one to 20 authorized public start URLs.
3. Choose the maximum page count and link depth.
4. Choose the file types and asset count.
5. Set byte and time budgets appropriate for the site.
6. Leave external assets enabled if CDN-hosted files belong in the archive.
7. Run the Actor.
8. Export the dataset manifest or open each `storageUrl`.

Start with a small crawl such as one page, depth zero, and ten assets.
Increase bounds only after checking the site's structure and your authorization.

### Input parameters

| Input | Type | Default | Description |
| --- | --- | --- | --- |
| `startUrls` | URL array | required | One to 20 public HTTP(S) pages. |
| `maxPages` | integer | `10` | HTML pages attempted across all roots. |
| `maxDepth` | integer | `1` | Same-site link hops; `0` means supplied pages only. |
| `maxAssets` | integer | `100` | Non-page assets attempted after discovery. |
| `assetTypes` | string array | all six types | File classes to store. |
| `maxFileBytes` | integer | `5000000` | Maximum bytes accepted for one file. |
| `maxTotalBytes` | integer | `50000000` | Maximum cumulative stored bytes. |
| `maxRunSecs` | integer | `240` | Elapsed scheduling limit. |
| `requestTimeoutSecs` | integer | `30` | Per-request timeout. |
| `includeExternalAssets` | boolean | `true` | Include referenced CDN/third-party assets. |
| `includeSubdomains` | boolean | `false` | Follow page links onto subdomains. |
| `continueOnError` | boolean | `true` | Continue and emit error records. |
| `proxyConfiguration` | object | direct | Optional Apify or custom proxy configuration. |

Page discovery always stays on the start host unless subdomains are enabled.
External assets never cause external HTML pages to be crawled.

### Example input

```json
{
  "startUrls": [{ "url": "https://www.python.org/about/" }],
  "maxPages": 3,
  "maxDepth": 1,
  "maxAssets": 12,
  "assetTypes": ["html", "css", "javascript", "image", "font"],
  "maxFileBytes": 5000000,
  "maxTotalBytes": 15000000,
  "maxRunSecs": 120,
  "includeExternalAssets": true,
  "continueOnError": true
}
```

This exact workflow is included as a public Task example and is tested locally.

### Example output

```json
{
  "url": "https://www.python.org/about/",
  "finalUrl": "https://www.python.org/about/",
  "rootUrl": "https://www.python.org/about/",
  "resourceType": "html",
  "statusCode": 200,
  "mimeType": "text/html",
  "byteSize": 53912,
  "sha256": "1599d6b9d1d8f67a153f2d5804c5bde5e3c2b1854e2ed66e99036c7877acdc70",
  "archivePath": "www.python.org/about/index.html",
  "storageKey": "FILE_09b8a7c6d5e4f321_index.html",
  "storageUrl": "https://api.apify.com/v2/key-value-stores/STORE_ID/records/FILE_09b8a7c6d5e4f321_index.html",
  "discoveredFrom": null,
  "relation": "start-url",
  "depth": 0,
  "retrievedAt": "2026-01-15T12:00:00.000Z",
  "status": "stored",
  "errorType": null,
  "errorMessage": null
}
```

Values vary as websites deploy new content.
The schema and field types remain stable for integrations.

### How files are organized

The dataset is the archive manifest.
The key-value store contains bytes under keys beginning with `FILE_`.

`archivePath` combines the final hostname and normalized URL path.
Directory-like HTML URLs receive `index.html`.
A short query hash distinguishes URL variants that share a path.

The Actor does not currently produce a ZIP or rewrite links inside documents.
A downstream workflow can download the files and place each one at `archivePath`.

### How much does it cost to download website source files?

The Actor uses pay per event pricing.
There is one small start event and one `file` event for every successfully stored file.
Error rows are not charged as files.

At the BRONZE rate of $0.00064 per stored file plus a $0.001 start:

| Stored files | Example charge |
| ---: | ---: |
| 1 | $0.00164 |
| 10 | $0.00740 |
| 100 | $0.06500 |
| 1,000 | $0.64100 |

Your active Apify plan tier determines the exact file price shown in Console.
Proxy traffic and platform usage may be billed separately according to your Apify plan.
Final limits, redirects, duplicates, and failed resources affect the actual count.

### Limits and stopping behavior

The Actor stops scheduling pages at `maxPages` and assets at `maxAssets`.
It stops adding files before `maxTotalBytes` would be exceeded.
Responses over `maxFileBytes` become error rows.
No new retrieval starts after `maxRunSecs`.

Each request has one bounded retry for transient 408, 429, and selected 5xx responses.
Redirects are limited to ten and every destination is validated again.
Malformed inputs and private-network destinations fail closed.

`continueOnError: false` makes the run fail at the first resource error.
With the default `true`, independent resources continue and failures remain visible.

### Reliability tips

- Begin with one start URL and depth zero.
- Raise `maxPages` only after inspecting internal navigation.
- Disable media for faster, smaller source-code inventories.
- Keep external assets enabled for sites that use a CDN.
- Disable external assets for a strict same-host archive.
- Use a proxy only when the authorized target requires that route.
- Avoid very high limits on sites with calendar or faceted URL spaces.
- Compare hashes rather than timestamps when detecting byte-level changes.

A site can return different files by geography, headers, cookies, or deployment time.
This Actor records what its anonymous HTTP client received during the run.

### Website migration workflow

For an authorized migration:

1. Run a conservative inventory with HTML, CSS, JavaScript, images, and fonts.
2. Review errors and increase only the necessary byte bounds.
3. Export dataset rows as JSON or CSV.
4. Download each successful `storageUrl`.
5. Write the bytes to the corresponding `archivePath`.
6. Use `discoveredFrom` to investigate missing dependencies.
7. Re-run near cutover and compare `sha256` values.

This captures public front-end files.
It does not recover databases, server code, private APIs, forms, or authentication.

### Change-detection workflow

Schedule bounded runs with the same inputs.
Join records by final URL or archive path.
Compare `sha256`, `byteSize`, and `status` between runs.

A changed hash means the retrieved bytes changed.
It does not explain semantic meaning, so downstream review may still be needed.
The Actor itself does not send alerts or retain cross-run state.

### Integrations

Dataset rows work with Apify integrations and webhooks.
Common destinations include:

- Google Sheets for compact manifests.
- Cloud storage for long-term authorized archives.
- Webhooks that start a file-copy or hash-comparison job.
- Make or Zapier for low-code review workflows.
- Python or Node.js pipelines that reconstruct `archivePath` trees.

Large binary files remain in the key-value store rather than being embedded in rows.
This keeps dataset exports useful and avoids base64 inflation.

### Run with the Apify API using cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-source-code-downloader/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://www.python.org/about/"}],
    "maxPages": 1,
    "maxDepth": 0,
    "maxAssets": 10
  }'
```

Poll the returned run ID, then read its default dataset and key-value store.
Keep API tokens in a secret manager rather than source code.

### Run with JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-source-code-downloader').call({
  startUrls: [{ url: 'https://www.python.org/about/' }],
  maxPages: 1,
  maxDepth: 0,
  maxAssets: 10,
  assetTypes: ['html', 'css', 'javascript'],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

Use each successful row's `storageUrl` to retrieve its source bytes.

### Run with Python

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("automation-lab/website-source-code-downloader").call(run_input={
    "startUrls": [{"url": "https://www.w3.org/standards/"}],
    "maxPages": 3,
    "maxDepth": 1,
    "maxAssets": 10,
    "assetTypes": ["html", "css", "image"],
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["status"], item["archivePath"])
```

Store the token outside notebooks and repositories.

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-source-code-downloader"
```

#### Claude Desktop

Add this remote server to your Claude Desktop MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-source-code-downloader"
    }
  }
}
```

#### Cursor

In Cursor MCP settings, add the same `apify` server name and remote URL shown above.

#### VS Code

In VS Code MCP settings, register the same HTTP URL as the `apify` server.

Example prompts:

- “Archive one Python.org page and ten code assets, with no link crawling.”
- “Build a five-page source-file manifest for this authorized public site.”
- “Return only stored CSS and image records with their archive paths and hashes.”

Always include realistic limits in autonomous-agent prompts.

### Responsible and legal use

Use this Actor only for websites and files you are authorized to retrieve.
Follow applicable law, contracts, terms, copyright rules, and website policies.
Do not use it to bypass authentication, paywalls, access controls, or technical blocks.

The Actor rejects local and private-network destinations to reduce SSRF risk.
That safeguard does not determine whether you have permission to archive a public URL.
You are responsible for selecting targets, limits, retention, and downstream use.

### Troubleshooting

#### Why do I see `UNEXPECTED_CONTENT_TYPE`?

A URL advertised as one file type returned another media type.
This can happen with error pages, bot challenges, login redirects, or stale asset links.
Inspect the URL and choose a route the site permits; do not simply raise limits.

#### Why are some CDN files absent?

Confirm `includeExternalAssets` is true and that the file type is selected.
Assets created only by client-side JavaScript are outside this HTTP-only workflow.
The Actor archives server-returned HTML and statically referenced dependencies.

#### Why did crawling stop early?

Check `maxPages`, `maxAssets`, `maxTotalBytes`, and `maxRunSecs`.
The run log reports attempted pages, attempted assets, stored files, failures, and bytes.
Duplicate URL-and-type pairs are intentionally fetched only once.

#### Can it download authenticated pages?

No.
The supported contract is anonymously reachable public HTTP(S) content.
URLs containing credentials are rejected.

### Frequently asked questions

#### Does it create a working offline clone?

It creates source bytes and a reconstruction manifest.
It does not rewrite links, emulate a server, copy databases, or guarantee interactive behavior.

#### Does it render JavaScript?

No.
It downloads JavaScript files referenced by server-returned HTML but does not run a browser.
This keeps the workflow faster and more predictable for static source archives.

#### Are failed downloads charged?

No file event is charged for an error row.
The one-time start event still applies to a run that initializes successfully.

#### Can I crawl multiple sites in one run?

Yes, up to 20 start URLs.
Each start URL owns a separate same-site page branch, while limits apply across the run.

#### How long are files available?

They follow the retention rules of your Apify account and run storage.
Copy files to long-term storage when the archive must be retained.

### Related automation-lab Actors

- [Public Webpage HTML Downloader](https://apify.com/automation-lab/public-webpage-html-downloader) for supplied-page raw or browser-rendered HTML without site crawling.
- [Website Image Extractor](https://apify.com/automation-lab/website-image-extractor) for image URLs, srcset variants, alt text, dimensions, and page context.
- [Website HTML & Text Change Monitor](https://apify.com/automation-lab/website-html-text-change-monitor) for versioned page snapshots and text-level differences.
- [Broken Link Checker](https://apify.com/automation-lab/broken-link-checker) for validating site links rather than storing source bytes.

Choose the narrower Actor when you do not need a complete bounded file manifest.

# Actor input Schema

## `startUrls` (type: `array`):

One to 20 anonymously public HTTP(S) pages where each bounded crawl starts.

## `maxPages` (type: `integer`):

Maximum number of HTML pages attempted across all start URLs.

## `maxDepth` (type: `integer`):

Maximum number of same-site link hops from each start URL. Use 0 for only the supplied pages.

## `maxAssets` (type: `integer`):

Maximum number of CSS, JavaScript, image, font, and media files attempted after page discovery.

## `assetTypes` (type: `array`):

Types to store. HTML controls page files; the remaining choices control referenced assets.

## `maxFileBytes` (type: `integer`):

Reject any individual response larger than this many bytes.

## `maxTotalBytes` (type: `integer`):

Stop before stored files exceed this cumulative byte budget.

## `maxRunSecs` (type: `integer`):

Stop scheduling retrievals after this elapsed run time.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request, including one bounded transient retry.

## `includeExternalAssets` (type: `boolean`):

Download referenced CDN and third-party assets after public-network validation. Page links are still site-scoped.

## `includeSubdomains` (type: `boolean`):

Allow HTML page discovery on subdomains of each start URL host.

## `continueOnError` (type: `boolean`):

Emit uncharged error rows and continue with other resources. Disabled runs fail on the first retrieval error.

## `proxyConfiguration` (type: `object`):

Optional Apify or custom proxy routing. Direct requests are used when omitted.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.python.org/about/"
    }
  ],
  "maxPages": 10,
  "maxDepth": 1,
  "maxAssets": 100,
  "assetTypes": [
    "html",
    "css",
    "javascript",
    "image",
    "font",
    "media"
  ],
  "maxFileBytes": 5000000,
  "maxTotalBytes": 50000000,
  "maxRunSecs": 240,
  "requestTimeoutSecs": 30,
  "includeExternalAssets": true,
  "includeSubdomains": false,
  "continueOnError": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset rows for stored files and bounded retrieval errors.

## `files` (type: `string`):

Key-value store containing retrieved source files and assets.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.python.org/about/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/website-source-code-downloader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://www.python.org/about/" }] }

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/website-source-code-downloader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.python.org/about/"
    }
  ]
}' |
apify call automation-lab/website-source-code-downloader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/website-source-code-downloader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wgYvJBEMn6rU5EHEy/builds/sZr0aoAQ2ahNpzsq9/openapi.json