# Bulk File Downloader - URL List, Images, Video & Audio (`rationalistic_candle_ucn/bulk-file-downloader`) Actor

Download a list of file URLs in bulk, or save every image, video and audio file found on a list of pages, straight into the run's key-value store. Each file is stored the moment it lands, with a dataset row of its metadata. Priced per file and per 10 MB moved, zipped on request, resumable.

- **URL**: https://apify.com/rationalistic\_candle\_ucn/bulk-file-downloader.md
- **Developed by:** [Bruce McGinley](https://apify.com/rationalistic_candle_ucn) (community)
- **Categories:** Automation, Developer tools, Integrations
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 file downloadeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Bulk File Downloader

Paste a list of file URLs, or a list of pages, and get the files back in the run's key-value store with one metadata row per file in the dataset. Each file is stored the moment it lands, so a run that stops part-way has delivered everything it fetched, and a second run can pick up where the first one stopped. Files can also be zipped, in parts of at most 500 MB unless one file is bigger than that on its own, so a big job never has to fit in memory to be delivered.

You are responsible for having the right to download and store what you point this Actor at. It fetches over plain HTTP with its own named user agent, honours every site's `robots.txt`, and refuses a fixed list of hosts outright.

### How it works

1. Every URL is checked before anything is requested. Hosts on the refused list, and URLs that are not plain `http` or `https` addresses, get a dataset row saying so and are never fetched.
2. For each remaining host the Actor reads `robots.txt` once. A URL the file disallows, for this Actor's user agent or for `*`, is skipped with a row and is not charged. There is no setting that turns this off.
3. Each file is fetched with one request started at a time per host and at most two requests a second per host, ten downloads across all hosts, with three attempts on a `429` or a server error. Redirects are followed one hop at a time and every hop gets the same checks.
4. The response is streamed to disk while it is hashed and counted. When it is complete it is written to the key-value store as `files-<hash>-<name>`, the two events are charged, and the dataset row is pushed with the key, the store URL, the content type, the size and the SHA-256.
5. With **Crawl pages** on, a URL that returns an HTML page is read for `img`, `source`, `video` and `audio` sources (including `srcset`) and links to files on the same host, and those are downloaded. Pages found on pages are not crawled.
6. With **Zip** set, the stored files are packed into zip parts named `zip-part-001.zip` and so on after the downloads, and `zip-manifest.json` lists the parts.

The run stops cleanly at **Max files**, **Max total bytes** or the run's maximum total charge, and the status message says how many URLs were left.

### Pricing

This Actor uses pay-per-event pricing with two events, charged only for a file that was actually stored:

- **file-downloaded**, once per file stored: $0.002.
- **transfer-10mb**, once per 10 MB of the file rounded up, minimum one: $0.001.

So a 200 KB image is 1 × $0.002 + 1 × $0.001 = $0.003, a 5 MB PDF the same, and a 50 MB video is 1 × $0.002 + 5 × $0.001 = $0.007, at the prices shown on this page; the Actor page's own pricing table is the one that bills. A refused URL in your list, a skip (robots, size, wrong type) or a failed request costs nothing and still gets a dataset row explaining why; a refused or unusable link found on a crawled page is dropped without a row, and the page's own row says how many links were queued. The maximum total charge you set on a run is honoured: the Actor checks the remaining allowance before each fetch and stops cleanly when the next file would not fit. Zipping is not charged. Platform compute and storage are billed by the platform as for any Actor.

### Tutorial

1. Open the Actor and paste URLs into **URLs**, one per line. File URLs work as they are; page URLs need **Crawl pages** switched on.
2. Choose the **File types** you want. The default is images only; `any` keeps every kind of file.
3. Set **Zip** if you want the files packed for download in one go. Leave it off to fetch files one by one from the store.
4. Click **Start**. Open **Storage**, then **Key-value store**, to browse the files (their keys start with `files-`), or the **Dataset** for one row per URL with its store URL. The dataset exports as JSON, CSV or Excel.
5. If a run stops early, start a new run with the same input and the first run's ID in **Resume from run**. Files already in the first run's store are skipped and not charged. Two things to know about a resumed run: its **Max files** and **Max total bytes** are its own, not a running total across both runs, and its zip parts hold only the files it fetched itself, so take the first run's zips as well. Resuming needs **Unique file names** on, because that is what makes a file's key depend on its URL.

### Input example

```json
{
  "urls": [
    "https://example.com/gallery/",
    "https://example.com/files/report-2026.pdf"
  ],
  "crawlPages": true,
  "fileTypes": ["image", "document"],
  "maxFiles": 500,
  "zip": "single"
}
```

### Output example

One dataset item per URL processed:

```json
{
  "url": "https://example.com/gallery/photo-1.jpg",
  "sourcePage": "https://example.com/gallery/",
  "status": "downloaded",
  "reason": null,
  "key": "files-3f9a1c2b-photo-1.jpg",
  "storeUrl": "https://api.apify.com/v2/key-value-stores/abc123/records/files-3f9a1c2b-photo-1.jpg",
  "contentType": "image/jpeg",
  "bytes": 48213,
  "sha256": "9b74c9897bac770ffc029102a200c5de3f2e5a4b2f9a9a3f5ad1c9e2a4c0f6d1",
  "charged": { "file": 1, "transfer10mb": 1 },
  "fetchedAt": "2026-09-08T10:15:42.000Z"
}
```

`status` is one of `downloaded`, `skipped`, `refused` or `failed`; `reason` says why for anything but a download, for example `robots.txt disallows`, `host refused: example.net`, `over maxFileBytes` or `HTTP 404`.

### Limits and rules

- **Refused hosts.** Major social and media platforms (YouTube, TikTok, Instagram, Facebook, X, Reddit, Pinterest, Vimeo, SoundCloud, Spotify, Twitch and their short-link and CDN hosts) and sites whose published terms forbid automated access are refused at input, before any request. The list ships with the Actor as a generated file and is not configurable.
- **robots.txt** is always honoured. If a site's `robots.txt` cannot be read because the server errors, its URLs are skipped rather than guessed at.
- **No proxies, headers or cookies.** The Actor fetches from its own address with a fixed, descriptive user agent. It cannot log in, cannot pass a bot challenge, and does not try.
- **Sizes.** A file over **Max file size** is skipped. The run stops at **Max total bytes**. Zip parts are capped at 500 MB each, except that a single file larger than that gets a part of its own rather than being dropped; the run log says so when it happens. A file over 32 MB is uploaded to the store as a stream, which the platform client cannot retry; if that upload fails the file gets a failed row and is not charged.
- **Each URL once.** A URL listed twice, or found on two pages, is fetched and charged once. Where a redirect lands counts too, so two spellings of one file are not stored and charged twice.
- **Unknown sizes.** When a server sends no Content-Length the size is known only once the file has landed, so two such files in flight can together overshoot **Max total bytes** by at most one **Max file size**.
- **Names.** A stored key is `files-<8-character hash of the URL>-<file name>`, with the name reduced to letters, digits, dots, dashes and underscores. The hash means the same URL always lands on the same key, which is what makes resuming work.

### Use cases

- **Archiving a gallery or a media library** from a site you run or have permission to copy, straight into storage, with a checksum for every file.
- **Collecting documents** from a list of report or dataset URLs into one zip for the people who need them.
- **Feeding another Actor or workflow** that expects files in a key-value store, with the dataset row as the index.
- **Agents** that need to fetch a set of files as one step: the input is a list of URLs, the output is plain JSON, and the price of a job can be worked out from the sizes before it runs.

### FAQ

**Does it need any login or API key?** No. It fetches public URLs only, with no credentials of any kind.

**Can it download from YouTube, Instagram or other platforms?** No. Those hosts are refused before any request is made, and the dataset row says so. There is no setting to change that.

**What happens to a file that is too big?** It is skipped with a row that says `over maxFileBytes`, and nothing is charged for it.

**Is anything charged for a file that failed?** No. Only a file that was stored is charged, and every stored file has a row with its key.

**How do I get the files out?** Each dataset row carries the store URL of its file. With **Zip** on, the parts are the `zip-part-` records in the key-value store and `zip-manifest.json` lists them. A resumed run's parts hold that run's files only, so collect the earlier run's parts too. If zipping itself fails, the run still finishes: every file is in the store and in the dataset, and the manifest carries the error.

**Something looks wrong.** Open an issue on the Actor's Issues tab with the URL and the dataset row in question.

# Actor input Schema

## `urls` (type: `array`):

File URLs to download, one per line, or page URLs when Crawl pages is on. You are responsible for having the right to download and store what you point this Actor at. URLs on refused hosts (major social and media platforms, and sites whose terms forbid automated access) are rejected here, before any request is made, and get a dataset row saying so.

## `crawlPages` (type: `boolean`):

When a URL returns an HTML page, collect the files it shows (img and source src and srcset, video and audio src, and links to files) on the same host, and download those. Pages found on pages are not crawled.

## `fileTypes` (type: `array`):

Which kinds of file to keep, matched on Content-Type first and the URL's extension second: image, video, audio, document, archive, or any.

## `maxFiles` (type: `integer`):

Stop after this many files have been stored. The run ends cleanly and says how many URLs were left.

## `maxFileBytes` (type: `integer`):

A file larger than this is skipped with a dataset row and is not charged. Default 104857600 (100 MB).

## `maxTotalBytes` (type: `integer`):

Stop when the files stored add up to this. Default 2147483648 (2 GB). The run ends cleanly and says how many URLs were left.

## `zip` (type: `string`):

After downloading, also store the files as zip archives: none, one zip for the whole run (single), or one per host (perHost). Zips are split into parts of at most 500 MB, stored as zip-part-001.zip and so on.

## `uniqueFileNames` (type: `boolean`):

Store each file as files-<short hash of the URL>-<original name>, so two files with the same name never collide. Off, the name stands alone and a second file of the same name gets a numeric suffix.

## `resumeFromRunId` (type: `string`):

The ID of an earlier run of this Actor. Files already present in that run's key-value store are skipped (and not charged), so a run that stopped part-way can be picked up without paying twice.

## Actor input object example

```json
{
  "urls": [
    "https://www.w3.org/Icons/w3c_home.png",
    "https://www.w3.org/Icons/w3c_main.png"
  ],
  "crawlPages": false,
  "fileTypes": [
    "image"
  ],
  "maxFiles": 1000,
  "maxFileBytes": 104857600,
  "maxTotalBytes": 2147483648,
  "zip": "none",
  "uniqueFileNames": true,
  "resumeFromRunId": ""
}
```

# Actor output Schema

## `files` (type: `string`):

The run's key-value store: one record per file with a files- key, zip parts with zip- keys.

## `rows` (type: `string`):

One item per URL processed: status, reason, store key and URL, content type, size, SHA-256 and what was charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.w3.org/Icons/w3c_home.png",
        "https://www.w3.org/Icons/w3c_main.png"
    ],
    "fileTypes": [
        "image"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("rationalistic_candle_ucn/bulk-file-downloader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://www.w3.org/Icons/w3c_home.png",
        "https://www.w3.org/Icons/w3c_main.png",
    ],
    "fileTypes": ["image"],
}

# Run the Actor and wait for it to finish
run = client.actor("rationalistic_candle_ucn/bulk-file-downloader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.w3.org/Icons/w3c_home.png",
    "https://www.w3.org/Icons/w3c_main.png"
  ],
  "fileTypes": [
    "image"
  ]
}' |
apify call rationalistic_candle_ucn/bulk-file-downloader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rationalistic_candle_ucn/bulk-file-downloader"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IiyA2sRou7K3NaDfl/builds/xngZ1ajHWizlt2rDB/openapi.json