# Bulk File & Image Downloader (`brimfold/bulk-file-downloader`) Actor

Download a list of file or image URLs straight into Apify storage, streamed in chunks so memory stays flat even for multi-GB files. Hard caps on file size and count, robots.txt respected, private and internal addresses blocked.

- **URL**: https://apify.com/brimfold/bulk-file-downloader.md
- **Developed by:** [Brimfold Studio](https://apify.com/brimfold) (community)
- **Categories:** Automation, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.83 / 1,000 file storeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Bulk File & Image Downloader

Give it a list of file or image links. It downloads each file **straight into the run's Apify key-value store** and returns one dataset row per file with a direct link to the stored copy, its size, type and SHA-256 checksum.

Built for big lists and big files:

- **Streaming in 256 KB chunks.** A file goes from the source to storage without ever being held in memory, so memory use stays flat. Measured: a 115 MB (120,357,629-byte) video stored in a 128 MB run on Apify with a platform peak of 65 MB of memory (79 MB by the Actor's own process count); locally the peak was 73 MB. Files of several GB work the same way, within the size cap you set.
- **Hard caps** on file count (`maxFiles`, default 100) and file size (`maxFileSizeMb`, default 100 MB, up to 5,000). A file over the size cap is skipped before download when the server states its size, or stopped and deleted the moment it passes the cap. Skipped files are never charged.
- **Polite fetching:** follows each site's `robots.txt`, at most 2 parallel downloads per host with a pause between requests, and a longer pause when `robots.txt` sets a `Crawl-delay` (up to 30 s).
- **Safe by default:** links to private, internal or cloud-metadata addresses are refused (see Security).

### Your responsibility: rights to the content

**Only download files you have the right to download, copy and store.** That means content you own, content under a licence that allows it (for example public-domain or Creative Commons files, used under the licence terms), or content you have permission for. This Actor is a download tool: it does not check copyright, it does not bypass logins, paywalls, captchas or DRM, and it must not be used to infringe copyright or to collect other people's private data. Respect each site's terms of service. Stored files stay in your own Apify account, and you are responsible for how you use them.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` | 3 public-domain NASA images | Direct links to files (http/https) |
| `maxFiles` | 100 | Hard cap on files stored in the run |
| `maxFileSizeMb` | 100 | Hard cap per file, in MB (1 MB = 1,048,576 bytes) |
| `allowedContentTypes` | any | Keep only files whose Content-Type starts with one of these, e.g. `image/`, `application/pdf` |
| `allowHtmlPages` | false | HTML web pages are skipped unless you allow them (a link to a page is usually a viewer, not the file) |
| `maxConcurrency` | 4 | Parallel downloads (max 2 per host) |
| `requestTimeoutSecs` | 600 | Per-file time limit |
| `keyPrefix` | none | Text added to the start of every storage key |

```json
{
  "urls": [
    "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/1/1b/Federal_Register-Vol74No33.pdf"
  ],
  "maxFiles": 500,
  "maxFileSizeMb": 250,
  "allowedContentTypes": ["image/", "application/pdf"]
}
```

### Output

One dataset row per stored file:

```json
{
  "url": "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
  "finalUrl": "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
  "key": "00001-Apollo_11_1998_scan.jpg",
  "recordUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/00001-Apollo_11_1998_scan.jpg",
  "fileName": "Apollo_11_1998_scan.jpg",
  "contentType": "image/jpeg",
  "sizeBytes": 154428,
  "sizeMb": 0.147,
  "extra10MbUnits": 0,
  "sha256": "…",
  "httpStatus": 200,
  "lastModified": "…",
  "etag": "…",
  "redirects": 0,
  "downloadSecs": 0.41,
  "downloadedAt": "2026-11-02T19:40:00Z"
}
```

The files themselves are in the run's default key-value store under `key`. `recordUrl` needs your API token unless you share the store. The `RUN-SUMMARY` record lists every skipped or failed link with its reason (`duplicate`, `unsupported-url`, `blocked-private-address`, `blocked-by-robots-txt`, `http-error`, `content-type-not-allowed`, `html-page-not-file`, `over-size-cap`, `over-max-files`, `over-run-budget`, `timeout`, `download-failed`), plus bytes stored and peak memory.

Storage note: files in a run's default key-value store follow your Apify plan's data retention (unnamed storages are removed after the retention period), and stored data counts toward storage usage on your Apify plan. Copy anything you need to keep.

### Security (SSRF protection)

- Only `http` and `https` links.
- The Actor resolves every host itself and refuses the link if **any** address is private, loopback, link-local (including the `169.254.169.254` cloud-metadata address), multicast, reserved or otherwise not public. `localhost`, `*.local` and `*.internal` names are refused.
- The connection is pinned to the checked address, so a DNS answer cannot change between the check and the download.
- Redirects are followed by the Actor (at most 5), and every hop is checked again, robots.txt included.
- Connections are never reused across hostnames.

### Robots, terms and rate limits

- `robots.txt` is read for every site before downloading (user agent token `BrimfoldStudio-FileDownloader`), following RFC 9309. If it can't be fetched because of a server error, timeout or HTTP 429, the site is treated as off limits. If it doesn't exist (other 4xx), downloads are allowed.
- At most 2 downloads at a time per host, 0.25 s apart, or the site's `Crawl-delay` (capped at 30 s). Rate-limit replies (429/503) are retried up to 3 times, honouring `Retry-After` up to 60 s.
- The Actor sends an honest user agent, never logs in and never solves captchas. Respect each site's terms of service. Some hosts (for example Wikimedia) publish their own policies for automated downloads.
- Sources: files come from the URLs you provide. The default input uses public-domain NASA images hosted on Wikimedia Commons.

### Pricing

Pay per event, nothing for skipped or failed files:

- **$0.0025 per file stored** (`file-downloaded`, one per dataset row)
- **$0.001 per extra 10 MB** (`extra-10mb`): each started 10 MB block beyond a file's first 10 MB. 10 MB here means 10,485,760 bytes. Examples: a 4 MB image costs $0.0025; a 25 MB file costs $0.0025 + 2 × $0.001 = $0.0045; a 115 MB video costs $0.0025 + 11 × $0.001 = $0.0135.
- Plus Apify's small actor-start fee.

If you set a maximum charge for the run, the Actor reserves the worst case for each file before downloading it (using the stated size, or the size cap when the size is unknown) and skips files that would not fit, so the cap is never passed. Discounts by Apify plan are listed in `PRICING.md`.

### FAQ

**Does it crawl pages to find files?** No. Give it direct file links. To collect links first, use a crawler, then pass the list here.

**Can it download from sites behind a login?** No, only public links.

**Why was my link "blocked-by-robots-txt"?** The site's `robots.txt` asks automated clients not to fetch that path. The Actor follows it, with no override.

***

Made by Brimfold Studio. Default sample images: NASA, public domain, via Wikimedia Commons.

# Actor input Schema

## `urls` (type: `array`):

Direct links to the files to download (http or https). Only download files you have the right to download and store.

## `maxFiles` (type: `integer`):

Hard cap on how many files are downloaded in this run. Extra URLs are listed as skipped and not charged.

## `maxFileSizeMb` (type: `integer`):

Hard cap per file, in MB (1 MB = 1,048,576 bytes). Bigger files are skipped before download when the server states the size, or stopped and discarded as soon as they pass the cap. Skipped files are not charged.

## `allowedContentTypes` (type: `array`):

Optional. Only keep files whose Content-Type starts with one of these, for example "image/" or "application/pdf". Empty = any type except HTML pages.

## `allowHtmlPages` (type: `boolean`):

By default links that return an HTML web page are skipped (uncharged), because they are usually a viewer page rather than the file itself.

## `maxConcurrency` (type: `integer`):

Files downloaded at once. At most 2 at a time go to the same host, with a pause between requests.

## `requestTimeoutSecs` (type: `integer`):

A file that takes longer than this to download is abandoned and not charged.

## `keyPrefix` (type: `string`):

Optional text put at the start of every stored file key, to tell runs apart. Letters, numbers and - \_ . only.

## Actor input object example

```json
{
  "urls": [
    "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/1/19/Apollo_11_2004_scan_cropped.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/0/0c/Apollo_11_Commemorative_Coin.jpg"
  ],
  "maxFiles": 100,
  "maxFileSizeMb": 100,
  "allowedContentTypes": [],
  "allowHtmlPages": false,
  "maxConcurrency": 4,
  "requestTimeoutSecs": 600,
  "keyPrefix": ""
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
        "https://upload.wikimedia.org/wikipedia/commons/1/19/Apollo_11_2004_scan_cropped.jpg",
        "https://upload.wikimedia.org/wikipedia/commons/0/0c/Apollo_11_Commemorative_Coin.jpg"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("brimfold/bulk-file-downloader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
        "https://upload.wikimedia.org/wikipedia/commons/1/19/Apollo_11_2004_scan_cropped.jpg",
        "https://upload.wikimedia.org/wikipedia/commons/0/0c/Apollo_11_Commemorative_Coin.jpg",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("brimfold/bulk-file-downloader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/1/19/Apollo_11_2004_scan_cropped.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/0/0c/Apollo_11_Commemorative_Coin.jpg"
  ]
}' |
apify call brimfold/bulk-file-downloader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,brimfold/bulk-file-downloader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jUVHqjFDMfjDSeCkF/builds/y2rcerF5JosNm9ATE/openapi.json