# Bulk File Downloader (direct URLs, robots.txt-respecting) (`canopy_ne3/bulk-file-downloader`) Actor

Downloads the files you list (PDFs, images, CSVs, archives, anything with a direct URL) into the run's key-value store, with a download link, size, content type and SHA-256 for each. Filters by content type, caps file size, obeys robots.txt and spaces requests per site. No proxy, no login.

- **URL**: https://apify.com/canopy_ne3/bulk-file-downloader.md
- **Developed by:** [Brenden Bushman](https://apify.com/canopy_ne3) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 file downloadeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Bulk File Downloader (direct URLs, robots.txt-respecting)

Download a list of files into Apify storage in one run. Give it direct links to PDFs, images, CSVs, spreadsheets,
archives or anything else with a URL. For each one you get back a download link, the record key, the file name, its
size, content type and SHA-256 checksum. Each file sits in the run's key-value store, ready for your next Actor, your
integration or a plain download.

### What you get

- **Every file stored** as a record in the run's key-value store, keyed by its position and file name
  (`0001-report.pdf`), with its original content type.
- **A dataset item per URL**, in input order: `downloadUrl`, `key`, `fileName` (from `Content-Disposition` or the
  URL), `contentType`, `sizeBytes`, `sha256`, or the reason it failed.
- **A content-type filter** (`allowedContentTypes`, for example `image/` or `application/pdf`). A file of another
  type is skipped before its body is transferred.
- **A size cap** (`maxFileSizeMb`). A larger file is refused from its `Content-Length` before the download starts,
  or stopped as soon as it passes the cap.
- Duplicate URLs are downloaded once.

Failed files are reported in the dataset with a reason and are **not charged**: `fetch-failed`, `file-too-large`,
`content-type-not-allowed`, `robots-disallowed`.

### Respectful by design

- **Only the URLs you list are downloaded**, one request each. Pages are not crawled and links are not followed.
  The site's robots.txt is read first and obeyed, and downloads from one site are spaced (500 ms by default).
  No proxy, no browser, no login.
- Only public internet addresses are fetched. A URL or redirect into a private, loopback or link-local network is
  refused.

Download only files you have the right to download and store.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `fileUrls` | array | - | Direct links to files. Required. |
| `allowedContentTypes` | array | - | Full types or prefixes ending in `/`; empty allows all. |
| `maxFileSizeMb` | integer | `50` | Up to 200. |
| `maxFiles` | integer | `1000` | Up to 10,000, in input order. |
| `requestDelayMs` | integer | `500` | Pause between downloads from one site. |

### Output

One dataset item per URL, in input order:

```json
{"url": "https://example.com/files/q3-report.pdf", "finalUrl": "https://example.com/files/q3-report.pdf",
 "status": "ok", "key": "0001-q3-report.pdf", "fileName": "q3-report.pdf",
 "downloadUrl": "https://api.apify.com/v2/key-value-stores/<store id>/records/0001-q3-report.pdf",
 "contentType": "application/pdf", "sizeBytes": 482113,
 "sha256": "9f2c...e41a", "error": null}
```

A failed file:

```json
{"url": "https://example.com/files/video.mp4", "status": "error", "key": null,
 "error": {"code": "file-too-large", "message": "104857600 bytes declared (maxFileSizeMb is 50)"}}
```

Key-value store record `SUMMARY`: files requested, downloaded and failed, bytes downloaded, megabytes charged, and why
the run stopped early if it did.

Files in a run's default storage are kept for as long as your Apify plan keeps unnamed storage (7 days on the free
plan). Copy anything you need to keep longer.

### Pricing

Pay per event, no start fee, and nothing for files that fail:

| Event | Price | Per 1,000 |
|---|---|---|
| `file-downloaded` (one file stored) | $0.002 | $2.00 |
| `megabyte-downloaded` (each started megabyte of it, at least one) | $0.001 | $1.00 |

A 300 KB image costs $0.003; a 12.5 MB PDF costs $0.015. When you set a maximum charge for a run, the Actor caps each
download at what the remainder pays for, and stops when not even a one-megabyte file fits.

### Limits

- Files only, one URL each. It does not crawl a page for its images or follow links.
- A run that waits too long on slow servers stops early with what it has downloaded, so its results are saved before
  the run's timeout.
- 200 MB per file at most.

# Actor input Schema

## `fileUrls` (type: `array`):

Direct links to the files to download, which you are allowed to download. Each URL is fetched once (duplicates are dropped); the site's robots.txt is respected. Pages are not crawled and links inside files are not followed.

## `allowedContentTypes` (type: `array`):

Download only files whose Content-Type matches one of these: a full type (application/pdf) or a prefix ending in / (image/). Others are skipped before their body is transferred, and not charged. Empty allows every type.

## `maxFileSizeMb` (type: `integer`):

Larger files are skipped, and not charged: refused from their Content-Length when the server sends one, otherwise as soon as the count passes this size.

## `maxFiles` (type: `integer`):

Download at most this many of the URLs, in the order given.

## `requestDelayMs` (type: `integer`):

Downloads from the same site are spaced by at least this long, to be gentle on small servers.

## Actor input object example

```json
{
  "fileUrls": [
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/field-report.pdf",
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/scanned-notice.png",
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv"
  ],
  "maxFileSizeMb": 50,
  "maxFiles": 1000,
  "requestDelayMs": 500
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `files` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "fileUrls": [
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/field-report.pdf",
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/scanned-notice.png",
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("canopy_ne3/bulk-file-downloader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "fileUrls": [
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/field-report.pdf",
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/scanned-notice.png",
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("canopy_ne3/bulk-file-downloader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "fileUrls": [
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/field-report.pdf",
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/scanned-notice.png",
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv"
  ]
}' |
apify call canopy_ne3/bulk-file-downloader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,canopy_ne3/bulk-file-downloader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/W6sPz3goutyRVP5gz/builds/ucsgu183NhNmbIlcV/openapi.json