# Parallel Bulk File Downloader – Fast URL Downloads (`automa-flow/parallel-bulk-file-downloader`) Actor

Download hundreds of public HTTP/HTTPS files in parallel from a URL list, stream them to Apify storage, and get one manifest row per file with checksums, status, size and download URL.

- **URL**: https://apify.com/automa-flow/parallel-bulk-file-downloader.md
- **Developed by:** [Vadim Bezrukov](https://apify.com/automa-flow) (community)
- **Categories:** Automation, Developer tools, Other
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 file downloadeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Parallel Bulk File Downloader – Fast URL Downloads

Download hundreds of public HTTP/HTTPS files in parallel from a URL list, stream
them directly to Apify storage, and get one manifest row per file with
checksums, status, size, and download URL.

Files are saved in the run's Key-Value Store. Open the Dataset to see which
downloads succeeded and follow each `storage_url` to retrieve the file.

Up to five downloads run at once by default, with a separate per-host limit.
Total time depends on the file sizes and the servers hosting them.

### Quick start

```json
{
  "urls": [
    {
      "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
      "fileName": "dummy.pdf"
    },
    {
      "url": "https://www.rfc-editor.org/rfc/rfc791.txt",
      "fileName": "rfc791.txt"
    }
  ],
  "concurrency": 5,
  "maxFileSizeMb": 100,
  "timeoutSeconds": 120,
  "maxRetries": 2,
  "overwrite": false
}
```

Default Console / API prefill uses the same bounded public demo URLs.

### Example workflows

Choose a batch you already have permission to download:

1. **Public PDF documents:** replace the first demo URL with a document URL and
   set `fileName` to a recognizable name. Inspect its `SUCCESS` row before
   expanding the batch.
2. **Mixed files:** use the PDF and text URLs in the quick start above. The
   Dataset manifest keeps each input's status, checksum, size and storage URL,
   including explicit failures.
3. **Recurring document batches:** pass the next list of document URLs from
   your workflow and use `sha256` downstream to compare file contents. Each run
   stores its own files; the Actor does not maintain a change-monitor baseline.

#### Download the files from your result

Open the run's Dataset and select a row with `status == "SUCCESS"`. Its
`storage_url` points to that file in the run's Key-Value Store; open it using
your authorized Apify access, or pass it to the next step of your automation.
Keep the manifest if you need to match downloaded files back to their input
URLs. Failed rows do not represent downloadable files. Files remain subject
to your Apify storage retention settings; they are not bundled into a ZIP or
uploaded to external storage.

### Related tools for finding or monitoring files

- **Get attachment URLs from a paginated API:** [Paginated REST API Fetcher](https://apify.com/automa-flow/paginated-rest-api-fetcher) exports API items across a configured pagination pattern. Select permitted public file URLs from the item data and map them into this Actor's `urls` input.
- **Monitor a PDF across revisions:** [PDF Change Monitor](https://apify.com/automa-flow/pdf-link-change-monitor) keeps a last-good baseline for a direct PDF or a PDF selected from a stable webpage. Choose it when you need content and link change events, rather than a fresh download manifest.
- **Read HTML as Markdown:** [Website Content Crawler](https://apify.com/automa-flow/website-content-crawler) extracts authorized website text and optional chunks for AI and RAG. Give it the original HTML page URL and rights declaration; it does not parse the downloaded binary files.

Each linked tool has separate input and pricing. Select the URLs and start the
appropriate run yourself or through your own automation.

### Output example

Successful row:

```json
{
  "source": "parallel-bulk-file-downloader",
  "source_id": "…",
  "status": "SUCCESS",
  "source_url": "https://example.com/files/report.pdf",
  "final_url": "https://cdn.example.com/report.pdf",
  "source_index": 0,
  "file_name": "report.pdf",
  "content_type": "application/pdf",
  "bytes": 4123456,
  "sha256": "…",
  "kvs_key": "files-000000-report.pdf",
  "storage_url": "https://api.apify.com/v2/key-value-stores/…/records/files-000000-report.pdf",
  "http_status": 200,
  "attempts": 1,
  "duration_ms": 842,
  "error_code": null,
  "error_message": null,
  "downloaded_at": "2026-09-17T21:00:00Z",
  "scraped_at": "2026-09-17T21:00:00Z",
  "fingerprint": "…",
  "schema_version": 2
}
```

Dataset views: **All**, **Downloaded**, **Failed**.

### Automation / API use

```bash
curl -X POST "https://api.apify.com/v2/acts/automa-flow~parallel-bulk-file-downloader/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":[{"url":"https://www.rfc-editor.org/rfc/rfc791.txt","fileName":"rfc791.txt"}],"concurrency":5}'
```

Schedule a recurring URL batch or call the Actor from n8n / Make / your backend.
A second run is useful whenever you have a new URL list, a refreshed CDN export,
or need checksums again for integrity checks.

### AI agents / MCP

Direct tool URL (authenticated; each caller uses their own Apify OAuth or token):

https://mcp.apify.com?tools=automa-flow/parallel-bulk-file-downloader

Anonymous MCP discovery (`search-actors`) indexes this public Store Actor.
The direct tool URL above is the authenticated path for `initialize`,
`tools/list` and a paid run. From 2026-10-06 20:23 UTC, direct agentic payment
is not available, because platform usage is charged in addition to events.

**When an agent should call it:** the user already has a list of public
HTTP/HTTPS file URLs and needs files in Apify storage plus a per-file manifest
(status, `storage_url`, SHA-256) for a downstream step (CRM attachment sync,
integrity re-check, report pack).

**When not to call it:** crawl/HTML extraction, authenticated Drive/Dropbox,
video-site downloaders, ZIP packing, or any private/localhost URL (SSRF policy
rejects those per item).

**Event charges are bounded. From 2026-10-06 20:23 UTC, platform usage is extra and depends on file size:**

| Example | Event charges | Measured platform usage |
| --- | ---: | ---: |
| Two small files | $0.00805 (`$0.00005` start + 2 × `$0.004`) | about $0.001 |
| 100 unique successful files | $0.40005 | depends on total bytes |
| Schema max 500 unique successes | $2.00005 | depends on total bytes |
| One 100 MiB file | $0.00405 | about $0.020 |

Until 2026-10-06 20:23 UTC the Store does not charge platform usage on top of
these events. After that time Apify meters compute, transfer and storage for
the user. The usage figures are measurements from 1 GiB runs on 2026-09-18,
not a quote.

Always set `maxTotalChargeUsd` on the run (Store minimum `$0.00405`). That cap
limits event charges. At the cap, already stored successful files remain;
remaining URLs get `BUDGET_EXCEEDED` / cancelled rows and are not charged.
From the date above, platform usage is billed separately from that cap.

Example agent ask: “Download these public PDF URLs in parallel with
`maxTotalChargeUsd=0.05`, then read `RUN_SUMMARY` and return Dataset rows with
`storage_url` and `sha256` for SUCCESS items only.”

After the run, read Key-Value Store `RUN_SUMMARY` first (`status`,
`successful` / `failed`, `chargedEventCounts`, `nextAction`), then the default
Dataset. Output schema also exposes `runSummary` and `billingReceipt` links.

### Failure semantics

| `error_code` | Meaning |
| --- | --- |
| `SECURITY_REJECTED` | URL/host/IP failed SSRF policy (localhost, private, metadata, bad scheme). |
| `HTTP_404` / `HTTP_403` / … | Remote server returned that status. |
| `RATE_LIMITED` | HTTP 429 after retry budget. |
| `TIMEOUT` | Per-attempt timeout exhausted. |
| `FILE_TOO_LARGE` | Content-Length or streamed bytes exceeded `maxFileSizeMb`. |
| `STORAGE_FAILED` | Key-Value Store write failed after a successful download. |
| `BUDGET_EXCEEDED` | Spending limit stopped storing/charging further successful files. |
| `BILLING_UNCERTAIN` | Charge API failed after a store write; the run fails closed (no replay). |

`NO_RESULTS` is not used: an empty `urls` array is invalid input and never
starts. An ordinary item failure never cancels independent downloads. A
`BILLING_UNCERTAIN` outcome stops the batch, cancels and drains sibling workers,
deletes files known to be uncharged, and retains the uncertain file plus its
Dataset row for reconciliation.

### Pricing

Pay per event, plus Apify platform usage:

| Event | Price |
| --- | --- |
| `apify-actor-start` | $0.00005 / run (platform synthetic; memory may multiply) |
| `file-downloaded` | $0.004 / successfully stored unique file |

Not charged as events: failed URLs, retries, redirects, security rejections, or
bytes that never reach a successful store write.

**Platform usage is passed through to the user** from 2026-10-06 20:23 UTC
(compute units, external transfer, Key-Value Store storage). Until then the
Store does not add that charge. It is not included in the `$0.004` file price.
The two-file demo measured about $0.001 of usage; a 100 MiB file measured
about $0.02 of usage on top of the $0.004 event.

### Security and SSRF protection

- Only `http` and `https` on ports 80/443.
- Localhost, loopback, RFC1918, link-local, multicast, reserved, IPv6 ULA and
  cloud metadata targets are rejected.
- Every redirect is re-validated (scheme + DNS + address policy), max 5 hops.
  HTTPS to HTTP downgrades are rejected.
- TCP connects use DNS pinning so validation cannot silently reconnect to a
  different address (DNS rebinding defense).
- Filenames are sanitized; remote path segments are never used as local paths.
- Query strings may contain signed URL parameters; logs redact query values.
  Dataset rows intentionally retain the exact `source_url` and `final_url` for
  retry/provenance. Keep the Dataset private and avoid signed URLs if retaining
  those values would expose credentials.
- Key-Value Store uploads are serialized because the Apify client buffers each
  record body; concurrency is also capped by run memory versus `maxFileSizeMb`.

### Responsible use and retention

Download only files you are authorized to copy and re-host. The Actor does not
bypass authentication, paywalls, robots controls, or access restrictions, but a
public URL does not by itself grant copyright or redistribution rights. Inputs,
Dataset rows, and generated storage URLs can contain personal, confidential, or
regulated data. Keep run storages private, apply the shortest practical
retention period, delete unneeded Key-Value Stores/Datasets, and do not use the
Actor for prohibited or illegal content.

### Limits

- 1–500 URLs per run.
- Concurrency 1–20 (default 5).
- Exact duplicate URLs are downloaded, stored, and billed once; duplicate rows
  share the first occurrence's storage key and `fileName`.
- Max file size 1–256 MiB (default 100). Runs require at least 1 GiB memory
  because the Apify KVS client buffers one serialized upload body.
- No browser, FTP, Drive/Dropbox auth, cookies, ZIP packing or resume ranges in
  this version.

# Actor input Schema

## `urls` (type: `array`):

1-500 public HTTP/HTTPS file URLs. Optional fileName overrides the stored name. Exact duplicate URLs are downloaded once and mirrored into every matching Dataset row; the first duplicate's fileName wins.

## `concurrency` (type: `integer`):

How many files to download at once (1-20). Default 5. Prefer 3-10 for most public hosts.

## `maxFileSizeMb` (type: `integer`):

Reject responses larger than this limit when Content-Length is present, and enforce the same limit while streaming.

## `timeoutSeconds` (type: `integer`):

Timeout applied to each download attempt.

## `maxRetries` (type: `integer`):

Extra attempts after the first try for timeouts, network errors and retryable HTTP statuses (408/425/429/5xx). Failed URLs, security rejections and oversize responses are never retried.

## `overwrite` (type: `boolean`):

When false (default), colliding sanitized file names within the run get a unique Key-Value Store key. When true, the same key may be overwritten.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
      "fileName": "dummy.pdf"
    },
    {
      "url": "https://www.rfc-editor.org/rfc/rfc791.txt",
      "fileName": "rfc791.txt"
    }
  ],
  "concurrency": 5,
  "maxFileSizeMb": 100,
  "timeoutSeconds": 120,
  "maxRetries": 2,
  "overwrite": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

## `billingReceipt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
            "fileName": "dummy.pdf"
        },
        {
            "url": "https://www.rfc-editor.org/rfc/rfc791.txt",
            "fileName": "rfc791.txt"
        }
    ],
    "concurrency": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("automa-flow/parallel-bulk-file-downloader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        {
            "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
            "fileName": "dummy.pdf",
        },
        {
            "url": "https://www.rfc-editor.org/rfc/rfc791.txt",
            "fileName": "rfc791.txt",
        },
    ],
    "concurrency": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("automa-flow/parallel-bulk-file-downloader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
      "fileName": "dummy.pdf"
    },
    {
      "url": "https://www.rfc-editor.org/rfc/rfc791.txt",
      "fileName": "rfc791.txt"
    }
  ],
  "concurrency": 5
}' |
apify call automa-flow/parallel-bulk-file-downloader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automa-flow/parallel-bulk-file-downloader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/suAdVsgPgBwgRtORs/builds/5H1qI2e1YaQpvHfto/openapi.json