Parallel Bulk File Downloader – Fast URL Downloads

Download public HTTP/HTTPS files in parallel from a URL list or another Actor's Dataset, stream them directly to Apify storage, and get one manifest row per file with checksums, status, size, and download URL.

Files are saved in the run's Key-Value Store. Open the Dataset to see which downloads succeeded and follow each storage_url to retrieve the file.

Up to five downloads run at once by default, with a separate per-host limit. Total time depends on the file sizes and the servers hosting them.

The per-host limit applies to each redirect destination, including a shared CDN. Repeated HTTP 429/503 responses reduce that host to one active download after existing transfers finish.

Set fileName to choose an output name. Otherwise the Actor uses the response's Content-Disposition filename (filename* takes priority), then the final URL. Names are sanitized for storage; an invalid header falls back to the URL.

See Changelog for release history.

Quick start

For a plain list, paste links into File URLs (one per line) in Console, or send:

{
  "urlsText": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf\nhttps://www.rfc-editor.org/rfc/rfc791.txt"
}

Blank lines and surrounding spaces are ignored. A non-empty urlsText takes priority over urls, so the demo JSON links are not downloaded alongside your list. Leave urlsText blank to use the existing JSON format with custom names:

{
  "urls": [
    {
      "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
      "fileName": "dummy.pdf"
    },
    {
      "url": "https://www.rfc-editor.org/rfc/rfc791.txt",
      "fileName": "rfc791.txt"
    }
  ],
  "concurrency": 5,
  "maxFileSizeMb": 100,
  "timeoutSeconds": 120,
  "maxRetries": 2,
  "overwrite": false
}

Default Console / API prefill uses the same bounded public demo URLs.

Download files from another Actor

Select the upstream Actor's Dataset in Source Dataset, ideally after that Actor finishes. Set urlField to the field containing one file URL per row:

{
  "datasetId": "YOUR_DATASET_ID",
  "urlField": "image.url",
  "metadataFields": ["productId", "sku", "SKU"],
  "maxItems": 1000
}

Use the actual 17-character Dataset ID. datasetId overrides the demo URLs, urlsText and urls. Dot paths such as image.url or images.0.url select one value, without expanding arrays. Optional fileNameField selects a custom file name. The Dataset must be accessible to your account; the input requests read-only access.

Each output row preserves source_dataset_id, the absolute source_item_index, and selected identifiers inside metadata, for example {"productId": 123, "SKU": "BLUE-M"}. The defaults preserve productId, sku and SKU. Missing identifiers are omitted. Select up to 20 scalar fields, at most 1024 JSON bytes per value and 4096 bytes together. Invalid values or missing URLs produce explicit failed rows. Unselected source fields are not copied.

The Actor reads pages of 50 rows and freezes the selected range before downloading. maxItems defaults to 1000 and supports up to 10,000 rows per batch. datasetOffset selects the starting row. NEXT_INPUT provides the next range when more rows exist. Finish the current range before starting the next one. Each range is independent, so URL deduplication does not span separate ranges. Apify Datasets only grow, so rows added later never change a range already read. If the upstream Actor is still adding rows, the run downloads the rows it read, sets RUN_SUMMARY.datasetChangedDuringRead and leaves the new rows to NEXT_INPUT. RUN_SUMMARY.datasetTotalIsExact is false when more rows exist; in that case datasetTotalItems is a lower bound, because Apify's total counter can lag writes.

Continue after an interruption or spending limit

After the run stops, open its Key-Value Store and copy CONTINUE_INPUT into a new run's JSON input. It contains resumeFromStoreId and resumeStoreIds. Copy both fields: the latter grants read-only access to earlier files in a continuation chain. The new run inherits the frozen source, field mapping and download settings, rebuilds the complete manifest for that range, and downloads only unfinished files. Confirmed files are reused without another file event; reused: true and file_store_id identify those rows. Repeated URLs with different product IDs retain separate output rows but share one file and one file event within the batch.

Keep all previous run storages until you no longer need their files. Reused storage_url values still point to the original storage. Do not start concurrent continuations or branch repeatedly from an older checkpoint: use the most recent stopped run. Up to 20 continuation generations are supported. Each new run still has the normal start event. A file already stored and charged is not charged again. A transfer that is still running when the spending limit is reached is not stored and is not charged.

If a charge acknowledgement is uncertain, continuation stops for reconciliation; it never guesses that charging again is safe. A missing previously stored file also stops continuation. Use a fresh batch to change field mappings or source values. If interruption happened before CONTINUE_INPUT was saved, no file downloads or file charges have started, so start the original input again. Use a new run for continuation.

If Apify restarts or migrates a running run, the run continues on its own in both URL-list and Dataset mode. Files already stored and charged are reused without another file event, rows already in the Dataset are not added twice, and RUN_SUMMARY.restartsRecovered records the restart. A charge that was in progress at the restart is checked against Apify's charge record; if Apify does not confirm it within a minute, the run stops with BILLING_UNCERTAIN for review instead of charging again.

Large batches and the run timeout

Shortly before the run timeout, the Actor stops starting downloads and cuts transfers that are still running. Files already stored finish their charge, and every remaining row is published with RUN_TIME_LIMIT, so the run ends SUCCEEDED with RUN_SUMMARY.status PARTIAL (or RUN_TIME_LIMIT when no file finished) and runTimeLimitRows. It keeps 45 seconds plus half a second per 50 rows for this, at most 40% of the remaining time. Start CONTINUE_INPUT (Dataset batches) or RETRY_INPUT (URL lists) in a new run, or raise the run timeout. A run without a timeout is not stopped early.

Checkpoints contain selected source identifiers and URLs, including signed query parameters. Treat them like the input and keep them private. This integration reduces workflow setup; it does not guarantee higher throughput or sales.

Example workflows

Choose a batch you already have permission to download:

1. Public PDF documents: replace the first demo URL with a document URL and set fileName to a recognizable name. Inspect its SUCCESS row before expanding the batch.
2. Mixed files: use the PDF and text URLs in the quick start above. The Dataset manifest keeps each input's status, checksum, size and storage URL, including explicit failures.
3. Recurring document batches: pass the next list of document URLs from your workflow and use sha256 downstream to compare file contents. Each run stores its own files; the Actor does not maintain a change-monitor baseline.

Download the files from your result

Open the run's Dataset and select a row with status == "SUCCESS". Its storage_url points to that file in the run's Key-Value Store; open it using your authorized Apify access, or pass it to the next step of your automation. Keep the manifest if you need to match downloaded files back to their input URLs. Failed rows do not represent downloadable files. Files remain subject to your Apify storage retention settings; they are not bundled into a ZIP or uploaded to external storage.

Retry unsuccessful files

After a completed batch, read RUN_SUMMARY. If retryStatus is AVAILABLE, open the Retry input output link or the RETRY_INPUT record in the run's Key-Value Store. Copy its complete JSON into a new run. It contains only failed URLs, deduplicated, with original explicit file names and the same download settings. Its blank urlsText prevents an old pasted list from overriding it. Successful files are excluded. No second run starts automatically.

Review the Failed Dataset view first: expired URLs, missing files, security rejections and size limits may need correction. Set the new run's spending limit, memory and timeout separately; these platform run options are not part of Actor input. A later successful download is billed normally. Files from the first run remain in that run's storage; the new run contains only retry results.

retryStatus: NOT_NEEDED means there is no retry input. If billing has an unknown outcome, retryStatus: BLOCKED_BILLING_UNCERTAIN suppresses the entire retry input until charges are reconciled. RETRY_INPUT is absent in both cases and is not promised for runs interrupted before the batch result is saved. Like INPUT and Dataset provenance, it retains exact URLs, including signed query parameters: keep it private.

In development: product image exports

I'm working on two additions for catalog migration workflows:

• ZIP archives organized by SKU, with JSON and CSV manifests.
• Export to your own Amazon S3 bucket.

These features are not available in the public version yet. Release details and pricing will be announced here.

Would this help your workflow? Open an Issue and share your preferred destination (ZIP or S3) and approximate batch size.

Related tools for finding or monitoring files

• Get attachment URLs from a paginated API: Paginated REST API Fetcher exports API items across a configured pagination pattern. Select permitted public file URLs from the item data and map them into this Actor's urls input.
• Monitor a PDF across revisions: PDF Change Monitor keeps a last-good baseline for a direct PDF or a PDF selected from a stable webpage. Choose it when you need content and link change events, rather than a fresh download manifest.
• Read HTML as Markdown: Website Content Crawler extracts authorized website text and optional chunks for AI and RAG. Give it the original HTML page URL and rights declaration; it does not parse the downloaded binary files.

Each linked tool has separate input and pricing. Select the URLs and start the appropriate run yourself or through your own automation.

Output example

Successful row:

{
  "source": "parallel-bulk-file-downloader",
  "source_id": "…",
  "status": "SUCCESS",
  "source_url": "https://example.com/files/report.pdf",
  "final_url": "https://cdn.example.com/report.pdf",
  "source_index": 0,
  "file_name": "report.pdf",
  "content_type": "application/pdf",
  "bytes": 4123456,
  "sha256": "…",
  "kvs_key": "files-000000-report.pdf",
  "storage_url": "https://api.apify.com/v2/key-value-stores/…/records/files-000000-report.pdf",
  "http_status": 200,
  "attempts": 1,
  "duration_ms": 842,
  "error_code": null,
  "error_message": null,
  "downloaded_at": "2026-09-17T21:00:00Z",
  "scraped_at": "2026-09-17T21:00:00Z",
  "fingerprint": "…",
  "schema_version": 2
}

Dataset views: All, Downloaded, Failed.

Automation / API use

curl -X POST "https://api.apify.com/v2/acts/automa-flow~parallel-bulk-file-downloader/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":[{"url":"https://www.rfc-editor.org/rfc/rfc791.txt","fileName":"rfc791.txt"}],"concurrency":5}'

Schedule a recurring URL batch or call the Actor from n8n / Make / your backend. A second run is useful whenever you have a new URL list, a refreshed CDN export, or need checksums again for integrity checks.

AI agents / MCP

Direct tool URL (authenticated; each caller uses their own Apify OAuth or token):

https://mcp.apify.com?tools=automa-flow/parallel-bulk-file-downloader

Anonymous MCP discovery (search-actors) indexes this public Store Actor. The direct tool URL above is the authenticated path for initialize, tools/list and a paid run. Direct agentic payment needs pay-per-event prices with platform usage included, limited permissions and no Standby mode. This Actor meets those product rules once the included-usage price below is the active Store price. Developer identity verification is a separate Apify requirement and is not claimed here.

When an agent should call it: the user already has a list of public HTTP/HTTPS file URLs and needs files in Apify storage plus a per-file manifest (status, storage_url, SHA-256) for a downstream step (CRM attachment sync, integrity re-check, report pack).

When not to call it: crawl/HTML extraction, authenticated Drive/Dropbox, video-site downloaders, ZIP packing, or any private/localhost URL (SSRF policy rejects those per item).

The event price is the whole bill. Platform usage is included. It is not added on top.

Example	Event charges at 1 GiB
Two files of 10 MiB or less	$0.00805 ($0.00005 start + 2 × $0.004)
100 such files	$0.40005
One 100 MiB file	$0.03105 ($0.00005 + $0.004 + 9 × $0.003)
One 256 MiB file, the maximum	$0.07905

stored-10mib is charged only when the active Store price lists that event. Until then a run bills one file-downloaded per stored file under the price entry that is active for the run.

Always set maxTotalChargeUsd (Store minimum $0.00405, enough for one small file). The cap limits event charges. At the cap, already stored successful files remain. A file that would not fit, including its size units, is not stored and is not charged. Remaining URLs get BUDGET_EXCEEDED or cancelled rows.

Example agent ask: “Download these public PDF URLs in parallel with maxTotalChargeUsd=0.05, then read RUN_SUMMARY and return Dataset rows with storage_url and sha256 for SUCCESS items only.”

After the run, read Key-Value Store RUN_SUMMARY first (status, successful / failed, chargedEventCounts, nextAction), then the default Dataset. Output schema also exposes runSummary, billingReceipt and conditional retryInput links.

Failure semantics

error_code	Meaning
SECURITY_REJECTED	URL/host/IP failed SSRF policy (localhost, private, metadata, bad scheme).
HTTP_404 / HTTP_403 / …	Remote server returned that status.
HTTP_ERROR	Unusable HTTP response, including partial HTTP 206 or HTTP 204/205 without a file.
RATE_LIMITED	HTTP 429 after retry budget.
TIMEOUT	Per-attempt source-download deadline exhausted, including DNS, redirects and body transfer.
FILE_TOO_LARGE	Content-Length or streamed bytes exceeded maxFileSizeMb.
STORAGE_FAILED	Key-Value Store write failed after a successful download.
BUDGET_EXCEEDED	Spending limit stopped storing/charging further successful files. If the limit cannot cover even one file, every URL gets this code, the run succeeds with RUN_SUMMARY.status: BUDGET_EXCEEDED and RETRY_INPUT holds all URLs.
BILLING_UNCERTAIN	Charge API failed after a store write; the run fails closed (no replay).
RUN_TIME_LIMIT	The run timeout arrived before this file finished. Nothing was stored or charged for it; continue in a new run.

NO_RESULTS is not used: when both URL inputs are empty, runtime validation returns INVALID_INPUT and the run fails without downloading files. Malformed text lists and field bounds are rejected by the platform before a run starts. An ordinary item failure never cancels independent downloads. A BILLING_UNCERTAIN outcome stops the batch, cancels and drains sibling workers, deletes files known to be uncharged, and retains the uncertain file plus its Dataset row for reconciliation.

timeoutSeconds excludes queue time, retry backoff and storage uploads. Slow sources may need a higher value. A valid empty file returned with HTTP 200 is accepted; partial downloads and no-content responses are not complete files.

Pricing

Pay per event. Platform usage (compute, transfer and storage) is included in these prices. There is no separate platform usage charge.

Event	Price
apify-actor-start	$0.00005 per run at 1 GiB. Apify charges one event per GiB of memory.
file-downloaded	$0.004 per successfully stored unique file. Includes the first 10 MiB.
stored-10mib	$0.003 per additional started 10 MiB after that first 10 MiB.

A 10 MiB file is $0.004. A file of 10 MiB plus one byte is $0.007. A 100 MiB file is $0.031. The 256 MiB maximum is $0.079. Add $0.00005 for the run start at 1 GiB.

Not charged: failed URLs, retries, redirects, security rejections, oversized responses, or duplicate copies of a URL that was already stored in the same batch.

The size event is billed only when your run's price list includes stored-10mib. Until that Store price is active, the Actor charges file-downloaded only. If a size charge cannot be confirmed, the run stops with BILLING_UNCERTAIN instead of reporting the file price as the whole bill.

Security and SSRF protection

• Only http and https on ports 80/443.
• Localhost, loopback, RFC1918, link-local, multicast, reserved, IPv6 ULA and cloud metadata targets are rejected.
• Every redirect is re-validated (scheme + DNS + address policy), max 5 hops. HTTPS to HTTP downgrades are rejected.
• TCP connects use DNS pinning so validation cannot silently reconnect to a different address (DNS rebinding defense).
• Filenames are sanitized; remote path segments are never used as local paths.
• Query strings may contain signed URL parameters; logs redact query values. Dataset rows intentionally retain the exact source_url and final_url for retry/provenance. Keep the Dataset private and avoid signed URLs if retaining those values would expose credentials.
• Key-Value Store uploads are serialized because the Apify client buffers each record body; concurrency is also capped by run memory versus maxFileSizeMb.

Responsible use and retention

Download only files you are authorized to copy and re-host. The Actor does not bypass authentication, paywalls, robots controls, or access restrictions, but a public URL does not by itself grant copyright or redistribution rights. Inputs, Dataset rows, and generated storage URLs can contain personal, confidential, or regulated data. Keep run storages private, apply the shortest practical retention period, delete unneeded Key-Value Stores/Datasets, and do not use the Actor for prohibited or illegal content.

Limits

• 1-500 URLs from urlsText or urls, or up to 10,000 Dataset rows per batch. Provide a URL source or resumeFromStoreId; this is checked at runtime.
• Concurrency 1–20 (default 5).
• Exact duplicate URLs are downloaded, stored, and billed once; duplicate rows share the first occurrence's storage key and fileName.
• Max file size 1–256 MiB (default 100). Runs require at least 1 GiB memory because the Apify KVS client buffers one serialized upload body.
• No browser, FTP, Drive/Dropbox auth, cookies, ZIP packing or HTTP byte-range resume in this version.