Parallel Bulk File Downloader – Fast URL Downloads

Download hundreds of public HTTP/HTTPS files in parallel from a URL list, stream them directly to Apify storage, and get one manifest row per file with checksums, status, size, and download URL.

Files are saved in the run's Key-Value Store. Open the Dataset to see which downloads succeeded and follow each storage_url to retrieve the file.

Up to five downloads run at once by default, with a separate per-host limit. Total time depends on the file sizes and the servers hosting them.

Quick start

{
  "urls": [
    {
      "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
      "fileName": "dummy.pdf"
    },
    {
      "url": "https://www.rfc-editor.org/rfc/rfc791.txt",
      "fileName": "rfc791.txt"
    }
  ],
  "concurrency": 5,
  "maxFileSizeMb": 100,
  "timeoutSeconds": 120,
  "maxRetries": 2,
  "overwrite": false
}

Default Console / API prefill uses the same bounded public demo URLs.

Example workflows

Choose a batch you already have permission to download:

1. Public PDF documents: replace the first demo URL with a document URL and set fileName to a recognizable name. Inspect its SUCCESS row before expanding the batch.
2. Mixed files: use the PDF and text URLs in the quick start above. The Dataset manifest keeps each input's status, checksum, size and storage URL, including explicit failures.
3. Recurring document batches: pass the next list of document URLs from your workflow and use sha256 downstream to compare file contents. Each run stores its own files; the Actor does not maintain a change-monitor baseline.

Download the files from your result

Open the run's Dataset and select a row with status == "SUCCESS". Its storage_url points to that file in the run's Key-Value Store; open it using your authorized Apify access, or pass it to the next step of your automation. Keep the manifest if you need to match downloaded files back to their input URLs. Failed rows do not represent downloadable files. Files remain subject to your Apify storage retention settings; they are not bundled into a ZIP or uploaded to external storage.

Related tools for finding or monitoring files

• Get attachment URLs from a paginated API: Paginated REST API Fetcher exports API items across a configured pagination pattern. Select permitted public file URLs from the item data and map them into this Actor's urls input.
• Monitor a PDF across revisions: PDF Change Monitor keeps a last-good baseline for a direct PDF or a PDF selected from a stable webpage. Choose it when you need content and link change events, rather than a fresh download manifest.
• Read HTML as Markdown: Website Content Crawler extracts authorized website text and optional chunks for AI and RAG. Give it the original HTML page URL and rights declaration; it does not parse the downloaded binary files.

Each linked tool has separate input and pricing. Select the URLs and start the appropriate run yourself or through your own automation.

Output example

Successful row:

{
  "source": "parallel-bulk-file-downloader",
  "source_id": "…",
  "status": "SUCCESS",
  "source_url": "https://example.com/files/report.pdf",
  "final_url": "https://cdn.example.com/report.pdf",
  "source_index": 0,
  "file_name": "report.pdf",
  "content_type": "application/pdf",
  "bytes": 4123456,
  "sha256": "…",
  "kvs_key": "files-000000-report.pdf",
  "storage_url": "https://api.apify.com/v2/key-value-stores/…/records/files-000000-report.pdf",
  "http_status": 200,
  "attempts": 1,
  "duration_ms": 842,
  "error_code": null,
  "error_message": null,
  "downloaded_at": "2026-09-17T21:00:00Z",
  "scraped_at": "2026-09-17T21:00:00Z",
  "fingerprint": "…",
  "schema_version": 2
}

Dataset views: All, Downloaded, Failed.

Automation / API use

curl -X POST "https://api.apify.com/v2/acts/automa-flow~parallel-bulk-file-downloader/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":[{"url":"https://www.rfc-editor.org/rfc/rfc791.txt","fileName":"rfc791.txt"}],"concurrency":5}'

Schedule a recurring URL batch or call the Actor from n8n / Make / your backend. A second run is useful whenever you have a new URL list, a refreshed CDN export, or need checksums again for integrity checks.

AI agents / MCP

Direct tool URL (authenticated; each caller uses their own Apify OAuth or token):

https://mcp.apify.com?tools=automa-flow/parallel-bulk-file-downloader

Anonymous MCP discovery (search-actors) indexes this public Store Actor. The direct tool URL above is the authenticated path for initialize, tools/list and a paid run. From 2026-10-06 20:23 UTC, direct agentic payment is not available, because platform usage is charged in addition to events.

When an agent should call it: the user already has a list of public HTTP/HTTPS file URLs and needs files in Apify storage plus a per-file manifest (status, storage_url, SHA-256) for a downstream step (CRM attachment sync, integrity re-check, report pack).

When not to call it: crawl/HTML extraction, authenticated Drive/Dropbox, video-site downloaders, ZIP packing, or any private/localhost URL (SSRF policy rejects those per item).

Event charges are bounded. From 2026-10-06 20:23 UTC, platform usage is extra and depends on file size:

Example	Event charges	Measured platform usage
Two small files	$0.00805 ($0.00005 start + 2 × $0.004)	about $0.001
100 unique successful files	$0.40005	depends on total bytes
Schema max 500 unique successes	$2.00005	depends on total bytes
One 100 MiB file	$0.00405	about $0.020

Until 2026-10-06 20:23 UTC the Store does not charge platform usage on top of these events. After that time Apify meters compute, transfer and storage for the user. The usage figures are measurements from 1 GiB runs on 2026-09-18, not a quote.

Always set maxTotalChargeUsd on the run (Store minimum $0.00405). That cap limits event charges. At the cap, already stored successful files remain; remaining URLs get BUDGET_EXCEEDED / cancelled rows and are not charged. From the date above, platform usage is billed separately from that cap.

Example agent ask: “Download these public PDF URLs in parallel with maxTotalChargeUsd=0.05, then read RUN_SUMMARY and return Dataset rows with storage_url and sha256 for SUCCESS items only.”

After the run, read Key-Value Store RUN_SUMMARY first (status, successful / failed, chargedEventCounts, nextAction), then the default Dataset. Output schema also exposes runSummary and billingReceipt links.

Failure semantics

error_code	Meaning
SECURITY_REJECTED	URL/host/IP failed SSRF policy (localhost, private, metadata, bad scheme).
HTTP_404 / HTTP_403 / …	Remote server returned that status.
RATE_LIMITED	HTTP 429 after retry budget.
TIMEOUT	Per-attempt timeout exhausted.
FILE_TOO_LARGE	Content-Length or streamed bytes exceeded maxFileSizeMb.
STORAGE_FAILED	Key-Value Store write failed after a successful download.
BUDGET_EXCEEDED	Spending limit stopped storing/charging further successful files.
BILLING_UNCERTAIN	Charge API failed after a store write; the run fails closed (no replay).

NO_RESULTS is not used: an empty urls array is invalid input and never starts. An ordinary item failure never cancels independent downloads. A BILLING_UNCERTAIN outcome stops the batch, cancels and drains sibling workers, deletes files known to be uncharged, and retains the uncertain file plus its Dataset row for reconciliation.

Pricing

Pay per event, plus Apify platform usage:

Event	Price
apify-actor-start	$0.00005 / run (platform synthetic; memory may multiply)
file-downloaded	$0.004 / successfully stored unique file

Not charged as events: failed URLs, retries, redirects, security rejections, or bytes that never reach a successful store write.

Platform usage is passed through to the user from 2026-10-06 20:23 UTC (compute units, external transfer, Key-Value Store storage). Until then the Store does not add that charge. It is not included in the $0.004 file price. The two-file demo measured about $0.001 of usage; a 100 MiB file measured about $0.02 of usage on top of the $0.004 event.

Security and SSRF protection

• Only http and https on ports 80/443.
• Localhost, loopback, RFC1918, link-local, multicast, reserved, IPv6 ULA and cloud metadata targets are rejected.
• Every redirect is re-validated (scheme + DNS + address policy), max 5 hops. HTTPS to HTTP downgrades are rejected.
• TCP connects use DNS pinning so validation cannot silently reconnect to a different address (DNS rebinding defense).
• Filenames are sanitized; remote path segments are never used as local paths.
• Query strings may contain signed URL parameters; logs redact query values. Dataset rows intentionally retain the exact source_url and final_url for retry/provenance. Keep the Dataset private and avoid signed URLs if retaining those values would expose credentials.
• Key-Value Store uploads are serialized because the Apify client buffers each record body; concurrency is also capped by run memory versus maxFileSizeMb.

Responsible use and retention

Download only files you are authorized to copy and re-host. The Actor does not bypass authentication, paywalls, robots controls, or access restrictions, but a public URL does not by itself grant copyright or redistribution rights. Inputs, Dataset rows, and generated storage URLs can contain personal, confidential, or regulated data. Keep run storages private, apply the shortest practical retention period, delete unneeded Key-Value Stores/Datasets, and do not use the Actor for prohibited or illegal content.

Limits

• 1–500 URLs per run.
• Concurrency 1–20 (default 5).
• Exact duplicate URLs are downloaded, stored, and billed once; duplicate rows share the first occurrence's storage key and fileName.
• Max file size 1–256 MiB (default 100). Runs require at least 1 GiB memory because the Apify KVS client buffers one serialized upload body.
• No browser, FTP, Drive/Dropbox auth, cookies, ZIP packing or resume ranges in this version.