Bulk File & Image Downloader

Give it a list of file or image links. It downloads each file straight into the run's Apify key-value store and returns one dataset row per file with a direct link to the stored copy, its size, type and SHA-256 checksum.

Built for big lists and big files:

• Streaming in 256 KB chunks. A file goes from the source to storage without ever being held in memory, so memory use stays flat. Measured: a 115 MB (120,357,629-byte) video stored in a 128 MB run on Apify with a platform peak of 65 MB of memory (79 MB by the Actor's own process count); locally the peak was 73 MB. Files of several GB work the same way, within the size cap you set.
• Hard caps on file count (maxFiles, default 100) and file size (maxFileSizeMb, default 100 MB, up to 5,000). A file over the size cap is skipped before download when the server states its size, or stopped and deleted the moment it passes the cap. Skipped files are never charged.
• Polite fetching: follows each site's robots.txt, at most 2 parallel downloads per host with a pause between requests, and a longer pause when robots.txt sets a Crawl-delay (up to 30 s).
• Safe by default: links to private, internal or cloud-metadata addresses are refused (see Security).

Your responsibility: rights to the content

Only download files you have the right to download, copy and store. That means content you own, content under a licence that allows it (for example public-domain or Creative Commons files, used under the licence terms), or content you have permission for. This Actor is a download tool: it does not check copyright, it does not bypass logins, paywalls, captchas or DRM, and it must not be used to infringe copyright or to collect other people's private data. Respect each site's terms of service. Stored files stay in your own Apify account, and you are responsible for how you use them.

Input

Field	Default	What it does
urls	3 public-domain NASA images	Direct links to files (http/https)
maxFiles	100	Hard cap on files stored in the run
maxFileSizeMb	100	Hard cap per file, in MB (1 MB = 1,048,576 bytes)
allowedContentTypes	any	Keep only files whose Content-Type starts with one of these, e.g. image/, application/pdf
allowHtmlPages	false	HTML web pages are skipped unless you allow them (a link to a page is usually a viewer, not the file)
maxConcurrency	4	Parallel downloads (max 2 per host)
requestTimeoutSecs	600	Per-file time limit
keyPrefix	none	Text added to the start of every storage key

{
  "urls": [
    "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/1/1b/Federal_Register-Vol74No33.pdf"
  ],
  "maxFiles": 500,
  "maxFileSizeMb": 250,
  "allowedContentTypes": ["image/", "application/pdf"]
}

Output

One dataset row per stored file:

{
  "url": "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
  "finalUrl": "https://upload.wikimedia.org/wikipedia/commons/1/1b/Apollo_11_1998_scan.jpg",
  "key": "00001-Apollo_11_1998_scan.jpg",
  "recordUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/00001-Apollo_11_1998_scan.jpg",
  "fileName": "Apollo_11_1998_scan.jpg",
  "contentType": "image/jpeg",
  "sizeBytes": 154428,
  "sizeMb": 0.147,
  "extra10MbUnits": 0,
  "sha256": "…",
  "httpStatus": 200,
  "lastModified": "…",
  "etag": "…",
  "redirects": 0,
  "downloadSecs": 0.41,
  "downloadedAt": "2026-11-02T19:40:00Z"
}

The files themselves are in the run's default key-value store under key. recordUrl needs your API token unless you share the store. The RUN-SUMMARY record lists every skipped or failed link with its reason (duplicate, unsupported-url, blocked-private-address, blocked-by-robots-txt, http-error, content-type-not-allowed, html-page-not-file, over-size-cap, over-max-files, over-run-budget, timeout, download-failed), plus bytes stored and peak memory.

Storage note: files in a run's default key-value store follow your Apify plan's data retention (unnamed storages are removed after the retention period), and stored data counts toward storage usage on your Apify plan. Copy anything you need to keep.

Security (SSRF protection)

• Only http and https links.
• The Actor resolves every host itself and refuses the link if any address is private, loopback, link-local (including the 169.254.169.254 cloud-metadata address), multicast, reserved or otherwise not public. localhost, *.local and *.internal names are refused.
• The connection is pinned to the checked address, so a DNS answer cannot change between the check and the download.
• Redirects are followed by the Actor (at most 5), and every hop is checked again, robots.txt included.
• Connections are never reused across hostnames.

Robots, terms and rate limits

• robots.txt is read for every site before downloading (user agent token BrimfoldStudio-FileDownloader), following RFC 9309. If it can't be fetched because of a server error, timeout or HTTP 429, the site is treated as off limits. If it doesn't exist (other 4xx), downloads are allowed.
• At most 2 downloads at a time per host, 0.25 s apart, or the site's Crawl-delay (capped at 30 s). Rate-limit replies (429/503) are retried up to 3 times, honouring Retry-After up to 60 s.
• The Actor sends an honest user agent, never logs in and never solves captchas. Respect each site's terms of service. Some hosts (for example Wikimedia) publish their own policies for automated downloads.
• Sources: files come from the URLs you provide. The default input uses public-domain NASA images hosted on Wikimedia Commons.

Pricing

Pay per event, nothing for skipped or failed files:

• $0.0025 per file stored (file-downloaded, one per dataset row)
• $0.001 per extra 10 MB (extra-10mb): each started 10 MB block beyond a file's first 10 MB. 10 MB here means 10,485,760 bytes. Examples: a 4 MB image costs $0.0025; a 25 MB file costs $0.0025 + 2 × $0.001 = $0.0045; a 115 MB video costs $0.0025 + 11 × $0.001 = $0.0135.
• Plus Apify's small actor-start fee.

If you set a maximum charge for the run, the Actor reserves the worst case for each file before downloading it (using the stated size, or the size cap when the size is unknown) and skips files that would not fit, so the cap is never passed. Discounts by Apify plan are listed in PRICING.md.

FAQ

Does it crawl pages to find files? No. Give it direct file links. To collect links first, use a crawler, then pass the list here.

Can it download from sites behind a login? No, only public links.

Why was my link "blocked-by-robots-txt"? The site's robots.txt asks automated clients not to fetch that path. The Actor follows it, with no override.

---

Made by Brimfold Studio. Default sample images: NASA, public domain, via Wikimedia Commons.