Bulk File Downloader

Paste a list of file URLs, or a list of pages, and get the files back in the run's key-value store with one metadata row per file in the dataset. Each file is stored the moment it lands, so a run that stops part-way has delivered everything it fetched, and a second run can pick up where the first one stopped. Files can also be zipped, in parts of at most 500 MB unless one file is bigger than that on its own, so a big job never has to fit in memory to be delivered.

You are responsible for having the right to download and store what you point this Actor at. It fetches over plain HTTP with its own named user agent, honours every site's robots.txt, and refuses a fixed list of hosts outright.

How it works

1. Every URL is checked before anything is requested. Hosts on the refused list, and URLs that are not plain http or https addresses, get a dataset row saying so and are never fetched.
2. For each remaining host the Actor reads robots.txt once. A URL the file disallows, for this Actor's user agent or for *, is skipped with a row and is not charged. There is no setting that turns this off.
3. Each file is fetched with one request started at a time per host and at most two requests a second per host, ten downloads across all hosts, with three attempts on a 429 or a server error. Redirects are followed one hop at a time and every hop gets the same checks.
4. The response is streamed to disk while it is hashed and counted. When it is complete it is written to the key-value store as files-<hash>-<name>, the two events are charged, and the dataset row is pushed with the key, the store URL, the content type, the size and the SHA-256.
5. With Crawl pages on, a URL that returns an HTML page is read for img, source, video and audio sources (including srcset) and links to files on the same host, and those are downloaded. Pages found on pages are not crawled.
6. With Zip set, the stored files are packed into zip parts named zip-part-001.zip and so on after the downloads, and zip-manifest.json lists the parts.

The run stops cleanly at Max files, Max total bytes or the run's maximum total charge, and the status message says how many URLs were left.

Pricing

This Actor uses pay-per-event pricing with two events, charged only for a file that was actually stored:

• file-downloaded, once per file stored: $0.002.
• transfer-10mb, once per 10 MB of the file rounded up, minimum one: $0.001.

So a 200 KB image is 1 × $0.002 + 1 × $0.001 = $0.003, a 5 MB PDF the same, and a 50 MB video is 1 × $0.002 + 5 × $0.001 = $0.007, at the prices shown on this page; the Actor page's own pricing table is the one that bills. A refused URL in your list, a skip (robots, size, wrong type) or a failed request costs nothing and still gets a dataset row explaining why; a refused or unusable link found on a crawled page is dropped without a row, and the page's own row says how many links were queued. The maximum total charge you set on a run is honoured: the Actor checks the remaining allowance before each fetch and stops cleanly when the next file would not fit. Zipping is not charged. Platform compute and storage are billed by the platform as for any Actor.

Tutorial

1. Open the Actor and paste URLs into URLs, one per line. File URLs work as they are; page URLs need Crawl pages switched on.
2. Choose the File types you want. The default is images only; any keeps every kind of file.
3. Set Zip if you want the files packed for download in one go. Leave it off to fetch files one by one from the store.
4. Click Start. Open Storage, then Key-value store, to browse the files (their keys start with files-), or the Dataset for one row per URL with its store URL. The dataset exports as JSON, CSV or Excel.
5. If a run stops early, start a new run with the same input and the first run's ID in Resume from run. Files already in the first run's store are skipped and not charged. Two things to know about a resumed run: its Max files and Max total bytes are its own, not a running total across both runs, and its zip parts hold only the files it fetched itself, so take the first run's zips as well. Resuming needs Unique file names on, because that is what makes a file's key depend on its URL.

Input example

{
  "urls": [
    "https://example.com/gallery/",
    "https://example.com/files/report-2026.pdf"
  ],
  "crawlPages": true,
  "fileTypes": ["image", "document"],
  "maxFiles": 500,
  "zip": "single"
}

Output example

One dataset item per URL processed:

{
  "url": "https://example.com/gallery/photo-1.jpg",
  "sourcePage": "https://example.com/gallery/",
  "status": "downloaded",
  "reason": null,
  "key": "files-3f9a1c2b-photo-1.jpg",
  "storeUrl": "https://api.apify.com/v2/key-value-stores/abc123/records/files-3f9a1c2b-photo-1.jpg",
  "contentType": "image/jpeg",
  "bytes": 48213,
  "sha256": "9b74c9897bac770ffc029102a200c5de3f2e5a4b2f9a9a3f5ad1c9e2a4c0f6d1",
  "charged": { "file": 1, "transfer10mb": 1 },
  "fetchedAt": "2026-09-08T10:15:42.000Z"
}

status is one of downloaded, skipped, refused or failed; reason says why for anything but a download, for example robots.txt disallows, host refused: example.net, over maxFileBytes or HTTP 404.

Limits and rules

• Refused hosts. Major social and media platforms (YouTube, TikTok, Instagram, Facebook, X, Reddit, Pinterest, Vimeo, SoundCloud, Spotify, Twitch and their short-link and CDN hosts) and sites whose published terms forbid automated access are refused at input, before any request. The list ships with the Actor as a generated file and is not configurable.
• robots.txt is always honoured. If a site's robots.txt cannot be read because the server errors, its URLs are skipped rather than guessed at.
• No proxies, headers or cookies. The Actor fetches from its own address with a fixed, descriptive user agent. It cannot log in, cannot pass a bot challenge, and does not try.
• Sizes. A file over Max file size is skipped. The run stops at Max total bytes. Zip parts are capped at 500 MB each, except that a single file larger than that gets a part of its own rather than being dropped; the run log says so when it happens. A file over 32 MB is uploaded to the store as a stream, which the platform client cannot retry; if that upload fails the file gets a failed row and is not charged.
• Each URL once. A URL listed twice, or found on two pages, is fetched and charged once. Where a redirect lands counts too, so two spellings of one file are not stored and charged twice.
• Unknown sizes. When a server sends no Content-Length the size is known only once the file has landed, so two such files in flight can together overshoot Max total bytes by at most one Max file size.
• Names. A stored key is files-<8-character hash of the URL>-<file name>, with the name reduced to letters, digits, dots, dashes and underscores. The hash means the same URL always lands on the same key, which is what makes resuming work.

Use cases

• Archiving a gallery or a media library from a site you run or have permission to copy, straight into storage, with a checksum for every file.
• Collecting documents from a list of report or dataset URLs into one zip for the people who need them.
• Feeding another Actor or workflow that expects files in a key-value store, with the dataset row as the index.
• Agents that need to fetch a set of files as one step: the input is a list of URLs, the output is plain JSON, and the price of a job can be worked out from the sizes before it runs.

FAQ

Does it need any login or API key? No. It fetches public URLs only, with no credentials of any kind.

Can it download from YouTube, Instagram or other platforms? No. Those hosts are refused before any request is made, and the dataset row says so. There is no setting to change that.

What happens to a file that is too big? It is skipped with a row that says over maxFileBytes, and nothing is charged for it.

Is anything charged for a file that failed? No. Only a file that was stored is charged, and every stored file has a row with its key.

How do I get the files out? Each dataset row carries the store URL of its file. With Zip on, the parts are the zip-part- records in the key-value store and zip-manifest.json lists them. A resumed run's parts hold that run's files only, so collect the earlier run's parts too. If zipping itself fails, the run still finishes: every file is in the store and in the dataset, and the manifest carries the error.

Something looks wrong. Open an issue on the Actor's Issues tab with the URL and the dataset row in question.