Bulk File and Image Downloader: URLs to ZIP

Paste a list of file links and get them back as a ZIP, as separate download links, or both. It works with any file type: photos, PDFs, spreadsheets, audio, video, archives. You can also give it web pages, and it collects the images or the linked files each page shows.

Every file you get has a row in the dataset with its name, type, size in bytes and SHA-256 checksum, so you can check nothing was damaged on the way.

What it does

• Downloads each direct link you list, following up to 5 redirects
• Reads each page you list once and collects its images: img tags, lazy loaded image attributes, social share images, inline style backgrounds and links to image files. Turn on srcset to get every responsive size as well.
• Collects linked files from pages by extension instead, or as well. The default list covers PDF, Word, Excel, PowerPoint, CSV, text, ZIP, MP3 and MP4, and you can set your own.
• Packs everything into ZIP files. A new part starts once the current one reaches the part size you set (200 MB by default).
• Names files from the server's Content-Disposition header or the URL. Two files with one name inside a ZIP become 1.jpg and 1-2.jpg, and a file with no extension gets one from its content type.
• Skips duplicate links, so a URL listed twice is downloaded and charged once
• Stores images, PDFs, DOCX, XLSX and PPTX files, audio, video and archives in the ZIP without recompressing them. Text and CSV files are compressed.

Example

One test run took four direct links and one page from the IRS forms site, with linked PDFs as the target, robots.txt off and ZIP only as the output. It returned a single ZIP of 4,958,516 bytes holding two JPEGs, a PNG and three PDFs. The same JPEG listed twice with a different query string came out as 1.jpg and 1-2.jpg, and a missing image got an HTTP 404 row.

Input

• File URLs: direct links to files
• Page URLs: pages to collect files from. Pages are not crawled any further.
• Find on pages: images (default), linked files, or both
• Linked file extensions, used when finding linked files
• Include srcset images (default off)
• Same host only (default off). Leave it off for sites that serve images from a CDN.
• Maximum files per page (default 500)
• Output: ZIP and files (default), ZIP only, or files only
• ZIP part size in MB (default 200). Keep it under half the run memory.
• Maximum file size in MB (default 100). Bigger files get an error row.
• Respect robots.txt (default on)
• Parallel downloads (default 4)
• Timeout per request in seconds (default 60)

You need at least one link in File URLs or Page URLs.

Output

The dataset has one row per file. A downloaded file has ok: true, fileName, contentType, bytes, sha256, and fileUrl (its own download link, in the ZIP and files and files only modes). zipFile and zipEntry give the ZIP part and the name inside it. pageUrl shows the page a file was found on, and finalUrl appears when the link redirected.

The OUTPUT record in the key value store lists every ZIP part with its download link, file count and size, plus the totals for the run.

A link that fails gets ok: false and an error, and the run goes on to the next one. Errors include HTTP 404, an unknown host, a link that is not http or https, a file over the size limit, "blocked by the site's robots.txt", or a page with no matching files.

Robots.txt and rate limits

With Respect robots.txt on, it reads each site's robots.txt before its first request there, skips any URL that file disallows, and waits the crawl delay the file sets between requests to that host. Some file hosts disallow every path in robots.txt, which blocks direct links too. Turn the option off only for files you are allowed to download.

No host gets more than 2 requests at once, whatever the parallel setting is. Timeouts and busy answers (429, 500, 502, 503 and 504) are retried twice, after the wait the server asks for in Retry-After, up to 30 seconds.

Pricing

Pay per event: one event for each file you get. Links that fail, pages and skipped duplicates cost nothing. When the run reaches the maximum charge you set, it stops downloading, uploads the ZIP it has so far and finishes cleanly. OUTPUT then shows stoppedAtChargeLimit: true and how many files were left.

Scope and conduct

Only the links and pages you supply are fetched. It does not log in, does not use proxies, and refuses hosts that resolve to private or local addresses. Download only files you have the right to use.

Support

Questions and problems: the Issues tab on this page.