Image Downloader: Bulk Images to ZIP, WebP & Audit

Urlwright — Little tools for the web.

Download every image from pages, sitemaps, CSV/Shopify exports or Apify datasets. One dataset row per image with alt text, dimensions, hashes and audit flags. Optional ZIP parts, WebP/AVIF conversion, renaming by SKU, and a diff mode that only fetches new or changed files.

Default User-Agent: UrlwrightBot/1.0 (image-downloader; +https://urlwright.com/bot) (UrlwrightBot for robots.txt matching).

What it does

Give it page URLs, direct image URLs, a sitemap, a domain, a CSV/Shopify export, or the dataset from a Sitemap URL Extractor run. For each discovered image you get one row with:

1. Discovery – img, srcset/picture, lazy-load attributes, <noscript>, Open Graph/Twitter cards, CSS backgrounds (inline and linked stylesheets), icons, image links, image sitemaps, CSV columns and dataset fields. Each hit records foundIn and the page URL(s).
2. Download or list – Full download with magic-byte validation, SHA-256 and perceptual hash (dHash), or audit mode that lists URLs and metadata without storing files (optional partial fetch for dimensions).
3. Files – Individual files in the run key-value store (public links), and/or streamed ZIP parts (STORE, no re-compression) split by size. Template-based renaming ({sku}, {handle}, {domain}, {pathSlug} = image URL file stem, {pageSlug}, {position}, …). Case and Unicode letters are kept in ZIP/fileName; KV keys are ASCII-safe.
4. Processing – Optional convert to WebP/AVIF/JPEG/PNG, resize, thumbnails, safe SVG rasterization. GPS, author, serial and similar EXIF/XMP fields are always stripped; orientation can be preserved.
5. Audit – missingAlt, oversized, legacyFormat, webpSavingsEstimate, exifFound, gpsFound (boolean only, never coordinates). Aggregated REPORT.md / REPORT.json and optional image-sitemap.xml.
6. Diff – With onlyNew, unchanged images return unchanged and cost only image-listed; removed URLs get status: removed.

Polite by default: robots.txt on page and image hosts, at most 1 request per second per host, honest User-Agent, Referer set to the discovering page.

Good for

Shop migration & SKU workflows – Shopify CSV preset (Image Src, Variant Image, Handle, SKU, alt text) plus {sku} / {handle} in file names · Agency relaunch backups – whole-site sitemap runs with alt text preserved in the dataset · SEO & accessibility audits – missing alt, heavy PNG/JPEG, WebP savings estimate without downloading every byte · AI / ML datasets – hashes, dimensions and source URLs from sites you are licensed to use · Archives & change tracking – scheduled runs with onlyNew and stateKey to avoid duplicate storage.

Input examples

Store console prefill: startUrls: [{ "url": "https://books.toscrape.com/" }], maxImages: 20.

Homepage scan (default):

{
  "startUrls": [{ "url": "https://books.toscrape.com/" }],
  "maxImages": 50,
  "output": "both",
  "zipGroupBy": "page",
  "dedupeSimilar": true
}

Shopify export (public CSV URL):

{
  "csvUrl": "https://example.com/exports/products.csv",
  "csvPreset": "shopify",
  "fileNameTemplate": "{handle}/{sku}-{position}",
  "maxImages": 5000
}

Whole site via sitemap:

{
  "domains": ["example.com"],
  "includePathGlobs": ["/products/**"],
  "maxPages": 500,
  "crawl": { "enabled": false }
}

Sitemap URL Extractor dataset:

{
  "sitemapDatasetId": "YOUR_DATASET_ID",
  "maxPages": 1000,
  "renderJs": false
}

Audit only (no file storage):

{
  "startUrls": [{ "url": "https://example.com/" }],
  "mode": "audit",
  "auditDimensions": true,
  "output": "files"
}

Incremental backup:

{
  "domains": ["example.com"],
  "onlyNew": true,
  "stateKey": "prod-images-weekly",
  "maxImages": 10000
}

• startUrls – HTML pages to scan (also seeds optional same-domain crawl).
• imageUrls – direct image URLs, no page fetch.
• csvUrl / csvText – CSV, TSV, XLSX or public Google Sheets URL; csvPreset: shopify maps standard export columns.
• datasetId + datasetField – any Apify dataset column with URLs.
• sitemapUrls / domains – XML or gzip sitemaps; domains discover sitemaps from robots.txt then /sitemap.xml.
• sitemapDatasetId – rows from Sitemap URL Extractor (url field; skips changeStatus=removed and statusCode≥400).
• renderJs – Playwright render instead of HTTP fetch; bills page-rendered instead of page-scanned. Images/fonts/media blocked in the browser; only DOM needed.
• onlyNew + stateKey – named KV store image-downloader-state holds ETag/Last-Modified/sha256 for conditional requests.

Recommended memory: 1024 MB (default). Use 2048 MB or more for large shops, heavy renderJs, or high maxImages.

Social and marketplace platform URLs are rejected at input validation (use your own site, CSV, or direct image URLs).

Output

Dataset row (shortened example)

{
  "schemaVersion": "1.0",
  "imageUrl": "https://books.toscrape.com/media/cache/.../cover.jpg",
  "finalUrl": "https://books.toscrape.com/media/cache/.../cover.jpg",
  "pageUrls": ["https://books.toscrape.com/"],
  "firstPageUrl": "https://books.toscrape.com/",
  "foundIn": ["img"],
  "status": "ok",
  "skipReason": null,
  "format": "jpeg",
  "bytes": 8934,
  "width": 189,
  "height": 280,
  "alt": "A Light in the Attic",
  "sha256": "…",
  "phash": "…",
  "fileName": "books.toscrape.com/cover.jpg",
  "fileKey": "files/books.toscrape.com/cover.jpg",
  "fileUrl": "https://api.apify.com/v2/key-value-stores/…",
  "zipPart": 1,
  "missingAlt": false,
  "oversized": false,
  "legacyFormat": false,
  "webpSavingsEstimate": { "bytes": 4200, "percent": 47 },
  "exifFound": false,
  "gpsFound": false,
  "charged": true,
  "chargedEvents": { "image-downloaded": 1, "page-scanned": 0 }
}

Statuses

ok (downloaded) · listed (audit list) · unchanged (diff, HTTP 304) · skipped (free when skipReason is set) · failed · removed (diff: URL gone since last run). Each row has charged: true|false and optional skipReason.

Common free skipReason values: robots-blocked, blocked-403, rate-limited-429, not-found-404, not-an-image, duplicate-exact, duplicate-similar, filtered-format, filtered-size, icon-or-pixel, too-large, timeout, ssrf-blocked, platform-not-supported, limit-reached.

Key-value store

Key	Content
RUN_SUMMARY	Totals, byStatus, bySkipReason, chargedEvents, truncated, stoppedByChargeLimit
REPORT.json / REPORT.md	Audit and size summary
image-sitemap.xml	Generated image sitemap from successful rows
images-part-N.zip	ZIP parts when output is zip or both
files/…	Individual images and derivatives

Diff state (not in the default store): named store image-downloader-state, key from stateKey.

Pricing

Pay per event (USD). No start fee. Recommended minimum Max total charge per run: $0.02.

Event	When	FREE	BRONZE	SILVER	GOLD
image-downloaded (primary)	Image validated, measured and stored (or fully measured in audit mode)	0.0020	0.0017	0.0014	0.0010
page-scanned	HTML page fetched and parsed (sitemap pages too)	0.0010	0.0009	0.0008	0.0006
page-rendered	Same as page-scanned when renderJs is on	0.0050	0.0045	0.0040	0.0030
image-listed	Audit list only, or diff unchanged	0.0005	0.00045	0.00040	0.00030
image-processed	Each derivative (convert, resize, thumbnail)	0.0006	0.0005	0.0004	0.0003
large-image-mb	Each started MB after the first MB of a stored image	0.0009	0.0008	0.0007	0.0006
zip-100mb	Each started 100 MB of ZIP output	0.010	0.009	0.008	0.0075

Per 1,000 (FREE → GOLD): image-downloaded $2.00 → $1.00 · page-scanned $1.00 → $0.60 · page-rendered $5.00 → $3.00 · image-listed $0.50 → $0.30 · image-processed $0.60 → $0.30. Platinum and Diamond pay the Gold price.

Bronze example: 500 images from one sitemap, one page each, no processing: 500 × 0.0017 + 500 × 0.0009 = $1.30.

What is free

You do not pay for:

• Failed fetches (timeout, HTTP errors, 403/429, redirect limit).
• robots.txt disallowed URLs (pages and images).
• Responses that are not images (HTML under an image URL, empty or truncated bodies).
• Exact or perceptual duplicates (the larger variant is kept).
• Filtered-out formats, sizes or URL patterns; icons and tracking pixels.
• Pages that fail or return no discoverable images (no page event).
• Skipped rows stopped by your max-image or charge limits (still listed with skipReason).

Every skipped row remains in the dataset with charged: false and a skipReason where applicable.

Fair use & privacy

Use responsibly. Images on the web are usually protected by copyright and may show people. Image Downloader only fetches publicly accessible files from URLs you provide, respects robots.txt, and does not log in or bypass any protection. You are responsible for having the rights or a legal basis (for example your own website, a licence, or a permitted exception) for how you use the downloaded files. Location, author and device data in EXIF are always removed.

• Robots & rate limits – UrlwrightBot rules apply to pages and image hosts; crawl-delay is honored (capped at 10 s). If robots.txt is missing (404), fetches proceed; server errors or timeouts on robots.txt are treated as disallow-all for that origin.
• SSRF – Hostnames are resolved before connect; private, link-local and metadata IPs are blocked, including after redirects.
• No platform scraping – Instagram, marketplaces, Google Images and similar inputs are rejected.
• Dataset – Alt text and titles from the source page are stored; GPS coordinates are never written. gpsFound is true/false only.
• Cookies – The actor does not send session cookies or log in.

More tools: urlwright.com · Bot details: urlwright.com/bot

Limitations

• Passive only – No login, cookies, captcha bypass or proxy rotation. Strict bot protection may yield free blocked-403 rows.
• Politeness – One request per second per host limits speed on single-origin bulk jobs; use direct imageUrls or CSV when you already have URLs.
• JavaScript – Without renderJs, client-rendered image URLs may be missing. Rendering costs more (page-rendered) but still blocks image bytes in the browser.
• Size – Default max file size 50 MB; configurable via maxFileMB. HTML pages are read up to 5 MB.
• Similar-image dedupe – dHash compares appearance; near-duplicates below your threshold may still appear twice.

FAQ

Why is my row skipped with robots-blocked?

The image or page URL is disallowed for UrlwrightBot in robots.txt (or robots could not be fetched safely). The row is free. Fix the rule on your site or provide direct image URLs you are allowed to fetch.

How do I migrate a Shopify catalog?

Export products to CSV, set csvUrl and csvPreset: "shopify", tune fileNameTemplate (e.g. {handle}/{sku}-{position}), and raise maxImages. Variant and product image columns are merged automatically. {pathSlug} is the image URL’s file name (without extension), not the page slug — use {pageSlug} for the page.

What does audit mode cost?

Each listed image bills image-listed (not image-downloaded). With auditDimensions: true, a small byte range is fetched to read dimensions; storage events still apply only when files are written.

How does onlyNew work?

State is stored under stateKey in the image-downloader-state store. Unchanged images respond with HTTP 304 and status unchanged (image-listed). New or changed bytes bill image-downloaded again. URLs that disappear from the crawl get status: removed (free).

Can I chain Sitemap URL Extractor?

Yes. Run Sitemap URL Extractor, pass sitemapDatasetId, optionally filter with includePathGlobs, then scan up to maxPages HTML pages for embedded images (plus image sitemap URLs when present).

Does rendering download every asset?

No. Playwright blocks image, font and media requests; only HTML/JS needed for the DOM is loaded. You still pay page-rendered per successful page with at least one discovered image.

Where are ZIP files?

In the run key-value store as images-part-1.zip, … Split size defaults to 1024 MB (zipPartMB). Billing uses zip-100mb on the compressed part size after the run writes each part.

Are EXIF and GPS kept?

No. Author, GPS, serial numbers and similar metadata are stripped on every stored image. Audit fields may report exifFound / gpsFound before strip; coordinates are never exported.