Bulk Image Scraper — Real Dimension & File Size Filter

Crawl a website (or a list of URLs) and collect every image it links to — but unlike most bulk image scrapers, this Actor filters on the real, decoded pixel width/height and file size of each image, not on whatever the page's HTML claims. It also detects the true image format from its magic bytes (catches a .jpg URL that is secretly serving WebP) and deduplicates by content hash, so the same picture served through two different CDN URLs is only counted once.

Why this Actor is different

Most bulk image scrapers on the Store either crawl a site for image URLs (no download, no real dimensions) or download everything into a ZIP (no filtering, no crawling). Neither downloads an image just to verify its actual resolution or file size — they trust the width/height HTML attributes, which are frequently missing, wrong, or stale (lazy-loading placeholders, CDN-resized srcset variants, responsive images). This Actor combines site-crawling, optional full download, and a lightweight header-probe step (reads only the first bytes of an image, no full download needed just to learn its resolution/format) so you can reliably ask for "only images at least 1200px wide" or "only files under 500KB" and get results that are actually true.

When should an AI agent use this?

• "Find every product photo on this webshop category page that's at least 800x800 pixels."
• "Download all the hero/banner images from this website, deduplicated, into a key-value store."
• "List every image on this page along with its real resolution and file size."
• "Collect CSS background images from this landing page above 1200px wide."
• "Check whether any image on this page is actually a different format than its URL extension suggests."
• "Build an image dataset from this site, but skip anything smaller than 400x300 or under 10KB."

What this Actor does

1. Fetches each start URL and, if maxCrawlDepth > 0, follows same-host links up to that depth (breadth-first), respecting robots.txt for every internally discovered link.
2. Extracts image candidates from each page: <img src>, <picture><source srcset>, <img srcset> (responsive variants, optional), and CSS background-image: url(...) inline styles (optional) — resolving every relative/protocol-relative URL against the page it was found on.
3. Filters candidates by URL extension first (cheap), then probes the remaining header bytes of each image with a lightweight library to learn its real width, height, true format, and — when the server sends a Content-Length header — its exact file size, all without downloading the full image.
4. Applies your width/height/file-size range filters using those real, verified values (never the HTML attribute).
5. Downloads the full image bytes only when you set downloadFiles: true, or when a file-size filter is active and the server didn't send a reliable Content-Length — the common metadata-only case never needs a full download.
6. Computes a SHA-256 content hash whenever the full bytes were fetched, and deduplicates results by that hash across the whole run (catches the same image served from two different URLs), in addition to always deduplicating by exact URL.
7. Pushes one dataset record per surviving image, and — with downloadFiles: true — saves the actual file to the run's key-value store.

Input

Field	Type	Description
startUrls	array of strings	One or more pages to scan for images. Example: ["https://example.com/gallery"]
maxCrawlDepth	integer	Link-hops to follow from each start URL. 0 = only the start URL itself. Default 1.
sameHostOnly	boolean	Only follow links on the same hostname as the start URL. Default true.
imageExtensions	array of strings	Only collect images whose URL ends in one of these extensions. Default: all common formats.
minWidth / maxWidth	integer	Real decoded pixel width range. Optional.
minHeight / maxHeight	integer	Real decoded pixel height range. Optional.
minFileSizeKb / maxFileSizeKb	integer	Real file size range in kilobytes. Optional.
includeBackgroundImages	boolean	Also collect CSS background-image: url(...) references. Default false.
includeSrcset	boolean	Also collect every URL in a srcset attribute, not just the main src. Default true.
downloadFiles	boolean	Download the full image bytes into the key-value store, not just metadata. Default false.
maxImagesPerPage	integer	Safety cap per page. Default 200.
maxTotalImages	integer	Safety cap for the whole run. Default 1000.

Output

One dataset item per surviving image:

{
  "imageUrl": "https://example.com/photos/product-42.jpg",
  "sourcePageUrl": "https://example.com/catalog/widgets",
  "discoveryMethod": "img",
  "extension": "jpg",
  "detectedFormat": "jpg",
  "width": 1600,
  "height": 1200,
  "fileSizeBytes": 284113,
  "contentHash": "1f3a9c...e02b",
  "altText": "Blue widget, front view",
  "downloadedFileKey": null
}

• discoveryMethod is one of img, picture, srcset, css-background.
• detectedFormat comes from the image's actual header bytes and can differ from extension when a URL's extension doesn't match what the server actually serves.
• contentHash (SHA-256) is only populated when the full bytes were fetched (see step 5/6 above) — a pure metadata run without a file-size filter leaves it null and dedups by URL only.
• downloadedFileKey is the key-value store key for the saved file, only set when downloadFiles: true and the image passed every filter.

Use cases

• AI/ML teams curating an image training dataset filtered to a minimum real resolution.
• Content marketers collecting usable (large enough) product or stock photos from a site without pulling in thousands of tracking pixels and icons.
• Competitive analysis: collecting a competitor's product photography at scale for comparison.
• Deduplicating a gallery that serves the same image through several CDN URLs.

Pricing

This Actor uses Apify's Pay-Per-Event (PPE) pricing model.

• Actor Start: $0.00005 (Apify default)
• Image result: $0.003 per verified image record (real dimensions, format, hash)
• Image download: $0.01 per image, only charged in addition to the result event when downloadFiles: true and the file was actually saved

Legal

• This Actor only reads and, optionally, downloads publicly reachable image files and the HTML/CSS that references them — no login or paywalled content is accessed.
• robots.txt is respected for every internally discovered link while crawling (maxCrawlDepth > 0); URLs you supply directly in startUrls are always fetched, the same convention used by other scraping tools.
• This Actor does not verify copyright status or usage rights for any image it finds. Downloading or reusing images you don't own or have permission to use is your responsibility as the operator of this Actor, not the Actor's.
• No personal data is extracted deliberately. If an image happens to depict an identifiable person, that is a property of the source content, not something this Actor detects or filters.
• A 403 Forbidden response from a page or image is treated as a normal per-item failure and skipped — this Actor does not attempt to bypass bot protection.

FAQ

Q: Does this render JavaScript-generated image galleries? A: No — this is an HTTP-only crawler (no headless browser). Images that only appear after client-side JavaScript runs (e.g. a React/Vue gallery with no server-rendered <img> tags) will be missed. Most sites still serve <img>/<picture> tags directly in the HTML.

Q: Why is contentHash sometimes null? A: Computing a content hash requires the full image bytes. If you didn't set downloadFiles: true and no file-size filter forced a full download, this Actor only reads the image's header bytes (cheaper and faster), so no hash is available for that record. Deduplication still happens by exact URL in that case.

Q: How is this different from just checking the width/height HTML attributes? A: Those attributes are frequently absent, wrong, or describe a placeholder/lazy-loaded version of the image. This Actor probes the image's own header bytes to get the resolution the browser would actually render, regardless of what the HTML says.

Q: Can I get only the largest/best photo when a page has multiple srcset sizes of the same image? A: Set a minWidth/minHeight filter and this Actor will keep only the variants meeting it; you can also compare contentHash values (when downloading) to spot exact duplicates across variants that decode to the same underlying file.

Q: Will this download images I don't have rights to? A: This Actor collects/downloads whatever is publicly reachable on the pages you point it at — see Legal above. Always confirm you have the right to use any image before republishing it.

Related Actors

No other Actor in this portfolio currently covers generic website scraping — this is the first.

---

Keywords: bulk image scraper, image downloader, website image crawler, image dimension filter, real resolution filter, file size filter, magic bytes format detection, content hash deduplication, srcset scraper, CSS background image scraper, image dataset builder.

Keywords

image scraper, bulk image downloader, website image crawler, image dimensions filter, srcset, background image scraper, content hash dedup, magic bytes, image dataset, web crawler

Changelog

0.1.0

• Initial release: HTTP-only site crawl, real dimension/file-size filtering via header probing, magic-bytes format detection, content-hash dedup, optional full download to key-value store.