Common Crawl Indexes Scraper

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

Ahrefs Free Tools Scraper

parseforge/ahrefs-tools-scraper

Pull SEO metrics from Ahrefs free tools: Domain Rating, backlinks, referring domains, organic traffic, keyword difficulty, and search volume. Submit a domain or keyword and get structured results. Export to JSON, CSV, or Excel for SEO research.

Wikipedia Pageviews Scraper

parseforge/wikipedia-pageviews-scraper

Pull Wikipedia pageview metrics for any article in any language edition. Daily or monthly granularity, filter by access type (desktop, mobile, app) and agent type (user, spider, automated). Pick a date range. Export to JSON, CSV, or Excel for SEO research and content benchmarking.

Common Crawl Scraper

crawlerbros/common-crawl-scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Shopify Store Finder — Target by Size & Tech Stack

cool_jacquard/shopify-store-enrichment

Find Shopify stores by installed apps (Klaviyo, Recharge, Gorgias, Yotpo, Loox), product count and country. Targeted B2B lead generation: enriched, revenue-qualified Shopify leads with email and socials — not a raw domain dump. Pay only for stores matching your filters.

Find companies using Greenhouse, Ashby, Workable, Recruitee or Lever (EU) and see who's hiring right now — live open-role counts across ~1,990 companies. Filter by keyword. $2 per 1,000 companies.

Common Crawl Corpus Builder

zentrafoundry/common-crawl-targeted-corpus-builder

Build targeted text corpora from Common Crawl with provenance.

Get every known URL of any domain from Wayback Machine, Common Crawl, AlienVault OTX, URLScan, crt.sh and sitemap — merged & deduped. Block-proof, no API key.

Technical SEO Audit & Website Crawler - Core Web Vitals

groupoject/technical-seo-audit-crawler

Crawl websites and detect technical SEO issues across metadata, indexability, headings, content, links, images, structured data, security headers, and optional PageSpeed Core Web Vitals.

Web Images Scraper

maximedupre/web-images-scraper

Extract image URLs from public webpages, domains, and direct image links. Get source pages, discovery methods, metadata, and optional saved files or ZIP archives.