Pricing

from $3.00 / 1,000 results

Common Crawl Scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Pricing

from $3.00 / 1,000 results

Rating

0.0

(0)

Developer

Crawler Bros

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

SEO tools

Developer tools

Automation

You can access the Common Crawl Scraper programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "mode": "urlCaptures",
12    "urlPattern": "en.wikipedia.org/wiki/*",
13    "crawl": "latest",
14    "matchType": "auto",
15    "maxItems": 15
16};
17
18// Run the Actor and wait for it to finish
19const run = await client.actor("crawlerbros/common-crawl-scraper").call(input);
20
21// Fetch and print Actor results from the run's dataset (if any)
22console.log('Results from dataset');
23console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
24const { items } = await client.dataset(run.defaultDatasetId).listItems();
25items.forEach((item) => {
26    console.dir(item);
27});
28
29// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Common Crawl Scraper API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Common Crawl Scraper API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Common Crawl Scraper API in Python

Common Crawl Scraper API through CLI

Common Crawl Scraper OpenAPI definition

Common Crawl Scraper API

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

ParseForge

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

The Howlers

Storeleads Scraper

saswave/storeleads-scraper

Storeleads scraper. Extract companies results from a search url. Get data about social network, name, monthly sales, visits, location, product sold, employees, technologies, trustpilot reviews, follower number, keywords, description and more

SASWAVE

654

5.0

(5)

[Aug25] BuiltWith Bulk Technology Scraper - fast & low cost

scrapegoats/builtwith-scraper

Extract tech stack data from any list of websites using BuiltWith. Fast, simple, and exportable.

ScrapeGoats

193

1.5

(2)

Shopify Store Leads Scraper - Emails & Agency Prospects

wishful_knowledge/shopify-lead-finder

Find contactable Shopify store leads with public business emails, social profiles, product samples, tech stack gaps, agency-fit scores, and outreach offers.

sanfeng zhang

Spotify Play Count & Insights Scraper

brilliant_gum/spotify-insights-scraper

Exact stream counts, monthly listeners, world rank, full discography PLUS historical trend tracking, related artists, playlists, contact info (emails/socials), revenue estimates, lyrics and podcasts. Drop-in superset of beatanalytics/spotify-play-count-scraper.

Yuliia Kulakova

Docs-to-RAG Crawler

automation-lab/docs-rag-crawler

Crawl documentation sites (ReadTheDocs, GitBook, Docusaurus, Mintlify) into RAG-ready Markdown/JSON chunks with stable chunk IDs, heading breadcrumbs, word counts, and token estimates.

Stas Persiianenko

Website Contact & Email Scraper

foo121/website-contact-scraper

Extract emails, phone numbers and social links from any list of websites — with contact-page crawling and email validation. Bulk B2B lead enrichment, pay per result.

ziv shay

Bonfire / Euna Procurement Opportunities Scraper

gallant_illustration/bonfire-procurement-scraper

Open RFPs/tenders across hundreds of US/Canada local government agencies on Bonfire (Euna).

Alexei Pannicov

Sitemap Finder & URL Extractor · Crawl Any XML Sitemap

corent1robert/sitemap-detector

Find and crawl XML sitemaps from any website. Follows sitemap indexes, handles gzip, and exports every page URL with source file and lastmod into a clean dataset. No config needed.