Pricing

from $8.25 / 1,000 items

Try for free

Go to Apify Store

Common Crawl URL Index Lookup Scraper

Try for free

Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Supports filters for date, status, and MIME type.

Pricing

from $8.25 / 1,000 items

Rating

0.0

(0)

Developer

ParseForge

Actor stats

Bookmarked

Total users

Monthly active users

0.5 hours

Issues response

16 days ago

Last modified

Categories

Developer tools

Automation

Other

You can access the Common Crawl URL Index Lookup Scraper programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "maxItems": 10,
12    "url": "apify.com",
13    "matchType": "domain"
14};
15
16// Run the Actor and wait for it to finish
17const run = await client.actor("parseforge/common-crawl-index-scraper").call(input);
18
19// Fetch and print Actor results from the run's dataset (if any)
20console.log('Results from dataset');
21console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
22const { items } = await client.dataset(run.defaultDatasetId).listItems();
23items.forEach((item) => {
24    console.dir(item);
25});
26
27// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Common Crawl Scraper - URL Index Lookup & Export API API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Common Crawl URL Index Lookup Scraper API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Common Crawl URL Index Lookup Scraper API in Python

Common Crawl URL Index Lookup Scraper API through CLI

Common Crawl URL Index Lookup Scraper OpenAPI definition

Common Crawl URL Index Lookup Scraper API

Common Crawl Scraper

crawlerbros/common-crawl-scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Crawler Bros

Common Crawl Corpus Builder

zentrafoundry/common-crawl-targeted-corpus-builder

Build targeted text corpora from Common Crawl with provenance.

Zentra

Common Crawl Indexes Scraper

benthepythondev/common-crawl-indexes-scraper

Collect Common Crawl records and export id, name, to, from, cdx api, timegate as structured JSON, CSV or Excel data.

Ben

Wayback Machine CDX Snapshot List Scraper

parseforge/wayback-cdx-scraper

Queries the Wayback Machine CDX index for a URL or domain and returns each snapshot as a flat row with timestamp, original URL, snapshot URL, HTTP status, MIME type, and content digest. Filter by date, status, and MIME type server-side.

ParseForge

RAG Web Browser Scraper

datapilot/rag-web-browser-scraper

RAG Web Browser Search & Crawl Actor uses to search Bing or crawl URLs, then extracts page content as clean markdown. It captures title, description, language, HTTP status, and structured metadata. Supports multiple queries, proxies, and outputs organized crawl + search results.

Data Pilot

Wayback Machine Snapshots — $0.20/1K

scrapesignal_labs/wayback-machine-snapshot-export

Export archived URL history from the Internet Archive CDX index with date, status, MIME type, digest, and replay links.

ScrapeSignal Labs

Website Content Extractor

glowing_glove/website-content-extractor

Crawl public pages and extract page titles, meta descriptions, headings, readable text, source URLs, and crawl metadata.

Ushba Khan

Wayback Machine Scraper — Archived Snapshots

hipersoft/wayback-machine-scraper

List every Internet Archive (Wayback Machine) snapshot of a URL or whole domain: timestamp, snapshot URL, status code, MIME type and content digest. Filter by date, status and dedupe. For SEO, OSINT and historical research. No setup.

hiper soft

Wayback Machine Snapshots Scraper — Internet Archive History

seemuapps/wayback-machine-snapshots-scraper

List every Internet Archive snapshot of a URL, page, or whole domain. Timestamp, snapshot URL, status code, mime type, content length. No login.

Andrew

Website URL Crawler & Link Extractor

maximedupre/website-url-crawler

Crawl public websites and collect URLs from rendered navigation and sitemaps. Export a link map with source pages, depth, anchor text, HTTP facts, crawl status, and available sitemap metadata.