Pricing

from $8.25 / 1,000 items

Common Crawl URL Index Lookup Scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

Pricing

from $8.25 / 1,000 items

Rating

0.0

(0)

Developer

ParseForge

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

Automation

Other

You can access the Common Crawl URL Index Lookup Scraper programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "maxItems": 10,
10    "url": "apify.com",
11    "matchType": "domain",
12}
13
14# Run the Actor and wait for it to finish
15run = client.actor("parseforge/common-crawl-index-scraper").call(run_input=run_input)
16
17# Fetch and print Actor results from the run's dataset (if there are any)
18print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
19for item in client.dataset(run["defaultDatasetId"]).iterate_items():
20    print(item)
21
22# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Common Crawl Index Scraper · Web Archive URL Discovery API in Python

The Apify API client for Python is the official library that allows you to use Common Crawl URL Index Lookup Scraper API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Common Crawl URL Index Lookup Scraper API in JavaScript

Common Crawl URL Index Lookup Scraper API through CLI

Common Crawl URL Index Lookup Scraper OpenAPI definition

Common Crawl URL Index Lookup Scraper API

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Zealty Profile Scraper

getdataforme/zealty-profile-scraper

Extract comprehensive real estate data from Zealty listings with this powerful Apify Actor. It scrapes property details like pricing, specs, agent info, and media using fast, flexible crawling....

GetDataForMe

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

mikolabs

RAG Web Browser

tugelbay/rag-web-browser

RAG web browser for AI agents. Search Google, scrape top results, return clean Markdown. One API call replaces Tavily + scraper combo. 256MB memory, 10x lighter than alternatives. Works with LangChain, Claude MCP, OpenAI.

Tugelbay Konabayev

SEC Form ADV Scraper - RIA Firm & Investment Advisor Data

jungle_synthesizer/sec-form-adv-iapd-crawler

Extract SEC-registered RIA firm records from IAPD by CRD, firm name, state, or full sweep. Includes identity, SEC file number, registrations, notice filings, brochure URLs, Form CRS URL, and office address.

BowTiedRaccoon

Google Index Checker API

tugelbay/google-serp-checker

Check Google indexation for a domain by comparing sitemap URLs against live Google site: search results. Find indexed, missing, and potentially deindexed pages, then export indexation reports via Apify API, datasets, webhooks, CSV, or Excel.

Tugelbay Konabayev

Docs-to-RAG Crawler

automation-lab/docs-rag-crawler

Crawl documentation sites (ReadTheDocs, GitBook, Docusaurus, Mintlify) and output chunked markdown ready for RAG/vector DB ingestion. Splits by heading hierarchy, strips nav/sidebar chrome.

Stas Persiianenko

Shine Jobs Search Scraper

stealth_mode/shine-jobs-search-scraper

Efficiently scrape job listings from NaukriGulf.com, the leading job portal for Middle East careers. Extract comprehensive data including job descriptions, company details, salary info, and application methods. Perfect for recruitment agencies, career platforms, and Gulf market employment research.

Stealth mode

Brave Search API — Web, News, Image & Video Search

junipr/brave-search

Search the web using the Brave Search API. Extract organic results, news, videos, and related searches for any query. Fast, private, and independent search results without Google dependencies.