You can access the Common Crawl Corpus Builder programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "buyerQuery": "buyer-ready records with source evidence",
10    "taskIntent": "buyer-ready-product-run",
11    "sourceMode": "sample",
12    "outputMode": "sample-records",
13    "startUrls": [],
14    "maxItems": 1,
15    "perSourceLimit": 25,
16    "maxTotalChargeUsd": 5,
17    "overallTimeoutSecs": 900,
18    "requestTimeoutSecs": 30,
19    "maxRequestRetries": 2,
20    "sinceLastRun": False,
21    "deltaMode": True,
22}
23
24# Run the Actor and wait for it to finish
25run = client.actor("zentrafoundry/common-crawl-targeted-corpus-builder").call(run_input=run_input)
26
27# Fetch and print Actor results from the run's dataset (if there are any)
28print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
29for item in client.dataset(run["defaultDatasetId"]).iterate_items():
30    print(item)
31
32# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Build Targeted Web Corpora from Common Crawl for RAG and Res API in Python

The Apify API client for Python is the official library that allows you to use Common Crawl Corpus Builder API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Common Crawl Corpus Builder API in JavaScript

Common Crawl Corpus Builder API through CLI

Common Crawl Corpus Builder OpenAPI definition

Common Crawl Corpus Builder API

Common Crawl Indexes Scraper

benthepythondev/common-crawl-indexes-scraper

Collect Common Crawl records and export id, name, to, from, cdx api, timegate as structured JSON, CSV or Excel data.

Ben

Research Corpus & Citation Graph Builder

zentrafoundry/openalex-research-graph-builder

Build research corpora and citation graph datasets from public metadata APIs.

Zentra

Common Crawl Scraper

crawlerbros/common-crawl-scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Crawler Bros

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

ParseForge

Actor Builder

handleco-app/actor-builder

handleco-app

Website Content Extractor

glowing_glove/website-content-extractor

Crawl public pages and extract page titles, meta descriptions, headings, readable text, source URLs, and crawl metadata.

Ushba Khan

Crawl4ai

kael_odin/crawl4ai

Extract page content (markdown/HTML/text), metadata, and link stats. Uses crawl4ai.

Kael Odin

Prophed Common Markets

prophed/prophed-common-markets

Fetch cross-platform prediction-market matches from Prophed Common Markets and output normalized arbitrage, liquidity, spread, and platform-comparison rows.

Prophed Com

RAG Corpus Quality Auditor

civicdataworks/rag-corpus-quality-auditor

Score web pages for RAG corpus quality: text length, heading structure, boilerplate hints, token estimate, metadata, and warnings.

Rowan Mercer

YATCO Builder Search Scraper - Cheap 🏗️🔍

scrapestorm/yatco-builder-search-scraper---cheap

🔍 Scrape Mass / Bulk YATCO Yacht Builders Enter your builder search URL to collect yacht builder listings at scale from YATCO including builder name, location, year established, construction type & builder profile URL 🏗️🛠️ Perfect for marine industry research & yachting supply chain analysis 📊