Pricing

from $8.25 / 1,000 items

Common Crawl URL Index Lookup Scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

Pricing

from $8.25 / 1,000 items

Rating

0.0

(0)

Developer

ParseForge

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Configure MCP server with Common Crawl Index Scraper · Web Archive URL Discovery

Get a ready-to-use configuration for your MCP client with the Common Crawl Index Scraper · Web Archive URL Discovery Actor preconfigured at mcp.apify.com?tools=parseforge/common-crawl-index-scraper .

You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Zealty Profile Scraper

getdataforme/zealty-profile-scraper

Extract comprehensive real estate data from Zealty listings with this powerful Apify Actor. It scrapes property details like pricing, specs, agent info, and media using fast, flexible crawling....

GetDataForMe

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

mikolabs

RAG Web Browser

tugelbay/rag-web-browser

RAG web browser for AI agents. Search Google, scrape top results, return clean Markdown. One API call replaces Tavily + scraper combo. 256MB memory, 10x lighter than alternatives. Works with LangChain, Claude MCP, OpenAI.

Tugelbay Konabayev

SEC Form ADV Scraper - RIA Firm & Investment Advisor Data

jungle_synthesizer/sec-form-adv-iapd-crawler

Extract SEC-registered RIA firm records from IAPD by CRD, firm name, state, or full sweep. Includes identity, SEC file number, registrations, notice filings, brochure URLs, Form CRS URL, and office address.

BowTiedRaccoon

Google Index Checker API

tugelbay/google-serp-checker

Check Google indexation for a domain by comparing sitemap URLs against live Google site: search results. Find indexed, missing, and potentially deindexed pages, then export indexation reports via Apify API, datasets, webhooks, CSV, or Excel.

Tugelbay Konabayev

Docs-to-RAG Crawler

automation-lab/docs-rag-crawler

Crawl documentation sites (ReadTheDocs, GitBook, Docusaurus, Mintlify) and output chunked markdown ready for RAG/vector DB ingestion. Splits by heading hierarchy, strips nav/sidebar chrome.

Stas Persiianenko

Shine Jobs Search Scraper

stealth_mode/shine-jobs-search-scraper

Efficiently scrape job listings from NaukriGulf.com, the leading job portal for Middle East careers. Extract comprehensive data including job descriptions, company details, salary info, and application methods. Perfect for recruitment agencies, career platforms, and Gulf market employment research.

Stealth mode

Brave Search API — Web, News, Image & Video Search

junipr/brave-search

Search the web using the Brave Search API. Extract organic results, news, videos, and related searches for any query. Fast, private, and independent search results without Google dependencies.