Common Crawl Scraper

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

Storeleads Scraper

saswave/storeleads-scraper

Storeleads scraper. Extract companies results from a search url. Get data about social network, name, monthly sales, visits, location, product sold, employees, technologies, trustpilot reviews, follower number, keywords, description and more

Extract tech stack data from any list of websites using BuiltWith. Fast, simple, and exportable.

Shopify Store Leads Scraper - Emails & Agency Prospects

wishful_knowledge/shopify-lead-finder

Find contactable Shopify store leads with public business emails, social profiles, product samples, tech stack gaps, agency-fit scores, and outreach offers.

Spotify Play Count & Insights Scraper

brilliant_gum/spotify-insights-scraper

Exact stream counts, monthly listeners, world rank, full discography PLUS historical trend tracking, related artists, playlists, contact info (emails/socials), revenue estimates, lyrics and podcasts. Drop-in superset of beatanalytics/spotify-play-count-scraper.

Docs-to-RAG Crawler

automation-lab/docs-rag-crawler

Crawl documentation sites (ReadTheDocs, GitBook, Docusaurus, Mintlify) into RAG-ready Markdown/JSON chunks with stable chunk IDs, heading breadcrumbs, word counts, and token estimates.

Extract emails, phone numbers and social links from any list of websites — with contact-page crawling and email validation. Bulk B2B lead enrichment, pay per result.

Open RFPs/tenders across hundreds of US/Canada local government agencies on Bonfire (Euna).

Sitemap Finder & URL Extractor · Crawl Any XML Sitemap

corent1robert/sitemap-detector

Find and crawl XML sitemaps from any website. Follows sitemap indexes, handles gzip, and exports every page URL with source file and lastmod into a clean dataset. No config needed.