AI Crawler Policy Audit API | 🤖 $1/1K

Audits websites for AI search readiness — scores how likely pages are to be cited by ChatGPT, Perplexity, Gemini, Claude, Copilot, Grok, and Google AI Overviews / AI Mode across 6 categories: crawlability, entity clarity, content structure, schema markup, authority signals, AI-specific optimisation.

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

SEO, GEO & AEO Audit: AI Search Readiness Checker

fayoussef/seo-geo-aeo-audit

Audit any website for Google SEO and AI search visibility in one run. Get 0–100 SEO, GEO & AEO scores per page, AI-crawler & llms.txt checks, schema validation, and a prioritized fix list with a shareable HTML report. No API keys needed — just enter a URL.

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Company Career Page Jobs API | 💵$2/1K

webdata_labs/company-career-page-jobs-api

[💵 $2.00 / 1K] Turn company websites and career pages into normalized open-job data. Auto-discovers Workday, Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee, or Personio boards.

France Notaires Real Estate Scraper API - Immobilier Notaires

webdata_labs/france-notaires-property-api

[💵 $1.5 / 1K] Annonces officielles des Notaires de France (immobilier.notaires.fr) en schema anglais propre: prix, honoraires, DPE, surface, geo et contact du notaire. Filter by departement, price, area; monitoring mode returns only new listings.

Stealth Website Crawler

nocturne/stealth-website-crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.