Pricing

$5.00 / 1,000 results

Try for free

Go to Apify Store

AI Crawler Access Checker

Try for free

Check if ChatGPT, Claude, Perplexity and Google AI can read your site: robots.txt rules, llms.txt, JSON-LD and server-rendered text, with a 0-100 score.

Pricing

$5.00 / 1,000 results

Rating

0.0

(0)

Developer

FluxLab

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

What it checks

robots.txt rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more
llms.txt presence
JSON-LD structured data types found on the home page
Server-rendered text: how much readable text is in the raw HTML (JavaScript-only sites are invisible to most AI crawlers)
Score 0-100 per domain

Input

A list of domains (example.com). Up to 100 per run by default.

Output (one record per domain)

domain, status, blocked_ai_bots, robots_txt_found, llms_txt_found, json_ld_types, visible_text_chars_in_raw_html, bad_certificate, score. Download as JSON, CSV or Excel, or call via API.

Use cases

GEO / AI-search audits for SEO agencies
Checking client sites before and after a robots.txt change
Monitoring a list of competitor or publisher domains

Pricing

Pay per result (one domain = one result). Only public files and the home page are requested, one request set per domain.

More tools by FluxLab

Built by FluxLab. Questions or a missing feature? Open an issue on the Actor page.

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

Tim Zinin

5.0

(6)

Web Accessibility Checker

louisdeconinck/web-accessibility-checker

Audit your entire website for WCAG 2.1 & 2.2 compliance using the industry-standard axe-core engine. Automatically crawl pages, detect accessibility violations, and generate detailed actionable reports to ensure legal compliance and improve SEO.

Louis Deconinck

AI Search Readiness Checker

epistemedeus/ai-search-readiness-checker

Audit public website signals that affect machine readability: HTML retrieval, metadata, JSON-LD, robots.txt, sitemap, and llms.txt. Get a disclosed technical score, evidence URLs, concrete fixes, and explicit limits. No ranking or citation claims.

Lucian Constantinescu

Google Index Checker API

tugelbay/google-serp-checker

Compare a page sitemap with observed first-page Google site: results. Export sampled links, sitemap matches and missing-from-sample candidates. This is not a complete indexation check. Guide: https://konabayev.com/tools/google-serp-checker/

Tugelbay Konabayev

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

HTTP Status Code Checker

onescales/simple-http-status-code-checker

Bulk check HTTP status codes (200, 301, 302, 402, 404, 500, and more) and redirect chains for any list of URLs. A must-have lookup tool for SEO audits, site migrations, and broken link detection

One Scales

1.7K

5.0

(2)

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Brand Visibility Tracker — ChatGPT, Gemini, Perplexity

inovaflow/ai-brand-monitoring

Monitor your brand visibility and brand mentions in AI search results — ChatGPT, Google AI Overviews, Gemini, Perplexity and Copilot. Track competitor mentions and share of voice, run it daily for continuous AEO/GEO optimization, and get the page that beat you plus how to win it back.