Pricing

Pay per event

AI Crawler Access & llms.txt Checker

See which AI crawlers a website allows or blocks. Evaluates robots.txt for 27 AI and search bots (training, AI search and assistant fetchers), checks llms.txt and llms-full.txt, content signals and noai meta tags, and summarizes each site's AI policy. Bulk, pay per site.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Ian McCann

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

Tim Zinin

5.0

(9)

AI Search Readiness Checker

epistemedeus/ai-search-readiness-checker

Audit public website signals that affect machine readability: HTML retrieval, metadata, JSON-LD, robots.txt, sitemap, and llms.txt. Get a disclosed technical score, evidence URLs, concrete fixes, and explicit limits. No ranking or citation claims.

Lucian Constantinescu

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

AI Website Spell Checker

louisdeconinck/ai-website-spell-checker

Discover and correct spelling errors across entire websites effortlessly with our AI-powered crawler. Start from any URL, and receive a comprehensive report detailing each error, suggested corrections, and severity levels. Enhance your site's professionalism and user experience today!

Louis Deconinck

5.0

(2)

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

AI Crawler Access Audit | Robots.txt & User-Agent Checks

a.klimm67/ai-crawler-reality-check

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Andrii Klymenko

AI Crawler Checker - robots.txt Checker for GPTBot & AI Bots

tidytools/ai-crawler-access-checker

Check if GPTBot, ClaudeBot, PerplexityBot and 26 more AI crawlers are blocked on your site: bulk robots.txt check of up to 10,000 domains, AI search score, fix snippet, Content Signals, change alerts. $2/1k sites.

Yukai Lin

Website Technology & Tracking Detector

vex_workflow/website-tech-stack-detector

Find out what any website runs on: CMS, e-commerce platform, analytics, ad pixels, marketing automation, chat, payments, reviews, consent tools, frameworks, hosting and CDN. Also extracts tracking IDs (GA4, GTM, Google Ads, Meta, TikTok and more). Bulk, fast, about 140 technologies.

Ian McCann

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Techionik

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.