Pricing

from $28.00 / 1,000 domain auditeds

GEO & llms.txt Readiness Auditor

Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.

Pricing

from $28.00 / 1,000 domain auditeds

Rating

0.0

(0)

Developer

Geo

Actor stats

Bookmarked

Total users

Monthly active users

18 days ago

Last modified

GEO & llms.txt Readiness Auditor — AI Crawler Blocker Checker

Use the ai crawler blocker checker to audit which AI crawlers your site allows or blocks, validate your llms.txt, and score your generative-engine-optimization (GEO) readiness — with a prioritized fix list per domain.

Find out — per domain — whether GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot and 18+ other AI crawlers can reach and cite your content, whether your llms.txt is valid, and whether your pages carry the on-page signals AI answer engines need. You get a GEO-readiness verdict, a score, and an exact prioritized fix list — per domain. Non-scraper: it reads only public well-known files (robots.txt, llms.txt, sitemap.xml) and one page's <head>, one polite request each, honoring robots.txt.

What makes it different

Most checkers just say a token is "present/absent." This one tiers crawler compliance, so you learn the truth:

A Disallow: PerplexityBot or Bytespider in robots.txt won't actually stop them — those operators are documented ignoring robots.txt (only a WAF will) — so we flag "ineffective block."
It separates AI-search/citation bots (blocking them = you vanish from ChatGPT/Claude/Perplexity/Google AI answers — a P0 error) from training bots (blocking them is a legitimate choice) from opt-out tokens (Google-Extended, Applebot-Extended — directives, not crawlers).

Use cases

Audit your own sites/subdomains before/after a robots.txt or CDN change.
Agencies: batch-audit a client portfolio and hand over a fix list.
Catch the silent killer: a noindex, a JS-only page, or an accidental AI-search block that's costing you citations.

Input

domains (required, your sites) · auditPath (page to check, default /) · verifyToken (optional — put <meta name="geo-audit-verify" content="TOKEN"> on the page to mark ownership_verified) · timeoutSecs.

{ "domains": ["yourbrand.com", "docs.yourbrand.com"], "auditPath": "/" }

Output example

{
  "domain": "yourbrand.com",
  "geo_ready_score": 85,
  "geo_ready_verdict": "READY",
  "top_fixes": [],
  "llms_txt": { "found": true, "valid": true, "content_summary": "3 sections, 42 entries" },
  "search_bots_blocked": 0,
  "issues": { "count": 0, "items": [] }
}

A clean domain scores READY with zero fixes; a misconfigured one lists exact actions.

The 16 checks (per domain)

HTTPS/200 + redirect chain · AI-search crawlers not blocked (P0) · ineffective-block flag (gray-zone/non-compliant crawlers) · training opt-out tokens · sitemap present/valid/referenced · content in raw HTML (no-JS, P0) · noindex/nosnippet (P0) · AI opt-out meta (noai) · schema.org JSON-LD · canonical · title + meta description · Open Graph · semantic structure (single H1, <main>/<article>) · freshness dates · llms.txt presence + validity · llms-full.txt. → weighted score + READY / NEEDS_WORK / NOT_READY verdict + top_fixes.

Pricing (pay-per-event) & cost example

Event	Price	When
`domain-audited` (primary)	$0.04	per domain fully audited
`issue-found`	$0.01	per fix-worthy issue flagged (capped at 8/domain)

A clean domain costs $0.04; a domain with 4 issues, $0.08; a badly-misconfigured one, at most $0.12. Measured platform compute is <$0.001/domain (<3% of revenue). Compare: brand-tracker SaaS runs $25–99/mo per domain; this is per-audit and pennies.

Integrate

Schedule it (weekly), then use Apify webhooks/API/MCP/Make/n8n to alert on any domain whose geo_ready_verdict drops or whose search_bots_blocked > 0.

HTTP Security Headers & OWASP Posture Auditor — audit your CSP, HSTS, XFO, and 20+ security headers alongside your GEO score.
Email Deliverability & Domain Posture Auditor — check SPF, DKIM, DMARC, BIMI, and MTA-STS across your domains.

FAQ

What is an AI crawler blocker checker and why do I need one? An AI crawler blocker checker audits your site's robots.txt to see whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are allowed to reach your content. If an AI-search bot is blocked, your site won't appear in ChatGPT, Claude, Perplexity, or Google AI answers — even if your page ranks #1 in traditional search. This tool detects those blocks and flags ineffective ones that won't actually stop non-compliant crawlers.

Does this tool scrape or save any data from my site? No. The auditor makes one polite GET request per well-known file and one per page, honors robots.txt, and never stores raw page content. It only retains the audit results (scores, verdicts, fix lists).

How is GEO readiness different from traditional SEO? GEO (Generative Engine Optimization) focuses on making your content discoverable and citable by AI answer engines (ChatGPT, Claude, Perplexity, Google AI Overviews). Traditional SEO optimizes for search-engine rankings; GEO adds llms.txt, AI-opt-out meta, semantic structure, and crawler-access compliance.

Can I audit domains I don't own? The tool is designed for domains you own or are authorized to audit. It reads only public files with single polite requests and respects robots.txt, but ownership verification is recommended via the optional verifyToken parameter.

What about other AI crawler checker tools? Most alternatives only tell you "this token is present/absent." This one tiers crawler compliance, flags ineffective blocks (crawlers that ignore robots.txt), validates llms.txt, and runs 16 structural checks — giving you both a score and an actionable fix list.

Limitations

Audits one page per domain (the root or your auditPath) — not a full crawl.
robots.txt matching uses the standard longest-prefix rule; unusual wildcard patterns may under-report.
Intended for domains you own or are authorized to audit; it reads public files with a single polite request and honors robots.txt. Compliance tiering reflects publicly documented crawler behavior and can change.

Found an issue or have a feature request? Open an issue on the GitHub Issues tab — we respond promptly.

AI Crawler Access Checker — robots.txt & llms.txt Audit

zinin/ai-crawler-access-checker

Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended & more) can access your website. Bulk audit of robots.txt rules, llms.txt presence and sitemap for AI search visibility (GEO/AEO).

Tim Zinin

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

LLMs.txt Checker — AI Readiness Monitor

q_services/llms-txt-checker

Check websites for llms.txt and llms-full.txt, extract sections and links for AI readiness and GEO audits.

Q Services

AI Crawler Policy Audit API | 🤖 $1/1K

webdata_labs/ai-crawler-policy-audit-api

[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.

WebData Labs

LLMs.txt Discovery Auditor

phoenix2810/llms-txt-discovery-auditor

Audit a website for /llms.txt and /llms-full.txt AI discovery readiness. Get a readiness score, markdown structure summary, and actionable recommendations. Built for GEO/AI-SEO consultants, documentation teams, and devrel.

Sanskar Jaiswal

llms.txt Checker, Generator & Monitor: AI Search Readiness.

apifmcpfactory/llms-txt-suite

Generate, audit and monitor your llms.txt so ChatGPT and AI agents can read your site. Get an AI-readiness score and a ready-to-publish file. Works on public pages only. We store nothing personal.

Howth Technology Factory

llms.txt Generator: Make Any Website AI-Readable

apifmcpfactory/llms-txt-generator

Generates a spec-compliant llms.txt file for any website from the site's own pages and metadata (no AI invention) - the fix when an AI-readiness audit finds llms.txt missing; $0.02 per site.

Howth Technology Factory

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

Ken Yao

llms.txt + llms-full.txt Generator for Any Docs Site

ianymu/llms-txt-converter

llms-txt-converter is an Apify Actor that crawls any documentation site and outputs the two GEO-standard files (llms.txt and llms-full.txt) used by ChatGPT, Claude, Perplexity, Gemini, and Bing Copilot to find and cite your docs.