AI Crawler Checker: robots.txt & llms.txt Audit

Find out whether AI answer engines can actually see your site — and fix what is blocking them. This Actor audits any URL for classic SEO health and for AI/LLM discoverability in a single pass: which AI crawlers your robots.txt allows, whether you publish an llms.txt, whether your content survives without JavaScript, and whether your structured data gives models something to cite. You get one scored row per page, with a concrete fix attached to every issue.

Most SEO auditors were designed for the ten-blue-links era. They will happily give you a green score while GPTBot, ClaudeBot and PerplexityBot are all locked out of your site.

Why use SEO & AI Visibility Auditor?

Search traffic increasingly arrives through answers, not links. Whether you appear in those answers depends on things a traditional audit never looks at:

• AI crawler access. Blocking Google-Extended removes you from Gemini's grounding. Blocking OAI-SearchBot removes you from ChatGPT Search. These are separate agents from Googlebot, and blocking them is usually an accident — a copied robots.txt snippet nobody revisited.
• Server-rendered content. Most AI crawlers do not execute JavaScript. A page that renders beautifully in your browser can be an empty shell to them. This Actor reports exactly how much text exists without JS.
• Structured data. Schema.org markup is how a model knows what your page is. Missing or malformed JSON-LD is silently ignored.
• The classic basics still matter — titles, descriptions, canonicals, headings, alt text, Open Graph — so they are all in the same report.

Try it on a publisher and the difference is immediate. nytimes.com blocks 21 of the 26 AI crawlers this Actor checks. wikipedia.org blocks none. theatlantic.com allows GPTBot while blocking ClaudeBot — a licensing decision, visible in one line of output.

How to use SEO & AI Visibility Auditor

1. Paste the URLs you care about into URLs to audit — your home page, top landing pages, best blog posts.
2. Leave everything else at its defaults for a first run.
3. Click Start. A handful of pages takes seconds.
4. Open the Scores tab for the summary, then AI visibility and Issues for the detail.
5. Work down the issue list. Every issue carries a fix field telling you what to change.

To sweep a whole site instead of specific pages, switch on Also follow internal links and raise Max pages.

Input

Field	Type	Default	What it does
startUrls	array	—	Pages to audit. One report row per URL. Required.
crawlSite	boolean	false	Follow internal links and audit those too, same domain only.
maxPages	integer	10	Hard cap on pages audited. Stops an accidental full-site crawl.
checkBrokenLinks	boolean	false	HEAD-request every link on the page and report failures.
maxLinksToCheck	integer	50	Per-page cap when link checking is on.
proxyConfiguration	object	off	Use a proxy if the target blocks datacenter traffic.

{
    "startUrls": [{ "url": "https://example.com" }],
    "crawlSite": false,
    "maxPages": 10,
    "checkBrokenLinks": false
}

Output

One dataset item per audited page. Download as JSON, CSV, Excel or HTML, or pull it from the API.

{
    "url": "https://example.com/",
    "statusCode": 200,
    "scoreTotal": 48,
    "scoreSeo": 72,
    "scoreAiAccess": 50,
    "scoreStructuredData": 20,
    "scoreSocial": 10,
    "title": "Example Domain",
    "titleLength": 14,
    "metaDescription": "",
    "h1Count": 1,
    "jsonLdTypes": [],
    "robotsTxtFound": false,
    "llmsTxtFound": false,
    "sitemapFound": false,
    "aiCrawlersAllowedCount": 26,
    "aiCrawlersBlocked": [],
    "mainTextChars": 125,
    "textToHtmlRatio": 0.224,
    "issueCount": 15,
    "issues": [
        {
            "severity": "warning",
            "area": "seo",
            "message": "No meta description.",
            "fix": "Add a 120-160 character summary. Answer engines often quote it verbatim."
        }
    ]
}

What it checks

Area	Fields
AI access	Per-bot allow/deny for 26 AI crawlers, matched robots.txt rule, llms.txt presence and size, sitemap discovery, text available without JavaScript, text-to-HTML ratio
SEO	Title and length, meta description and length, canonical (and whether it is self-referential), meta robots, X-Robots-Tag, lang, charset, viewport, favicon
Structure	H1/H2 counts, heading level skips, image count, images missing alt, internal/external/nofollow link counts, optional broken-link report
Structured data	JSON-LD block count, every @type found (including inside @graph), JSON parse errors, microdata types, RDFa presence, and flags for Organization, Article, FAQPage, HowTo, Product and BreadcrumbList
Social	og:title, og:description, og:image, og:type, twitter:card
Scoring	Four 0-100 sub-scores plus a weighted total, and a severity-tagged issue list with a fix for each

AI crawlers covered

OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended, Googlebot), Microsoft (Bingbot), Apple (Applebot, Applebot-Extended), Meta (meta-externalagent), Amazon (Amazonbot), ByteDance (Bytespider), Common Crawl (CCBot), plus Cohere, Mistral, You.com, Diffbot, Webz.io, ImageSift, AI2 and Timpi.

Each one is reported with its vendor, what blocking it costs you, and the exact robots.txt line that decided the verdict.

Cost estimation

Auditing is cheap because it is plain HTTP — no browser. A single page is one request plus up to three small site-level files (robots.txt, llms.txt, sitemap.xml), and those are fetched once per domain, not once per page. Ten pages on one domain typically finish in seconds on the smallest memory setting.

Turning on Check links for 404s is the one thing that meaningfully increases run time, since it adds up to maxLinksToCheck extra requests per page. Leave it off for routine audits.

Tips

• Audit templates, not just pages. One product page, one blog post, one category page tells you more than ten pages of the same type.
• Check scoreAiAccess before anything else. If AI crawlers are blocked, no amount of on-page work will get you cited.
• Watch mainTextChars. Under a few hundred characters on a content page means your text is being injected by JavaScript, and most AI crawlers will never see it.
• Re-run after deploys and schedule it. Crawler policy is the kind of thing that regresses silently when someone edits robots.txt.

FAQ

Does this modify my site? No. Every request is a read.

Does it respect robots.txt? Yes. It would be strange for a tool that reports on crawler policy to ignore it. If your own robots.txt blocks a URL you asked to audit, that page is skipped — which is itself a useful finding.

Can I audit a competitor? You can audit any publicly reachable page, and the AI-access picture is often the interesting part. Only public data is read.

Why does my JavaScript site score badly on AI access? Because that is the honest answer for a non-JS crawler. Server-side rendering or prerendering fixes it.

Is llms.txt actually used? Adoption is early and no major model vendor has committed to it. It is scored as a small bonus, not a requirement — it costs nothing to publish and may pay off later.

Support

Found a bug or want a check added? Open an issue on the Issues tab of this Actor. Requests for additional crawlers or extra checks are welcome.