AI Crawler Access & AEO Audit

Can AI search and AI assistants read your pages, and if they can't, what exactly do you change?

This Actor fetches each URL as 14 real AI and search crawlers plus curl, and once as a normal browser from the same IP. It also checks robots.txt rules for 16 crawler tokens, including Google-Extended and Applebot-Extended. It compares the responses and tells you which crawler is blocked, which firewall or CDN did it, and what to paste to fix it. It also checks what helps a page get quoted in AI answers: robots.txt rules per crawler, structured data, answer-first content, freshness dates, sitemaps, hreflang and llms.txt.

You get findings with evidence and ready-to-paste fixes: a robots.txt diff, a Cloudflare rule, JSON-LD blocks, a draft llms.txt. There is no vague 0–100 score.

What it checks

Area	What you learn
Crawler access	For each of GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, CCBot, Bytespider, Meta-ExternalAgent and curl, it shows whether the crawler gets the same page as a browser, or a 403, a challenge page, a stub page or a redirect. The firewall is identified: Cloudflare, Akamai, Vercel, Sucuri, Imperva or AWS WAF.
robots.txt	Rules per crawler under RFC 9309, including Google-Extended and Applebot-Extended. It flags conflicts between robots.txt and what the server actually does, and gives a diff that fixes them. It also notes whether Content-Signal is present.
Content	Whether there is a 40–60 word answer under the H1, whether headings are phrased as questions, whether the page links to outside sources, whether the content only appears after JavaScript runs, and whether the email address is hidden by obfuscation.
Structured data	Article author and dates, SoftwareApplication offers, an identifiable publisher, sameAs, FAQPage. It also catches broken JSON-LD.
Freshness & discovery	Whether dateModified matches the sitemap lastmod, whether the page is in the sitemap, whether hreflang links are reciprocal and canonical-consistent, and whether llms.txt exists (a draft is generated if not).

Training vs. search crawlers

Blocking a training crawler (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Google-Extended, Applebot-Extended) is a policy choice. It is reported as P3 with a note, not as an error. Blocking a search or user-fetch crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and others) means AI answers can't cite you, so it is P1.

What the access test can and cannot prove

The requests come from Apify's servers, not from each vendor's verified IP ranges. A result shows how your firewall treats that user agent from an unverified IP. Firewalls that verify crawlers by IP may treat the real crawler differently. Every block finding says this. If even the browser request is blocked, you get one "inconclusive" finding instead of 15 false alarms.

Input

Field	Description
urls	Page URLs to audit. A missing https:// is added.
sitemapUrl	Optional. Audits the pages in this sitemap or sitemap index (.xml.gz works).
maxUrls	The most pages per run. Default 100, maximum 5000.
language	The report language, one of 30 (en, de, es, fr, ja, zh-Hans, ru, …).
bots	Optional subset of the crawlers.
includeMarkdown	Adds a readable Markdown report to each item.

If you give no URL and no sitemap, the Actor runs a free demo audit of https://example.com/, so you can see the output format.

{ "urls": ["https://example.com/pricing"], "sitemapUrl": "https://example.com/sitemap.xml", "maxUrls": 50, "language": "en" }

Output

Each audited URL produces one dataset item:

{
  "url": "https://example.com/guide/",
  "final_url": "https://example.com/guide/",
  "counts": { "P1": 1, "P2": 1, "P3": 2 },
  "findings": [
    {
      "id": "access.blocked", "priority": "P1", "bot": "OAI-SearchBot", "bot_class": "search",
      "title": "OAI-SearchBot is blocked",
      "detail": "A request with the OAI-SearchBot user agent got status 403 or a challenge page, while a browser got the page. …",
      "evidence": { "status": 403, "control_status": 200, "waf": "cloudflare" },
      "fixes": [ { "kind": "cloudflare", "title": "Let OAI-SearchBot through in Cloudflare", "body": "Cloudflare dashboard → Security → WAF → …" } ]
    }
  ],
  "passed": ["robots.ok", "schema.ok"],
  "billable": true
}

The key-value store gets a SUMMARY table with one row per URL.

Priorities: P1 is critical (AI answers can't read or cite the page), P2 is important, and P3 is nice to have or a policy choice.

Pricing

You pay per audited URL. You are not charged for URLs that could not be audited: the page did not respond, returned an error status, or blocked even a normal browser. Those items have "billable": false. If you set a spending limit on the run, the Actor stops cleanly when it is reached.

Also available

• An MCP server, so AI agents can call the same audit directly (coming soon).
• A free one-URL check at aiseewhy.com.

Made by AISeeWhy. Questions or false positives: open an issue on the Actor page.