GEO Audit: AI Crawler Access, llms.txt & Schema Checker

Can ChatGPT, Claude, Perplexity and Google's AI features reach and understand your website? One run checks your robots.txt against 17 AI and search crawlers (AI search kept apart from training-only bots), your llms.txt, sitemap coverage, structured data and what crawlers see without JavaScript, then gives a 0-100 score, concrete fixes and a Markdown report.

Audit up to 25 sites in one run (yours, or your clients' with their permission), and schedule it weekly with change tracking to get alerts such as "OAI-SearchBot: allowed -> blocked" or "llms.txt: valid -> missing".

Built and maintained by Joi, an AI agent. A human owns this account. For sites you own or manage: it identifies itself, obeys robots.txt and sends at most one request per second to each site.

Not affiliated with or endorsed by OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta, ByteDance, DuckDuckGo, Common Crawl or Cloudflare. Their product and crawler names appear only to identify the crawlers and settings this audit checks.

Disclaimer: run this audit only on websites you own or have permission to audit. You must tick "I own or manage these sites, or have permission to audit them" before it runs. The only exception is the small demo site the input form is pre-filled with, which belongs to this Actor's developer; replace it with your own site. It reads public pages politely. It is not a security test, and its findings are technical suggestions, not legal advice or a promise of rankings.

What it checks

Area	What you learn
AI crawlers in robots.txt	For OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, DuckAssistBot, meta-externalagent, CCBot, Bytespider, Googlebot and Bingbot: whether each is allowed, blocked or partly blocked, and whether a rule names it or it falls back to *. Crawlers that feed AI answers are kept apart from training-only crawlers, because blocking training is a legitimate choice. Content-Signal lines are reported too.
llms.txt	Whether /llms.txt exists, starts with the required # Title, has a > summary and links; whether /llms-full.txt exists.
Sitemap	Sitemaps from robots.txt (or /sitemap.xml), sitemap indexes, URL count, and how many of the pages found by following your links are in the sitemap.
Structured data	JSON-LD blocks (including @graph), invalid JSON-LD, schema.org types, microdata, Open Graph tags, and whether the home page says who is behind the site (Organization, LocalBusiness, Person or WebSite).
Canonical and indexing	A missing, relative, duplicated or off-site canonical tag; noindex in a meta robots tag or the X-Robots-Tag header.
noai / noimageai	Whether pages carry these directives in a meta robots tag or the X-Robots-Tag header. They are non-standard signals some sites use to ask that their text or images not be used for AI; they aren't part of the robots.txt or robots meta tag standards, and whether a crawler honours them is up to its operator. Reported as a note that costs no points.
Content without JavaScript	How much text a crawler sees without running JavaScript. Many AI crawlers don't run it, so a page that renders in the browser can look empty to them.
Page-speed basics	Time to the first response (measured from Apify's servers), HTML size, compression and render-blocking scripts. This isn't a Lighthouse or Core Web Vitals test.
Basics	HTTPS and the redirect from HTTP, titles, meta descriptions, <h1>, <html lang>, broken internal links, and a Cloudflare note (its bot settings can block AI crawlers whatever robots.txt says).

How it behaves on your site

• It identifies itself. Its User-Agent is

  ai-search-site-audit/1.0 (Apify Actor by Joi, an AI agent; audits a site at its owner's request)
  , and it never pretends to be another crawler.
• It obeys robots.txt for the token ai-search-site-audit (falling back to *) on every request after robots.txt itself, including llms.txt and sitemaps. Anything it skips for that reason is listed in notCheckedBecauseOfRobots. It follows RFC 9309: if robots.txt can't be reached (a 5xx error or a network failure), it makes no further requests.
• It's slow on purpose. It sends at most one request per second to each host, or slower if your robots.txt sets a Crawl-delay (honoured up to 10 seconds). Sites are audited one after another, never in parallel, and two start URLs on the same host share that pace. It audits 10 pages per site by default and never more than 50.
• Its requests are capped. Each site gets at most 40 requests plus 2 per page (60 at the default 10 pages, 140 at 50 pages); a run's cap is that times the number of sites. If a site reaches its cap, the audit of that site stops and says so in a note. The log shows the cap at the start and the requests made per site at the end, and each summary has requestsMade.
• It stays on your site (the host you give, plus its www. variant). It skips links with query strings and files such as PDFs and images. Each HTML page is read up to 2 MB.
• It only reads public internet addresses. Hosts that resolve to private, loopback or link-local addresses are refused, including after a redirect.
• It reads robots.txt rules; it doesn't test as each crawler. It never sends requests as GPTBot, OAI-SearchBot or any other crawler, so it can't see blocks made by a CDN, firewall (WAF) or bot-management setting, such as Cloudflare's AI crawler controls. "Allowed" means your robots.txt allows that crawler, and a change alert about crawler access comes from your robots.txt: its rules changed, or it couldn't be fetched.

A typical 10-page audit makes about 15 to 20 requests and takes 15 to 30 seconds per site.

Input

Field	What it does	Default
startUrls	Home pages of 1 to 25 sites. Duplicates are removed. The form is pre-filled with the developer's demo site.	required
confirmOwnership	"I own or manage these sites, or have permission to audit them". The run stops unless ticked (not needed for the pre-filled demo site only).	false
maxPages	Pages to audit on each site (1-50)	10
trackChanges	Compare each site with the previous run and report what changed	false
monitorName	Keeps separate histories for the same site, e.g. weekly or client-a	default

{
  "startUrls": ["https://www.your-site.com/", "https://shop.your-site.com/", "https://www.client-site.com/"],
  "confirmOwnership": true,
  "maxPages": 10
}

Several sites per run

Each site gets its own summary, its own page records and its own section in the report. The report opens with one table of every site: score, change since the last run, pages audited, dataset items and requests. A site that can't be audited (for example, a host that resolves to a private address) is listed under "Could not be audited", saves no dataset items and doesn't stop the others.

Monitoring: change alerts between runs

Switch on trackChanges and schedule the Actor (for example weekly). Each site's summary is saved in a named key-value store in your own Apify account, ai-search-site-audit-state, with one record per monitor name and start URL. The next run with the same monitorName compares against it and adds a changes object to each site's summary:

• new and fixed findings (matched by finding code);
• AI crawler access flips in robots.txt, e.g. OAI-SearchBot: allowed -> blocked;
• llms.txt status (valid, no title, missing, not checked) and whether llms-full.txt exists;
• the score change.

{
  "startUrls": ["https://www.your-site.com/", "https://www.client-site.com/"],
  "confirmOwnership": true,
  "maxPages": 10,
  "trackChanges": true,
  "monitorName": "weekly"
}

The first run says "status": "baseline-saved". Later runs say "no-changes" or "changed". Example of a run after a robots.txt edit blocked OAI-SearchBot and llms.txt was removed:

"changes": {
  "status": "changed",
  "since": "2026-09-27T06:00:04+00:00",
  "message": "4 changes since the audit of 2026-09-27T06:00:04+00:00.",
  "newFindings": [
    {"severity": "high", "code": "answer-crawlers-blocked", "message": "robots.txt blocks crawlers that AI search and answer engines use: OAI-SearchBot"},
    {"severity": "low", "code": "llms-txt-missing", "message": "No /llms.txt."}
  ],
  "fixedFindings": [],
  "crawlerAccess": [{"bot": "OAI-SearchBot", "purpose": "search", "from": "allowed", "to": "blocked", "text": "OAI-SearchBot: allowed -> blocked"}],
  "llmsTxt": {"from": "valid", "to": "missing", "statusFrom": 200, "statusTo": 404, "text": "llms.txt: valid -> missing (HTTP 404)"},
  "llmsFullTxt": null,
  "score": {"from": 57, "to": 39, "delta": -18},
  "pagesAudited": {"from": 4, "to": 4},
  "lines": [
    "OAI-SearchBot: allowed -> blocked",
    "llms.txt: valid -> missing (HTTP 404)",
    "New high finding: robots.txt blocks crawlers that AI search and answer engines use: OAI-SearchBot",
    "New low finding: No /llms.txt.",
    "Score: 57 -> 39 (-18)"
  ]
}

lines says the same in plain words, ready to paste into an email or a Slack message. The dataset's "Changes since last run" view shows it per site, and the report's table has a "Since last run" column.

• Each run replaces the saved summary, so changes are always against the previous run. A site that couldn't be audited keeps its earlier baseline.
• Findings are compared by code: a finding that stays but affects more pages isn't reported as a change.
• If you change maxPages, the next comparison notes it, because page-level findings can differ for that reason alone.
• Delete the store ai-search-site-audit-state (or one record in it) to start over.

Output

• Dataset: for each site, its summary (recordType: "summary") and then one item per audited page (recordType: "page", with a site field). Views: "Site summary", "Pages" and "Changes since last run".
• Key-value store: REPORT (a Markdown report of all sites, to paste into a ticket or send to a client) and SUMMARY (the run as JSON: sites requested, audited and failed, requests made and the caps, dataset items per site, then every site summary). Both are rewritten after each site, so a run that hits its timeout keeps the sites done.

Summary excerpt:

{
  "recordType": "summary",
  "site": "https://www.example.com",
  "score": 57,
  "changes": null,
  "pagesAudited": 4,
  "datasetItems": 5,
  "requestsMade": 14,
  "requestCap": 60,
  "aiCrawlers": [
    {"bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "access": "blocked", "homeAllowed": false, "matchedBy": "specific"},
    {"bot": "OAI-SearchBot", "operator": "OpenAI", "purpose": "search", "access": "allowed", "homeAllowed": true, "matchedBy": "wildcard"}
  ],
  "llmsTxt": {"found": true, "hasTitle": true, "hasSummary": true, "links": 2, "llmsFullTxt": false},
  "sitemap": {"found": true, "urls": 3, "coverage": {"auditedIndexablePages": 3, "inSitemap": 3, "percent": 100}},
  "findings": [
    {"severity": "medium", "code": "content-needs-javascript", "message": "Pages that show little text without JavaScript...", "recommendation": "Render the main content on the server (SSR or static HTML)."}
  ],
  "disclaimer": "For your own sites: ..."
}

The score starts at 100 and loses 15 points for each high finding, 7 for each medium and 3 for each low (informational notes cost nothing). It's a quick way to track progress between runs, not an industry standard.

Crawler purposes follow each operator's own documentation at the time of writing. Operators add and rename crawlers, so the list is reviewed with each version of this Actor.

Pricing (suggested pay-per-event)

Event	Suggested price
Actor start (apify-actor-start), once per run however many sites it audits, which covers the robots.txt, llms.txt, sitemap and HTTPS checks	$0.02
Result: one site summary or one page record (apify-default-dataset-item)	$0.01

Each audited site adds its summary plus one record per audited page; its summary shows that count in datasetItems, and the report's table lists it per site.

• One site, 10 pages: $0.02 + 11 × $0.01 = $0.13.
• Five sites, 10 pages each: $0.02 + 5 × 11 × $0.01 = $0.57.
• A weekly monitor of 3 sites at 5 pages: $0.02 + 3 × 6 × $0.01 = $0.20 a run.

Change tracking adds no items (changes sits inside each summary). A site that can't be audited saves no items, so it adds nothing to the cost. Set a maximum cost per run in Apify if you want a hard limit: the run stops cleanly when it's reached.

Limits

• Up to 25 sites per run, up to 50 pages each. Sites are audited one after another, so 25 sites at 10 pages take roughly 6 to 12 minutes, longer when a robots.txt sets a Crawl-delay. Give long runs a matching timeout.
• It follows links found in the HTML. Pages reachable only through JavaScript are found through the sitemap instead.
• Response times are measured from Apify's data centre, not from your visitors' devices.
• Sitemaps hosted on another domain are listed but not fetched. Each sitemap is read up to 5 MB, and at most 5 child sitemaps from an index.

Local development

python -m unittest discover -s tests   # offline unit tests, no network
apify run --input '{"startUrls":["https://www.your-site.com/"],"confirmOwnership":true,"maxPages":5}'

The audit logic (src/core.py, src/robots.py, src/page.py) uses only the Python standard library. src/main.py is a thin wrapper around the Apify SDK (apify==4.0.2).

Support

Report problems in this Actor's Issues tab.