AI Crawler & GEO Readiness Audit

Find out how ready a website is for AI search and AI assistants. For every site in your list the Actor reads robots.txt, reports which AI crawlers are allowed or blocked, checks llms.txt, counts sitemap URLs, inspects the homepage and returns a 0-100 score with plain-English fixes. It can also draft an llms.txt file. Plain HTTP requests only: no browser, no proxies, no personal data.

What it does

For each site the Actor makes a few polite requests and returns one dataset item.

1. robots.txt and AI crawlers. The file is parsed as RFC 9309 describes: user-agent groups, the longest matching rule wins, Allow beats Disallow on a tie, * and $ wildcards work and user-agent names are case-insensitive. A missing file (4xx) means everything is allowed; an unreachable one (5xx) means crawlers should stay away. Each AI crawler gets allowed, partial (typical paths such as /blog/, /docs/ or /pricing are blocked) or blocked. Declared Sitemap: lines are listed.
2. llms.txt and llms-full.txt. Is the file served, with which HTTP status and content type, and is it real text rather than an HTML error page? The structure is checked against the llms.txt proposal (H1 title, blockquote summary, sections with link lists) and up to 20 internal links are tested for broken URLs.
3. Sitemaps. Sitemaps declared in robots.txt, or /sitemap.xml and /sitemap_index.xml when none is declared. Indexes and gzip files are followed. You get the URL count (capped by maxSitemapUrls), the newest lastmod date and any problems.
4. Homepage. Status, final URL, response time, HTML size, title, meta description, canonical, robots meta tag and X-Robots-Tag header (including noai and noimageai), hreflang count, Open Graph tags, JSON-LD types, and a heuristic telling whether the main text is in the HTML without running JavaScript.
5. Score and recommendations. A 0-100 score with the weights below and a prioritised list of fixes (high, medium, low).
6. Optional llms.txt draft. With generateLlmsTxt on, the Actor writes a draft from the sitemap URLs and the titles and descriptions of up to maxPagesForLlmsTxt pages, fetching only pages that robots.txt allows. The draft is extractive: no AI model writes anything, so review it before publishing.

The built-in list has 21 user-agent tokens taken from vendor documentation: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google-Extended, Applebot-Extended, CCBot, Amazon (Amazonbot, Amzn-SearchBot, Amzn-User), Meta (meta-externalagent, meta-webindexer, meta-externalfetcher), DuckAssistBot and MistralAI-User. Bytespider and cohere-ai are common in robots.txt files but not documented by their vendors, so treat them as unofficial. Use aiBots to check your own tokens instead.

Score part	Points	What earns the points
AI crawler access	30	Search and assistant crawlers allowed (partial counts half). Training-only crawlers are reported but not scored, because blocking them is a legitimate choice.
Readable without JavaScript	20	Homepage text is in the HTML: yes 20, partial 10, no 0.
Sitemap	15	Found with URLs 8, declared in robots.txt 3, lastmod on half the URLs 2, newest lastmod under 12 months old 2.
Metadata and structured data	20	Title 4, meta description 4, canonical 3, JSON-LD 5, complete Open Graph 2, an H1 2.
llms.txt	10	Served as a file 6, H1 plus a link 2, links checked and none broken 2.
Page directives	5	No noindex 3, no nosnippet, noai or noimageai 2.

llms.txt is an emerging convention and AI systems differ in whether they read it, which is why it is worth 10 points only.

Use cases

• GEO and SEO agencies: audit a client or prospect list in one run and sort by score.
• Publishers: confirm that you block the training crawlers you want to block while letting AI search crawlers in, or the reverse.
• In-house teams: schedule weekly runs and watch for robots.txt, sitemap or llms.txt changes after a release.
• Developers: check a site before launch for leftover noindex, empty server output and a missing sitemap.
• Documentation teams: generate a first llms.txt draft from the sitemap, then edit it by hand.

Input

Field	Default	Description
urls	required	Sites to audit, up to 1,000, as full URLs. The site root of each address is audited. Duplicates run once.
aiBots	built-in 21	Your own user-agent tokens instead of the built-in list. The score then averages over your tokens.
maxSitemapUrls	5000	Stop counting after this many sitemap URLs per site (1 to 50,000). The result says when the count is a lower bound.
generateLlmsTxt	false	Draft an llms.txt per site.
maxPagesForLlmsTxt	30	Pages fetched for each draft (1 to 200). Each fetched page is a billed event.
respectRobotsTxt	true	Apply robots.txt to this Actor's own requests. Turn off only for sites you own or may audit.
maxConcurrency	10	Sites in parallel (1 to 30), at most 2 requests at a time per domain.
requestTimeoutSecs	20	Timeout per request (1 to 120).

{
  "urls": ["https://example.com", "https://wordpress.org"],
  "generateLlmsTxt": false,
  "maxSitemapUrls": 5000
}

Output example

One dataset item per site. A site that fails gets an item with error set, no score and no charge. This is a shortened real result for example.com (long sections and lists trimmed):

{
  "url": "https://example.com",
  "domain": "example.com",
  "checkedAt": "2026-10-08T10:28:20.898Z",
  "score": 59,
  "scoreBreakdown": { "aiCrawlerAccess": { "points": 30, "max": 30 }, "jsReadability": { "points": 20, "max": 20 }, "sitemap": { "points": 0, "max": 15 }, "metadata": { "points": 4, "max": 20 }, "llmsTxt": { "points": 0, "max": 10 }, "directives": { "points": 5, "max": 5 } },
  "robots": { "url": "https://example.com/robots.txt", "found": false, "fetchStatus": "not-found", "httpStatus": 404, "groupCount": 0, "sitemapsDeclared": [], "counts": { "allowed": 21, "partial": 0, "blocked": 0, "unknown": 0 } },
  "aiBots": { "GPTBot": "allowed", "OAI-SearchBot": "allowed", "ClaudeBot": "allowed", "Claude-SearchBot": "allowed", "PerplexityBot": "allowed", "Google-Extended": "allowed" },
  "llmsTxt": { "url": "https://example.com/llms.txt", "found": false, "httpStatus": 404, "valid": false, "wellStructured": false, "brokenLinks": [] },
  "llmsFullTxt": { "url": "https://example.com/llms-full.txt", "found": false, "httpStatus": 404 },
  "sitemaps": { "found": false, "declaredInRobotsTxt": false, "urlCount": 0, "newestLastmod": null, "issues": [] },
  "homepage": { "status": 200, "finalUrl": "https://example.com/", "responseTimeMs": 48, "htmlBytes": 577, "title": "Example Domain", "metaDescription": null, "canonical": null, "hreflangCount": 0, "jsonLdTypes": [], "noindex": false, "noai": false, "readableWithoutJs": "yes", "visibleTextChars": 156 },
  "recommendations": [
    { "priority": "high", "area": "sitemap", "message": "No XML sitemap was found. Publish /sitemap.xml and add a \"Sitemap:\" line to robots.txt so crawlers can discover all your pages." }
  ],
  "generatedLlmsTxt": null,
  "error": null
}

Bot statuses are allowed, partial, blocked or unknown (robots.txt unreachable). The Console AI readiness table shows the domain, score, the status of six key bots, llms.txt found, sitemap URL count and JavaScript readability; two more views list recommendations and generated drafts.

Pricing

Pay per event. You pay for sites that were actually audited, never for failures.

Event	Price	When it is charged
Actor start	$0.005	Once per run.
Site audited	$0.015	Once per site whose homepage answered with an HTTP status below 400 and was analysed.
Page fetched for llms.txt draft	$0.001	Per extra page fetched, only when generateLlmsTxt is on.

Invalid URLs, robots.txt blocks, timeouts, DNS or TLS errors and 4xx or 5xx homepages are free. Examples: 100 sites cost about $1.51, 1,000 sites about $15.01, and one site with a 30-page draft $0.05. The Actor never charges beyond the maximum total charge you set; it stops early and says so in the run status.

FAQ

Does it respect robots.txt? Yes. With respectRobotsTxt on, the Actor reads each site's robots.txt first and fetches the homepage, llms files, sitemaps, link checks and draft pages only where that file allows its own user agent. If robots.txt cannot be read because of a server error, the site is skipped and not charged. It identifies itself as OttomathGeoAudit/1.0 (+https://apify.com), uses no proxies and no fingerprint tricks.

Does it run a browser or JavaScript? No. "Readable without JavaScript" is a heuristic built from visible text, empty app containers such as #root or #__next, noscript warnings and the share of script in the page. It shows what a crawler that does not run JavaScript is likely to see, and it can be wrong on unusual pages.

What does "partial" mean? The crawler may fetch the site root but is blocked from some typical paths. The blocked paths are listed in robots.aiBotDetails.

Does it collect personal data? No. It does not extract e-mail addresses, phone numbers or names, and text taken from pages is scrubbed of anything that looks like an e-mail address or phone number.

How is the draft written, and what is billed? From what the pages already say: the homepage title and description, then page titles and meta descriptions grouped by the first part of the URL path. Pages that are not HTML, missing or noindex are left out, but every page that is actually fetched counts as one event. Pages disallowed by robots.txt are not fetched and not billed.

Limitations

• robots.txt shows a site's stated policy, not what its servers enforce. A firewall can block a crawler that robots.txt allows, and some assistants fetch pages for a user in ways robots.txt may not govern.
• Only robots.txt, llms files, sitemaps and the homepage are analysed, plus the pages used for the optional draft. This is not a full-site crawl.
• Sites that block data-centre traffic may answer 403; they are reported with an error and are free.
• Size limits: homepage HTML up to 2 MB, llms.txt up to 1 MB, each sitemap file up to 20 MB (50 MB decompressed), at most 30 sitemap files per site. Counts above maxSitemapUrls are flagged as a lower bound.
• Each site has an overall time limit of at least 90 seconds, and HTML analysis stops after 20 seconds.
• The bot list reflects vendor documentation read in October 2026. Vendors add and rename crawlers, so use aiBots for new tokens.
• The score is a transparent checklist, not a ranking factor or a promise of visibility in any AI product. The Actor is not affiliated with any AI company or search engine.

Changelog

• 1.0 (October 2026): first release.