AI Crawler & llms.txt Checker – GEO / AI Search Audit

Check in seconds whether ChatGPT, Claude, Perplexity and Google's AI can crawl a website. This AI crawler checker and llms.txt checker reads each site's robots.txt rules for 15 AI and search bots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more), validates llms.txt / llms-full.txt, finds the XML sitemap, inspects homepage SEO and Schema.org JSON-LD, and returns a 0–100 GEO (Generative Engine Optimization) readiness score with fix codes — for one domain or 1,000 in a single run.

• 💸 $0.02 per audited domain = $2 per 100 domains. Failed or partial audits are free.
• ⚡ ~5–10 seconds per domain, 5 domains in parallel by default. No login, no browser, no proxies.
• 🧾 Clean JSON / CSV / Excel output, ready for client reports, spreadsheets, dashboards or AI agents.

What does the AI Crawler & llms.txt Checker do?

For every domain you submit it fetches at most five small public URLs (/robots.txt, /llms.txt, /llms-full.txt, the sitemap and the homepage) and reports:

Area	What you get
AI crawler access (robots.txt)	allowed / blocked at the site root for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, DuckAssistBot (list is configurable, up to 50 tokens), plus whether the bot is named by an explicit rule
llms.txt checker	present? starts with a # Title? has a > summary blockquote? number of ## sections and Markdown links; is llms-full.txt present?
XML sitemap	found via robots.txt Sitemap: line or /sitemap.xml (gzip supported), type (sitemapindex / urlset), entry count
Homepage SEO for AI search	title, meta description length, canonical, <html lang>, hreflang, Open Graph tags, Twitter card, Schema.org JSON-LD types + parse errors, H1 count, approx. word count, noindex (meta or X-Robots-Tag)
GEO readiness score	0–100 score + machine-readable recommendations such as ADD_LLMS_TXT or UNBLOCK_AI_SEARCH_BOTS:PerplexityBot

Works for websites in any language (output uses language-neutral codes).

Try one domain free first

Want to check a single site without an Apify account? Use the free AI-search readiness checker on tesselyn.com. It is a separate, simpler one-domain check with its own bot list and score, so its numbers can differ from this Actor's. Use this Actor for lists of domains, the full homepage SEO and Schema.org checks, scheduled monitoring and API/MCP use.

Who is it for? Use cases

• SEO & GEO agencies — bulk-audit a client portfolio and turn the table into an AI-search readiness report.
• In-house SEO / marketing teams — check that a CDN, plugin or robots.txt change didn't accidentally block OAI-SearchBot or PerplexityBot.
• Competitor research — see which competitors block AI crawlers or already publish llms.txt.
• Publishers & legal/compliance teams — get an inventory of which AI training crawlers (GPTBot, CCBot, Google-Extended…) your sites opt out of.
• Developers & AI agents — feed structured site-readiness data into pipelines, dashboards or MCP-based agents.

How to use it

1. Click Try for free / Start. The input is pre-filled with three example sites (apify.com, stripe.com, bbc.com), so the first run works instantly.
2. Replace them with your own domains or homepage URLs (bulk-paste or upload a list; example.com without https:// works too).
3. Optionally edit the list of AI crawler user-agents, the timeout or concurrency.
4. Run it and open the Overview, AI crawlers & llms.txt or Homepage SEO tab of the results, or export to JSON, CSV, Excel or HTML.

Input example

{
  "startUrls": [
    { "url": "https://apify.com" },
    { "url": "https://stripe.com" },
    { "url": "https://www.bbc.com" }
  ]
}

Field	Default	Description
startUrls	– (required)	Domains or homepage URLs, max 1,000 per run. Duplicates are audited once.
aiBots	15 tokens listed above	User-agent tokens to evaluate in robots.txt (max 50).
maxSitemapUrlsToCount	5000	Stop counting sitemap entries after this many.
requestTimeoutSecs	20	Per-request timeout.
maxConcurrency	5	Domains audited in parallel (max 20).

Output example

ℹ️ Scores are automated indicators, not guarantees of visibility in any search or AI product.

Overview table from a real run (October 2026; results change as sites update their files):

Domain	Score	Blocked AI bots (robots.txt)	llms.txt	Sitemap	JSON-LD types
apify.com	100	–	✅ 9 sections, 73 links	✅ index, 17 sitemaps	Organization, WebSite, SoftwareApplication…
stripe.com	95	–	✅ 31 sections, 308 links	✅ index, 9 sitemaps	Organization, WebSite, ContactPoint…
www.bbc.com	65	GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot + 4 more	❌	✅ urlset	NewsMediaOrganization, WebPage

One dataset item (shortened):

{
  "input": "https://apify.com",
  "domain": "apify.com",
  "score": 100,
  "aiBotsBlocked": [],
  "robotsTxt": {
    "found": true,
    "sitemaps": ["https://apify.com/sitemap.xml"],
    "bots": { "GPTBot": "allowed", "ClaudeBot": "allowed", "PerplexityBot": "allowed", "Google-Extended": "allowed" }
  },
  "llmsTxt": { "found": true, "hasH1Title": true, "hasSummaryBlockquote": true, "sectionCount": 9, "linkCount": 73, "llmsFullTxtFound": false },
  "sitemap": { "found": true, "url": "https://apify.com/sitemap.xml", "type": "sitemapindex", "entries": 17 },
  "httpStatus": 200,
  "https": true,
  "homepage": {
    "title": "Apify: Marketplace of ready-to-run tools for AI",
    "metaDescriptionLength": 143,
    "canonical": "https://apify.com",
    "lang": "en",
    "openGraph": ["og:description", "og:image", "og:title", "og:type", "og:url"],
    "jsonLdTypes": ["Organization", "SoftwareApplication", "WebSite"],
    "h1Count": 1,
    "noindex": false
  },
  "recommendations": []
}

Robots.txt values look like allowed, blocked, blocked (explicit rule) (the bot is named in its own User-agent group) or allowed (no robots.txt).

How the GEO readiness score works

ℹ️ The score is an automated indicator based on the checks below, not a guarantee of visibility, ranking or citation in any search engine or AI product.

The score is a transparent checklist (max 100):

Check	Points
Served over HTTPS	10
robots.txt present	10
AI search crawlers not blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot)	15
llms.txt present (+5 if it starts with an # H1 title)	10 + 5
XML sitemap found	10
Schema.org JSON-LD on homepage	10
Title, meta description (≥ 50 chars), canonical, Open Graph, html lang, exactly one H1	5 each
Homepage noindex	−30

Blocking training crawlers such as GPTBot, CCBot or Google-Extended does not lower the score — that is a content-licensing decision, and the per-bot table still shows it.

Recommendation codes: SERVE_OVER_HTTPS, ADD_ROBOTS_TXT, UNBLOCK_AI_SEARCH_BOTS:<bots>, ADD_LLMS_TXT, LLMS_TXT_SHOULD_START_WITH_H1, ADD_XML_SITEMAP_AND_REFERENCE_IN_ROBOTS, ADD_TITLE, ADD_META_DESCRIPTION, ADD_CANONICAL, ADD_SCHEMA_ORG_JSON_LD, FIX_INVALID_JSON_LD, ADD_OPEN_GRAPH_TAGS, ADD_HTML_LANG, HOMEPAGE_IS_NOINDEX, USE_EXACTLY_ONE_H1.

How much does it cost? Pricing

Pay-per-event pricing — you pay only for results:

Event	Price
Domain audited successfully (domain-audited)	$0.02 per domain (= $2 per 100 domains, $20 per 1,000)
Actor start (apify-actor-start, per run, per GB of memory)	$0.00005

Platform compute is included in these prices. Rows with an error field are not charged: invalid or non-public inputs, unreachable sites, robots.txt server errors/timeouts, homepages that return an HTTP error (e.g. 403 bot protection) or that disallow this auditor in robots.txt. Such rows still contain whatever site-level data could be collected (e.g. the AI-crawler table), with score: null. Apify's Free plan includes monthly platform credit you can use to try it. You can cap spending per run with the Maximum cost per run option.

Integrations and API

• API: run it from any language via the Apify API or the Python / JavaScript clients — see the API tab for ready-made code.
• No-code: connect it to Make, Zapier, n8n, Google Sheets, Slack or webhooks via Apify integrations.
• Monitoring: use Apify Schedules to re-audit your sites weekly and catch newly blocked AI crawlers.
• AI agents: available to MCP clients (Claude, Cursor, etc.) through the Apify MCP server.

Responsible use

• Fetches at most ~5 small public URLs per domain (robots.txt, llms.txt, llms-full.txt, sitemap, homepage), one at a time with at least 1 s between requests (longer if the site sets a Crawl-delay, capped at 10 s).
• Identifies itself honestly with the User-Agent TesselynAuditBot/1.0 (+https://apify.com/tesselyn). It never impersonates GPTBot or other crawlers.
• Obeys robots.txt (RFC 9309) for its own token TesselynAuditBot: disallowed paths are skipped; if the homepage is disallowed the domain is reported and not charged; if robots.txt is unreachable (server error/timeout) the site is not crawled at all.
• Only public websites: private, loopback, link-local, cloud-metadata and other non-public addresses are refused, including after DNS resolution and on every redirect. Only ports 80/443, max 5 redirects, per-request timeouts and response-size caps (robots.txt 512 KB, llms.txt 1 MB, sitemap 10 MB, homepage 3 MB).
• Max 1,000 domains per run. No login, no CAPTCHA bypass, no proxies, no JavaScript rendering.
• Collects no contact data (no emails, phones or addresses are extracted; JSON-LD is reduced to @type names only).

Terms of use

These usage terms supplement, and do not replace, the Apify Standard Actor Contract, which governs your use of this Actor.

• Only submit domains you own, manage, or otherwise have a legitimate interest and right to audit. You are responsible for your use of the Actor and its output complying with applicable laws and the target websites' terms.
• Do not use the Actor to probe internal networks, to load-test, or to circumvent access controls. Abusive use may be blocked.
• Sanctions and restricted countries. You may not use this Actor if you are located in, ordinarily resident in, or acting on behalf of a person or organisation in a restricted country or region. Currently these are: Afghanistan, Belarus, Central African Republic, Cuba, Democratic Republic of the Congo, Haiti, Iran, Iraq, Lebanon, Libya, Mali, Myanmar, Nicaragua, North Korea, Russia, Somalia, South Sudan, Sudan, Syria, Venezuela, Yemen, Zimbabwe, Antarctica, and the Crimea, Sevastopol, Donetsk, Luhansk, Kherson and Zaporizhzhia regions of Ukraine. You also may not use it if you are on, or owned or controlled by someone on, any applicable sanctions list (including those of the United Nations, the United States (OFAC SDN), the European Union, the United Kingdom or Israel). We may block such use, and we may update this list when the law or our policy changes.
• Output is an automated, best-effort technical assessment provided "as is"; verify it before relying on it for consequential decisions. The score is not an endorsement or a guarantee of visibility in any search or AI product. Names of third-party crawlers/products are used only to describe compatibility; no affiliation is implied.
• No warranty; limitation of liability. To the maximum extent permitted by applicable law, the Actor and its output are provided "as is" and "as available", without warranties of any kind, express or implied (including accuracy, completeness, merchantability, fitness for a particular purpose and non-infringement), as set out in section 7 of the Apify Standard Actor Contract. The Actor creator (Tesselyn) is not liable for any indirect, incidental, special, consequential or punitive loss or damage, including lost traffic, rankings, revenue, profits or data, arising from use of or reliance on the Actor or its output; any other liability is limited as set out in section 8 of that Contract, and nothing here limits liability that cannot be limited under applicable law. Output is informational only and is not SEO, legal or other professional advice. Results may change as websites, crawlers and standards change.

Privacy notice

• What is processed: the domains/URLs you submit and the public technical data listed above (robots.txt rules, llms.txt structure, sitemap counts, homepage metadata such as title, language, canonical URL, Open Graph tag names and Schema.org types). A homepage title can occasionally contain a person's or business name; nothing else about individuals is collected.
• Where it is stored: only in your own Apify run storage (default dataset, run log and input), under your Apify account and Apify's retention settings. The Actor sends nothing to third parties other than the HTTP requests to the sites you submit, and stores nothing elsewhere. It runs with Apify limited permissions (it can access only its own run storage).
• Roles: for any personal data in your input or output you are the controller and the Actor creator acts as processor, as set out in section 5 of the Apify Standard Actor Contract. Run logs contain only domain names, scores and error codes.
• Deletion: delete the run or dataset in Apify Console at any time. Questions or privacy requests: open an issue on the Actor's Issues tab.

Related Tesselyn tools

• Broken Link & Redirect Checker: crawl a site for broken links, redirect chains and missing #anchors.
• Domain Health Checker: domain expiry, DNS, SPF/DMARC/DKIM, SSL and HSTS in bulk. Coming to the Apify Store soon. Until then, try the free domain and email health checker on tesselyn.com.
• Bulk PageSpeed & Core Web Vitals Checker: Google PageSpeed Insights scores for many URLs. Coming to the Apify Store soon.
• See all Tesselyn Actors on the Apify Store and the free single-site tools on tesselyn.com.

FAQ

What is llms.txt and does it matter?

llms.txt is a community proposal (2024) for a Markdown file at the site root that gives language models a curated map of a site's key content. It is not an official standard and AI search engines have not confirmed using it for ranking; it is cheap to add and used by some AI tools and agents. This Actor checks presence and basic structure (H1 title, summary, sections, links).

Which AI crawlers should I allow?

Search/answer crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) affect whether you can be found and cited in AI answers; training crawlers (GPTBot, CCBot, Google-Extended…) are an opt-out choice. The report shows each separately so you can decide.

Which bots matter for ChatGPT, Claude and Perplexity answers?

The search and answer bots. By default this Actor checks OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot. You can add other tokens, such as Claude-User or Perplexity-User, in the aiBots input. OpenAI and Anthropic describe GPTBot and ClaudeBot as their model-training crawlers, so blocking those is a training opt-out and is reported separately. Vendors change their crawlers, so check their current documentation before you edit robots.txt.

Is there a free version?

For a single domain, yes: the free AI-search readiness checker on tesselyn.com runs a simpler check in your browser, with no account needed. Apify's Free plan also includes monthly platform credit you can use to try this Actor.

Does "allowed" mean the bot can really reach my site?

It means your robots.txt allows that user-agent at the site root. robots.txt is advisory; firewalls/CDNs may still block or allow bots regardless, and path-specific rules below the root are not evaluated. This Actor does not impersonate crawlers, so it cannot test CDN-level blocking.

Does it render JavaScript or crawl the whole site?

No. It analyzes the raw homepage HTML and the root-level files only. That keeps it fast, cheap and polite.

Why did a domain return an error?

See the error field: e.g. HOMEPAGE_HTTP_403 (bot protection), ROBOTS_TXT_UNREACHABLE:TIMEOUT, HOMEPAGE_DISALLOWED_BY_ROBOTS_FOR_THIS_AUDITOR, INVALID_OR_NON_PUBLIC_DOMAIN:…. Error rows are not charged.

Is it legal?

It only reads small public files that websites publish for crawlers (robots.txt, llms.txt, sitemap) and the public homepage, identifies itself, obeys robots.txt and extracts no contact data. You remain responsible for auditing only sites you have a right to audit (see Terms of use). This is not legal advice.

How fast / how polite is it?

Each domain gets ~5 sequential requests spaced ≥1 s apart (about 5–10 s per domain); different domains are processed in parallel with limited concurrency (default 5).

Support

Built and maintained by Tesselyn. Found a bug, need another check or a different crawler list? Open an issue on the Actor's Issues tab — feedback is welcome.

Website: Tesselyn website (tesselyn.com)