AI Crawler & llms.txt Checker – GEO / AI Search Audit
Pricing
from $16.00 / 1,000 domain auditeds
AI Crawler & llms.txt Checker – GEO / AI Search Audit
Check which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more) a site's robots.txt allows or blocks, validate llms.txt, sitemap & Schema.org JSON-LD, and get a 0-100 GEO readiness score with fix codes. Bulk, no login. $2 per 100 domains.
Pricing
from $16.00 / 1,000 domain auditeds
Rating
0.0
(0)
Developer
Tesselyn
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a month ago
Last modified
Categories
Features
Share
Check in seconds whether ChatGPT, Claude, Perplexity and Google's AI can crawl a website. This AI crawler checker and llms.txt checker reads each site's robots.txt rules for 15 AI and search bots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more), validates llms.txt / llms-full.txt, finds the XML sitemap, inspects homepage SEO and Schema.org JSON-LD, and returns a 0–100 GEO (Generative Engine Optimization) readiness score with fix codes — for one domain or 1,000 in a single run.
⚠️ Automated, informational results, provided "as is". Not legal, compliance or SEO advice. A high score does not guarantee crawling, indexing, rankings, or that any search engine or AI assistant will show or cite a site. See Terms of use.
- 💸 $0.02 per audited domain = $2 per 100 domains. Failed or partial audits are free.
- ⚡ ~5–10 seconds per domain, 5 domains in parallel by default. No login, no browser, no proxies.
- 🧾 Clean JSON / CSV / Excel output, ready for client reports, spreadsheets, dashboards or AI agents.
What does the AI Crawler & llms.txt Checker do?
For every domain you submit it fetches at most five small public URLs (/robots.txt, /llms.txt, /llms-full.txt, the sitemap and the homepage) and reports:
| Area | What you get |
|---|---|
| AI crawler access (robots.txt) | allowed / blocked at the site root for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, DuckAssistBot (list is configurable, up to 50 tokens), plus whether the bot is named by an explicit rule |
| llms.txt checker | present? starts with a # Title? has a > summary blockquote? number of ## sections and Markdown links; is llms-full.txt present? |
| XML sitemap | found via robots.txt Sitemap: line or /sitemap.xml (gzip supported), type (sitemapindex / urlset), entry count |
| Homepage SEO for AI search | title, meta description length, canonical, <html lang>, hreflang, Open Graph tags, Twitter card, Schema.org JSON-LD types + parse errors, H1 count, approx. word count, noindex (meta or X-Robots-Tag) |
| GEO readiness score | 0–100 score + machine-readable recommendations such as ADD_LLMS_TXT or UNBLOCK_AI_SEARCH_BOTS:PerplexityBot |
Works for websites in any language (output uses language-neutral codes).
Use cases for the AI crawler and llms.txt checker
- SEO & GEO agencies — bulk-audit a client portfolio and turn the table into an AI-search readiness report.
- In-house SEO / marketing teams — check that a CDN, plugin or robots.txt change didn't accidentally block OAI-SearchBot or PerplexityBot.
- Competitor research — see which competitors block AI crawlers or already publish
llms.txt. - Publishers & legal/compliance teams — get an inventory of which AI training crawlers (GPTBot, CCBot, Google-Extended…) your sites opt out of.
- Developers & AI agents — feed structured site-readiness data into pipelines, dashboards or MCP-based agents.
Ready-made example tasks
Open one, check the input and click Try to run a copy in your own account:
- Check GPTBot, ClaudeBot & PerplexityBot access in robots.txt
- Validate llms.txt files on documentation websites
- Score websites for AI search (GEO) readiness
How to check a website for AI crawlers and llms.txt
- Click Try for free / Start. The input is pre-filled with three example sites (apify.com, stripe.com, bbc.com), so the first run works instantly.
- Replace them with your own domains or homepage URLs (bulk-paste or upload a list;
example.comwithouthttps://works too). - Optionally edit the list of AI crawler user-agents, the timeout or concurrency.
- Run it and open the Overview, AI crawlers & llms.txt or Homepage SEO tab of the results, or export to JSON, CSV, Excel or HTML.
Input example
{"startUrls": [{ "url": "https://apify.com" },{ "url": "https://stripe.com" },{ "url": "https://www.bbc.com" }]}
| Field | Default | Description |
|---|---|---|
startUrls | – (required) | Domains or homepage URLs, max 1,000 per run. Duplicates are audited once. |
aiBots | 15 tokens listed above | User-agent tokens to evaluate in robots.txt (max 50). |
maxSitemapUrlsToCount | 5000 | Stop counting sitemap entries after this many. |
requestTimeoutSecs | 20 | Per-request timeout. |
maxConcurrency | 5 | Domains audited in parallel (max 20). |
Output example
ℹ️ Scores are automated indicators, not guarantees of visibility in any search or AI product.
Overview table from a real run (October 2026; results change as sites update their files):
| Domain | Score | Blocked AI bots (robots.txt) | llms.txt | Sitemap | JSON-LD types |
|---|---|---|---|---|---|
| apify.com | 100 | – | ✅ 9 sections, 73 links | ✅ index, 17 sitemaps | Organization, WebSite, SoftwareApplication… |
| stripe.com | 95 | – | ✅ 31 sections, 308 links | ✅ index, 9 sitemaps | Organization, WebSite, ContactPoint… |
| www.bbc.com | 65 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot + 4 more | ❌ | ✅ urlset | NewsMediaOrganization, WebPage |
One dataset item (shortened):
{"input": "https://apify.com","domain": "apify.com","score": 100,"aiBotsBlocked": [],"robotsTxt": {"found": true,"sitemaps": ["https://apify.com/sitemap.xml"],"bots": { "GPTBot": "allowed", "ClaudeBot": "allowed", "PerplexityBot": "allowed", "Google-Extended": "allowed" }},"llmsTxt": { "found": true, "hasH1Title": true, "hasSummaryBlockquote": true, "sectionCount": 9, "linkCount": 73, "llmsFullTxtFound": false },"sitemap": { "found": true, "url": "https://apify.com/sitemap.xml", "type": "sitemapindex", "entries": 17 },"httpStatus": 200,"https": true,"homepage": {"title": "Apify: Marketplace of ready-to-run tools for AI","metaDescriptionLength": 143,"canonical": "https://apify.com","lang": "en","openGraph": ["og:description", "og:image", "og:title", "og:type", "og:url"],"jsonLdTypes": ["Organization", "SoftwareApplication", "WebSite"],"h1Count": 1,"noindex": false},"recommendations": []}
Robots.txt values look like allowed, blocked, blocked (explicit rule) (the bot is named in its own User-agent group) or allowed (no robots.txt).
How the GEO readiness score works
ℹ️ The score is an automated indicator based on the checks below, not a guarantee of visibility, ranking or citation in any search engine or AI product.
The score is a transparent checklist (max 100):
| Check | Points |
|---|---|
| Served over HTTPS | 10 |
| robots.txt present | 10 |
| AI search crawlers not blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) | 15 |
llms.txt present (+5 if it starts with an # H1 title) | 10 + 5 |
| XML sitemap found | 10 |
| Schema.org JSON-LD on homepage | 10 |
Title, meta description (≥ 50 chars), canonical, Open Graph, html lang, exactly one H1 | 5 each |
Homepage noindex | −30 |
Blocking training crawlers such as GPTBot, CCBot or Google-Extended does not lower the score — that is a content-licensing decision, and the per-bot table still shows it.
Recommendation codes: SERVE_OVER_HTTPS, ADD_ROBOTS_TXT, UNBLOCK_AI_SEARCH_BOTS:<bots>, ADD_LLMS_TXT, LLMS_TXT_SHOULD_START_WITH_H1, ADD_XML_SITEMAP_AND_REFERENCE_IN_ROBOTS, ADD_TITLE, ADD_META_DESCRIPTION, ADD_CANONICAL, ADD_SCHEMA_ORG_JSON_LD, FIX_INVALID_JSON_LD, ADD_OPEN_GRAPH_TAGS, ADD_HTML_LANG, HOMEPAGE_IS_NOINDEX, USE_EXACTLY_ONE_H1.
How much does it cost to check a website for AI search readiness?
Pay-per-event pricing — you pay only for results:
| Event | Price |
|---|---|
Domain audited successfully (domain-audited) | $0.02 per domain (= $2 per 100 domains, $20 per 1,000) |
Actor start (apify-actor-start, per run, per GB of memory) | $0.00005 |
Examples at the current price: 1 domain ≈ $0.02, 100 domains ≈ $2, 1,000 domains ≈ $20, plus the per-run start fee. Platform compute is included in these prices. Rows with an error field are not charged: invalid or non-public inputs, unreachable sites, robots.txt server errors/timeouts, homepages that return an HTTP error (e.g. 403 bot protection) or that disallow this auditor in robots.txt. Such rows still contain whatever site-level data could be collected (e.g. the AI-crawler table), with score: null. Apify's Free plan includes monthly platform credit you can use to try it. You can cap spending per run with the Maximum cost per run option.
Recommended schedule and monitoring
robots.txt files, CDN bot rules and SEO plugins change without notice, so a one-off audit goes stale.
- Weekly is a sensible default for your own sites: save your domain list as a task and attach an Apify Schedule to it.
- After every release, CDN or robots.txt change, run it once to confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot are still allowed.
- Compare
aiBotsBlockedandscorebetween runs (or add a webhook) to get alerted when a crawler becomes blocked orllms.txtdisappears.
Integrations and API
- API: run it from any language with the Apify API or the Python / JavaScript clients. The API tab has ready-made code.
- No-code: connect it to Make, Zapier, n8n, Google Sheets, Slack or webhooks via Apify integrations.
- AI agents: available to MCP clients (Claude, Cursor, etc.) through the Apify MCP server.
Responsible use
- Fetches at most ~5 small public URLs per domain (robots.txt, llms.txt, llms-full.txt, sitemap, homepage), one at a time with at least 1 s between requests (longer if the site sets a
Crawl-delay, capped at 10 s). - Identifies itself honestly with the User-Agent
TesselynAuditBot/1.0 (+https://apify.com/tesselyn). It never impersonates GPTBot or other crawlers. - Obeys robots.txt for its own token
TesselynAuditBot, read with Python's standardurllib.robotparser(a simple parser: it does not support*or$wildcards inside paths): disallowed paths are skipped; if the homepage is disallowed the domain is reported and not charged; if robots.txt is unreachable (server error/timeout) the site is not crawled at all. - Only public websites: private, loopback, link-local, cloud-metadata and other non-public addresses are refused, including after DNS resolution and on every redirect. Only ports 80/443, max 5 redirects, per-request timeouts and response-size caps (robots.txt 512 KB, llms.txt 1 MB, sitemap 10 MB, homepage 3 MB).
- Max 1,000 domains per run. No login, no CAPTCHA bypass, no proxies, no JavaScript rendering.
- Collects no contact data (no emails, phones or addresses are extracted; JSON-LD is reduced to
@typenames only).
Terms of use
These usage terms supplement, and do not replace, the Apify Standard Actor Contract, which governs your use of this Actor.
- Only submit domains you own, manage, or otherwise have a legitimate interest and right to audit. You are responsible for your use of the Actor and its output complying with applicable laws and the target websites' terms.
- Do not use the Actor to probe internal networks, to load-test, or to circumvent access controls. Abusive use may be blocked.
- Sanctions and restricted countries. You may not use this Actor if you are located in, ordinarily resident in, or acting on behalf of a person or organisation in a restricted country or region. Currently these are: Afghanistan, Belarus, Central African Republic, Cuba, Democratic Republic of the Congo, Haiti, Iran, Iraq, Lebanon, Libya, Mali, Myanmar, Nicaragua, North Korea, Russia, Somalia, South Sudan, Sudan, Syria, Venezuela, Yemen, Zimbabwe, Antarctica, and the Crimea, Sevastopol, Donetsk, Luhansk, Kherson and Zaporizhzhia regions of Ukraine. You also may not use it if you are on, or owned or controlled by someone on, any applicable sanctions list (including those of the United Nations, the United States (OFAC SDN), the European Union, the United Kingdom or Israel). We may block such use, and we may update this list when the law or our policy changes.
- Output is an automated, best-effort technical assessment provided "as is"; verify it before relying on it for consequential decisions. The score is not an endorsement or a guarantee of visibility in any search or AI product. Names of third-party crawlers/products are used only to describe compatibility; no affiliation is implied.
- Informational only; no guarantee of outcomes. Output is not legal, regulatory, compliance, security, SEO or other professional advice, and using the Actor does not create an advisory relationship. Tesselyn does not promise any outcome, including rankings, indexing, AI-search visibility or citations. You are responsible for checking results and for the changes and decisions you make based on them.
- No warranty; limitation of liability. To the maximum extent permitted by applicable law, the Actor and its output are provided "as is" and "as available", without warranties of any kind, express or implied (including accuracy, completeness, merchantability, fitness for a particular purpose and non-infringement), as set out in section 7 of the Apify Standard Actor Contract. The Actor creator (Tesselyn) is not liable for any indirect, incidental, special, consequential or punitive loss or damage, including lost traffic, rankings, revenue, profits or data, arising from use of or reliance on the Actor or its output; any other liability is limited as set out in section 8 of that Contract, and nothing here limits liability that cannot be limited under applicable law. Output is informational only and is not SEO, legal or other professional advice. Results may change as websites, crawlers and standards change.
Privacy notice
- What is processed: the domains/URLs you submit and the public technical data listed above (robots.txt rules, llms.txt structure, sitemap counts, homepage metadata such as title, language, canonical URL, Open Graph tag names and Schema.org types). A homepage title can occasionally contain a person's or business name; nothing else about individuals is collected.
- Where it is stored: only in your own Apify run storage (default dataset, run log and input), under your Apify account and Apify's retention settings. The Actor sends nothing to third parties other than the HTTP requests to the sites you submit, and stores nothing elsewhere. It runs with Apify limited permissions (it can access only its own run storage).
- Roles: for any personal data in your input or output you are the controller and the Actor creator acts as processor, as set out in section 5 of the Apify Standard Actor Contract. Run logs contain only domain names, scores and error codes.
- Deletion: delete the run or dataset in Apify Console at any time. Questions or privacy requests: open an issue on the Actor's Issues tab.
Related Actors on the Apify Store
- Broken Link & Redirect Checker: crawl a site for broken links, redirect chains and missing #anchors.
- Domain Health Checker: domain expiry, DNS, SPF/DMARC/DKIM, SSL and HSTS in bulk.
- Bulk PageSpeed & Core Web Vitals Checker: Lighthouse scores, LCP, CLS, INP and TBT for many URLs on mobile and desktop, no API key needed.
- Website Content Crawler: crawl a site and export its text as Markdown, for example to build an
llms-full.txtor feed a RAG pipeline. - RAG Web Browser: fetch pages the way an AI agent does and get clean Markdown back.
- Google Search Results Scraper: track how your pages appear in Google results, including AI overviews.
- See all Tesselyn Actors on the Apify Store.
FAQ
What is llms.txt and does it matter?
llms.txt is a community proposal (2024) for a Markdown file at the site root that gives language models a curated map of a site's key content. It is not an official standard and AI search engines have not confirmed using it for ranking; it is cheap to add and used by some AI tools and agents. This Actor checks presence and basic structure (H1 title, summary, sections, links).
Which AI crawlers should I allow?
Search/answer crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) affect whether you can be found and cited in AI answers; training crawlers (GPTBot, CCBot, Google-Extended…) are an opt-out choice. The report shows each separately so you can decide.
Which bots matter for ChatGPT, Claude and Perplexity answers?
The search and answer bots. By default this Actor checks OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot. You can add other tokens, such as Claude-User or Perplexity-User, in the aiBots input. OpenAI and Anthropic describe GPTBot and ClaudeBot as their model-training crawlers, so blocking those is a training opt-out and is reported separately. Vendors change their crawlers, so check their current documentation before you edit robots.txt.
Is there a free version?
There is no separate free version. Apify's Free plan includes monthly platform credit you can use to try this Actor, and rows with an error field are not charged.
Does "allowed" mean the bot can really reach my site?
It means your robots.txt allows that user-agent at the site root. robots.txt is advisory; firewalls/CDNs may still block or allow bots regardless, and path-specific rules below the root are not evaluated. Rules are read with Python's standard robots.txt parser, which does not support * or $ wildcards inside paths, so a wildcard rule may be reported differently from how a search or AI crawler reads it. This Actor does not impersonate crawlers, so it cannot test CDN-level blocking.
Does it render JavaScript or crawl the whole site?
No. It analyzes the raw homepage HTML and the root-level files only. That keeps it fast, cheap and polite.
Why did a domain return an error?
See the error field: e.g. HOMEPAGE_HTTP_403 (bot protection), ROBOTS_TXT_UNREACHABLE:TIMEOUT, HOMEPAGE_DISALLOWED_BY_ROBOTS_FOR_THIS_AUDITOR, INVALID_OR_NON_PUBLIC_DOMAIN:…. Error rows are not charged.
Is it legal?
It only reads small public files that websites publish for crawlers (robots.txt, llms.txt, sitemap) and the public homepage, identifies itself, obeys robots.txt and extracts no contact data. You remain responsible for auditing only sites you have a right to audit (see Terms of use). This is not legal advice.
How fast / how polite is it?
Each domain gets ~5 sequential requests spaced ≥1 s apart (about 5–10 s per domain); different domains are processed in parallel with limited concurrency (default 5).
Support
Built and maintained by Tesselyn (a brand of HevraBest Group Ltd). Found a bug or a wrong result, or need a feature? Open an issue on the Actor's Issues tab. That is our support channel and the route Apify uses for credit compensation.
Failed runs and unexpected charges. Rows that the "How much does it cost?" section lists as free are not charged. Apify's small per-run "Actor start" charge, when it is listed on the Pricing tab, still applies. If you think a run charged you for work it didn't deliver, open an issue with the run link. If we confirm the problem, we ask Apify to compensate you. Apify gives compensation as platform credits, not money. Apify's General Terms (section 7.9) require charge disputes to be sent to Apify in writing within 15 days of the charge or invoice date, so please report problems promptly.