AI Readiness Checker: GPTBot, ClaudeBot, llms.txt & AI SEO
Pricing
from $1.40 / 1,000 website checkeds
AI Readiness Checker: GPTBot, ClaudeBot, llms.txt & AI SEO
Check if AI search can read and cite a website: robots.txt rules for 17 AI bots (GPTBot, ClaudeBot, PerplexityBot...), llms.txt, schema.org, a score and fixes.
Pricing
from $1.40 / 1,000 website checkeds
Rating
0.0
(0)
Developer
Open Data Tools
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
10 hours ago
Last modified
Categories
Features
Share
Give a list of websites, see if ChatGPT, Claude, Perplexity, Gemini and other AI assistants can read and cite each one. The checker reads the robots.txt rules for 17 AI bots (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended…), looks for llms.txt and llms-full.txt, a sitemap, schema.org structured data and the text readable without JavaScript, then gives a score out of 100, a grade and the fixes in plain words. Fast and cheap: $2.00 per 1,000 websites.
What you get
One row per website:
- Each AI bot: company, purpose (training, search or user request), allowed or blocked, and whether robots.txt names it.
- Blocked bots, split into AI search and assistants (the ones that cite you) and training crawlers.
- llms.txt: present or not, its title and number of links; llms-full.txt.
- Sitemap, schema.org types, title, meta description, words readable without JavaScript, noai meta tags.
- Score, grade and fixes, worst first.
Use cases
- AI SEO (GEO): find why a site is never cited by ChatGPT, Claude or Perplexity.
- Agencies: audit clients and prospects in bulk and sell the fixes.
- Publishers and brands: check that robots.txt blocks AI training but still lets AI search cite you.
- Market research: measure how many sites of a sector block AI bots or publish a llms.txt.
How to use
- Without changing anything: apify.com, nytimes.com and stripe.com.
- Websites: one per line, up to 10,000 per run. Without
http://orhttps://, the check starts fromhttps://.
Input example
{"websites": ["apify.com", "nytimes.com", "stripe.com"]}
Output example
{"recordType": "website","id": "https://www.anthropic.com/","url": "https://www.anthropic.com/","input": "www.anthropic.com","site": "www.anthropic.com","robotsTxtFound": true,"allowedBotCount": 17,"blockedBots": [],"searchBotsBlocked": [],"trainingBotsBlocked": [],"blocksAllAi": false,"aiBots": [{"bot": "GPTBot","company": "OpenAI","purpose": "training","allowed": true,"namedInRobots": false}],"hasLlmsTxt": false,"llmsTxtTitle": null,"llmsTxtLinkCount": 0,"hasLlmsFullTxt": false,"hasSitemap": true,"homeReadable": true,"title": "Home \\ Anthropic","metaDescription": "Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.","structuredDataTypes": [],"serverRenderedWords": 310,"noAiMeta": false,"score": 75,"grade": "B","issues": ["No /llms.txt: AI assistants get no summary or map of the site","No structured data (schema.org JSON-LD) to describe the business to AI"],"status": "ok","sourceName": "robots.txt, llms.txt and home page of www.anthropic.com","sourceUrl": "https://www.anthropic.com/robots.txt","licence": "Public files a site publishes for robots; page text is not kept","sourceUpdatedAt": null,"actorVersion": "0.1.1","retrievedAt": "2026-11-19T12:41:05.512Z"}
aiBots lists the 17 bots; one is shown here. On nytimes.com, 15 of the 17 bots are blocked, AI search included, and the score is 30 (F).
How the score works
The score starts at 100 and each issue removes points: home page not readable 30; AI search or assistant bots blocked 25; fewer than 150 words without JavaScript 20; no llms.txt 15; no structured data, no sitemap or no title 10; no meta description or a noai tag 5. Blocking training bots only is a choice, not a fault: it is reported but costs no points. Grades: A from 90, B from 75, C from 60, D from 40, F below.
Pricing
Pay per event: $2.00 per 1,000 websites checked (website-checked, $0.002 each), plus Apify's start fee of $0.00005 per run. A website that does not answer and text that is not a website give a free error row.
| Websites | Price |
|---|---|
| 100 websites | $0.20 |
| 1,000 websites | $2.00 |
| 10,000 websites | $20 |
Data source
Files each site publishes for robots: robots.txt, llms.txt, llms-full.txt, sitemap.xml, and the HTML of the home page. Robots.txt rules are read the way each AI bot reads them (its own group, else the * group). Five requests per site at most, robots.txt respected.
Is it legal?
These files are published so that robots read them. The checker sends a few ordinary requests per site, waits between them, follows robots.txt, never visits private addresses and keeps no page text.
FAQ
- What is llms.txt? A Markdown file at the root of a site that tells AI assistants what the site is and where its key pages are, like a sitemap written for language models.
- Why is "the home page could not be read"? The site blocks robots (robots.txt or a bot protection) or answers with an error: AI crawlers likely meet the same wall.
- Does it run JavaScript? No, on purpose: most AI crawlers do not, so the words count shows what they actually see.
Related Actors
- Website to Markdown: Fast Crawler for LLM & RAG (No Browser)
- AI Model Prices: Compare 400+ LLM Prices and Features
- llms.txt Generator: AI-Ready Site Map for Any Website
- Structured Data Extractor: Schema.org, JSON-LD & Open Graph
- Sitemap Extractor: All URLs of a Site, Lastmod & Priority
Every Actor here also works as a tool for AI agents (Claude, ChatGPT, Cursor…) through Apify’s MCP server.
Changelog
See CHANGELOG.md.
En français
Votre site est-il lisible par ChatGPT, Claude et Perplexity ?
Donnez des sites, obtenez pour chacun l'accès de 17 robots d'IA d'après robots.txt, la présence d'un llms.txt, du sitemap et des données structurées, le texte lisible sans JavaScript, une note sur 100 et les corrections en clair.
Prix
2,00 $ les 1 000 sites. Les lignes d'erreur sont gratuites.
Source et légalité
Les fichiers que chaque site publie pour les robots et sa page d'accueil. Quelques requêtes par site, robots.txt respecté, aucun texte gardé.