llms.txt Checker — AI Readiness Audit for any website

Is your site — or your client's — visible to AI assistants? This Actor runs a complete AI readiness audit in one pass: it checks for llms.txt (the emerging standard that tells AI assistants what your site is about), verifies whether the major AI crawlers are allowed or blocked in robots.txt, confirms llms-full.txt and sitemap.xml, and returns a clear 0–100 AI readiness score with a plain-English list of exactly what to fix.

Built for SEO and content teams tracking AI visibility, and for agencies selling AI-readiness audits — feed in a client list, get back scored, explainable results you can put straight into a report.

~$5 per 1,000 sites — competitors charge $10–100+ per 1,000, often with per-recommendation surcharges. Here the actionable reasons list is part of every result.

Use with AI agents (MCP)

This Actor is also an MCP tool — AI agents like Claude and Cursor can run it for you. Your agent can call this tool directly; usage bills through your Apify account.

MCP server URL:

https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker

Claude Desktop (claude_desktop_config.json):

{
    "mcpServers": {
        "llms-txt-checker": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker",
                "--header",
                "Authorization: Bearer YOUR_APIFY_TOKEN"
            ]
        }
    }
}

Cursor (.cursor/mcp.json):

{
    "mcpServers": {
        "llms-txt-checker": {
            "url": "https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker",
            "headers": {
                "Authorization": "Bearer YOUR_APIFY_TOKEN"
            }
        }
    }
}

Replace YOUR_APIFY_TOKEN with your API token from Apify Console.

No Apify account? AI agents can also pay per call with USDC via the x402 protocol: buy a prepaid token at agi.apify.com and use it as the Bearer token above.

What gets checked

1. llms.txt — does it exist? Is it well-formed (title heading, sections, links)? Any syntax problems are listed. Sites that return an HTML error page instead of a real file are correctly counted as not having one.
2. llms-full.txt — the companion full-content file.
3. robots.txt AI crawler policy — per-crawler status (allowed / blocked / partially blocked) for GPTBot (ChatGPT), ClaudeBot / Claude-Web (Claude), PerplexityBot, Google-Extended (Gemini), CCBot (Common Crawl), and Bytespider (TikTok) — with whether that comes from an explicit rule or the default * policy.
4. sitemap.xml — fetched directly and via robots.txt Sitemap: declarations.
5. AI readiness score (0–100) — llms.txt quality (up to 40), llms-full.txt (10), AI crawler access (30), sitemap (10), robots.txt present (10) — each point explained in the reasons list.

Input

{
    "urls": ["https://docs.anthropic.com", "https://www.irishtimes.com"],
    "maxConcurrency": 10
}

Output

One item per site:

{
    "url": "https://www.irishtimes.com",
    "aiReadinessScore": 33,
    "llmsTxt": { "exists": false, "url": "https://www.irishtimes.com/llms.txt" },
    "llmsFullTxt": { "exists": false },
    "robotsTxt": {
        "exists": true,
        "aiCrawlers": [
            { "crawler": "GPTBot", "status": "blocked", "source": "explicit rule" },
            { "crawler": "ClaudeBot", "status": "partially blocked", "source": "default (*) policy" }
        ]
    },
    "sitemap": { "exists": true },
    "reasons": [
        "Missed 40: no llms.txt file — this is the single biggest AI-readiness fix (create one at /llms.txt).",
        "+13 of 30: robots.txt blocks GPTBot — blocked crawlers cannot index the site for AI answers.",
        "+10: sitemap.xml is present."
    ],
    "checkedAt": "2026-07-19T19:27:14.733Z"
}

Unreachable sites produce an item with an error field — a batch never dies mid-run, and output always reconciles 1:1 with input.

Why it's fast and cheap

No headless browser — just four lightweight text fetches per site. A 1,000-site audit finishes in minutes and costs cents in compute.

FAQ

What is llms.txt? A proposed standard (llmstxt.org) — a markdown file at /llms.txt giving AI assistants a curated guide to your site, like robots.txt for the AI era. Sites from Stripe to Shopify to WordPress.org have already adopted it.

Why does AI crawler access matter for SEO? If GPTBot or ClaudeBot is blocked, your content can't be cited by ChatGPT or Claude — an increasingly large source of referral traffic. The audit shows each crawler's status so blocking is a decision, not an accident.

Does a high score guarantee AI visibility? No tool can promise that — the score measures the technical foundations you control: discoverability files and crawler access.

More tools from MCP Factory

• Tech Stack Detector — find out what any website is built with (CMS, ecommerce, frameworks, analytics) at ~$5 per 1,000 sites.
• EU VAT Validator — bulk-validate EU VAT numbers against the EU's official VIES service, with the official consultation number as audit evidence on every check.