Agent Readiness Audit

AI agents are becoming a major consumer of the web. They fetch pages, read robots.txt, look for llms.txt, extract structured data, and increasingly connect through MCP servers. Most websites were never checked against any of that.

This Actor audits any website for AI-agent readiness and returns a scored report with concrete recommendations.

What it checks

Check	Why it matters for agents
robots.txt exists	The only standard place to express a crawler policy
AI crawler access	Whether GPTBot, ClaudeBot, PerplexityBot, CCBot and others are allowed, restricted, or blocked
llms.txt	A markdown site map made for language models (llmstxt.org), consumed directly by agents
XML sitemap	Lets agents enumerate content instead of crawling blind
MCP server card	/.well-known/mcp/server-card.json, the discovery point for a site's MCP server
Structured data	JSON-LD (schema.org) lets agents extract facts without guessing from layout
Meta basics	Title, meta description, canonical: the first thing any agent reads
Works without JavaScript	Most agents fetch, they do not render; content that only exists after JS is invisible to them
Indexing signals	noindex / noai directives in meta tags or X-Robots-Tag headers

Each check has a weight. The report gives a 0–100 score, a grade (A–F), per-check details, and a recommendation for every failed check.

Input

{
    "urls": ["https://example.com"],
    "aiCrawlers": ["GPTBot", "ClaudeBot", "PerplexityBot", "CCBot"]
}

urls is required. aiCrawlers is optional and defaults to the major AI agents and training crawlers.

Output

One dataset item per audited site:

{
    "url": "https://example.com",
    "score": 62,
    "grade": "C",
    "checks": {
        "robotsTxt": { "passed": true, "weight": 10, "status": 200 },
        "aiCrawlerAccess": { "passed": true, "weight": 15, "policies": { "GPTBot": "allowed", "ClaudeBot": "blocked" } },
        "llmsTxt": { "passed": false, "weight": 15, "status": 404 },
        "worksWithoutJs": { "passed": true, "weight": 15, "visibleTextChars": 18240 }
    },
    "recommendations": [
        "Add /llms.txt: a short markdown map of the site for language models (llmstxt.org). Cheap to add, directly consumed by agents."
    ]
}

A run summary is stored in the OUTPUT record of the key-value store.

Use cases

• Site owners: find out what an AI agent actually sees before deciding on an AI traffic policy.
• Agencies: audit client portfolios for the answer-engine era, the same way you audit SEO today.
• Agent builders: check whether a target site is agent-friendly before building on top of it.

Notes

• The audit reads a handful of public URLs per site (homepage, robots.txt, llms.txt, sitemap, well-known paths). It does not crawl the whole site and does not execute JavaScript.
• "Blocked" verdicts describe what robots.txt requests; they are not a legal statement about what any crawler does.