Agent Accessibility Auditor

Can an AI agent read this site?

Give it a domain. It returns one flat row with 42 fields covering five families of fact: the llms.txt family, robots AI crawler policy including the newer Content Signal directives, structured data presence and health, render mode, and machine readable endpoint discovery.

Every field is a fact read off a fetch. No model is called at any point, so the same domain returns the same row today and next month unless the site changed.

Who this is for

A technical SEO or growth engineer preparing a site for AI crawlers and agent traffic, or an agency selling that work and needing a before and after audit across a client list.

What it checks

Twelve requests per domain. robots.txt first, then the homepage and ten probes concurrently. Typical wall clock is 2 to 4 seconds.

The llms.txt family

Field	What it holds
has_llms_txt	True only when /llms.txt returns 200 and the body is real markdown
llms_txt_bytes	File size. Measured range on real sites: 6,780 to 207,885 bytes
llms_txt_reject_reason	Why a 200 was not counted
has_llms_full_txt	The long form companion
has_ai_txt	Same validity treatment

HTTP 200 is not enough on its own. A site with a catch-all route answers 200 for /llms.txt and hands back its normal page, byte identical to the homepage. A status only check marks every such site as publishing an llms.txt. The body is validated instead, and when it fails, llms_txt_reject_reason says why.

Robots AI policy

Field	What it holds
has_robots_txt	True only when /robots.txt returns 200 with a text body
robots_ai_agents_named	AI crawler agents with an explicit rule, matched against a 26 agent list
robots_ai_agent_count	How many
robots_ai_policy	allow, block, mixed, none, or null
content_signal_present	Whether a Content-Signal line exists
content_signal_search	The declared position on search indexing
content_signal_ai_input	The declared position on use as AI inference input
content_signal_ai_train	The declared position on use as AI training data

The 26 agent list covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, Meta-ExternalFetcher, cohere-ai, Amazonbot, Diffbot, omgili, FacebookBot, YouBot, Timpibot, ImagesiftBot, AI2Bot, DuckAssistBot and MistralAI-User.

Content Signal is the newer syntax and most tools do not read it. A reader that only knows named bots reports "no AI policy" for a site that has an explicit one. In the ten domain build sample, two sites published a Content Signal policy and five named AI bots, and only one site did both.

Structured data

jsonld_block_count, jsonld_parsed_count, jsonld_broken_count, jsonld_types, has_microdata, has_open_graph, has_canonical.

Parsed and broken are separate counts on purpose. A broken JSON-LD block is invisible to an agent, so a site with three blocks where one does not parse is not the same as a site with two clean blocks.

Render mode

html_bytes, text_bytes, text_ratio_pct, render_mode.

text_ratio_pct is the extracted text divided by the raw HTML, as a percentage. Measured range on real sites: 0.5 to 7.8 percent.

Ratio	render_mode
5.0 percent or more	server_rendered
2.0 to 5.0	mixed
below 2.0	client_rendered

These thresholds are arbitrary and are published so you can re-cut them. They were set against measurements of real pages: the known JavaScript shells land in client_rendered, the known readable content pages land in server_rendered, and marketing homepages sit in between.

Machine readable endpoints

endpoints_found, endpoints_soft_404, and six flags: has_sitemap, has_openapi, has_ai_plugin, has_mcp_manifest, has_security_txt, has_feed.

Probed paths: /sitemap.xml, /openapi.json, /.well-known/ai-plugin.json, /.well-known/mcp.json, /.well-known/security.txt, /feed, /rss.xml.

endpoints_soft_404 is the trap field. Some sites answer HTTP 200 with a normal page for any URL you ask for. One site in the build sample did it for five of the eleven paths, including /openapi.json and /ai.txt. Those paths are listed here rather than counted as present, and a long list is itself a finding: that site is hard for an agent to probe at all.

The score

agent_readable_score is 0 to 100. agent_readable_tier is open at 60 or more, partial at 35 to 59, closed at 15 to 34, unreadable below 15. score_reasons names every rule that fired with its points, so you can rebuild the number from the raw fields or throw it away and use the fields directly.

Signal	Points
has_llms_txt	25
jsonld_parsed_count is 1 or more	15
has_sitemap	10
content_signal_present	10
robots_ai_policy is allow or mixed	10
render_mode is server_rendered	8
render_mode is mixed	4
robots_ai_policy is none	5
has_llms_full_txt	5
has_ai_txt	5
jsonld_broken_count is 0 with at least one block	5
has_openapi or has_mcp_manifest or has_ai_plugin	5
has_open_graph	3
has_canonical	2
has_security_txt	2

The weights are arbitrary and are labeled so. They were set against one principle: llms.txt plus valid structured data plus a sitemap should on their own clear 50, because those three are what actually determine whether an agent can consume a site today.

False versus null, which is the whole discipline here

Twenty of the 42 fields are booleans and you will read every one of them as a verdict about a company. So the two values mean different things and they are never mixed up:

• false means the request completed and the answer was no.
• null means we could not ask.

robots_ai_policy: "none" means we read a valid robots.txt and no AI agent appears in it. robots_ai_policy: null means the robots.txt was unreadable, which is not the same thing at all. Same for jsonld_block_count: 0, which means we parsed the page and found no blocks, against null, which means we never got the page.

agent_readable_score is null, not 0, when fetch_status is blocked or unreachable. A site that refused us has no score. Check fetch_status before you conclude anything.

fetch_status	What it means
ok	Every request answered
partial	Some requests did not complete, or robots.txt told us not to make them
blocked	The site refused us. Every boolean and both verdicts are null
unreachable	The domain did not resolve. One summary row with is_summary_row: true

Input

Field	Type	Default	What it does
domain	string	required	One domain, for example vercel.com. Protocol and path are stripped
check_endpoints	boolean	true	Probes sitemap, OpenAPI, well known files and feeds. Adds 7 concurrent requests
check_structured_data	boolean	true	Parses JSON-LD, microdata, Open Graph and canonical off the homepage. Costs no extra requests
skipCache	select	false	Leave as false to use the 7 day cache

Results are cached for 7 days. That is a short TTL by design: llms.txt adoption and Content Signal adoption are both moving fast, and a before and after audit is defeated by a stale row. Only a fully answered row is cached, so a timeout is never served back to you as a result.

Known limits, stated plainly

• Render mode is measured on the homepage only. A marketing homepage can be almost entirely JavaScript while the documentation an agent actually wants is plain HTML. A client_rendered verdict describes the front door, not the whole site. One large payments company in the build sample reads at 2.1 percent, which reflects a heavy marketing homepage and not an unreadable site.
• The score is a composite of things we can check, not a complete picture of agent readability. The weights above are published so you can rebuild it or ignore it.
• A missing sitemap.xml at the root does not always mean there is no sitemap. One large company in the build sample serves its sitemap from a sub-path and declares it in robots.txt. We check the standard location and we also read the Sitemap: line in robots.txt, and endpoints_found records which route found it. Layouts that use neither will still read as absent.
• Some sites answer HTTP 200 to every URL you ask for. We detect that and list those paths in endpoints_soft_404 rather than counting them as present.
• A robots.txt that redirects to a normal web page is unreadable, not empty. That row comes back with a null AI policy, not "no policy". Check for nulls before concluding anything about a company's stance.
• Content Signal is new and adoption is low. Two of ten domains in the build sample publish one. Named AI crawler rules are more common at five of ten, and they are almost all publishers. Most companies have made no explicit choice at all, and robots_ai_policy: "none" is the honest way to say that.
• This actor honors robots.txt as well as reading it. If a site disallows a path for a generic crawler, we do not fetch it, and every field behind that request comes back null with robots_disallowed in score_reasons. A site that tells every crawler to stay out will therefore score low on very little data, which is the correct reading: it has told agents to stay out.
• We do not ask an AI whether a site is readable. We check facts. No model is called at any point.
• Every domain is billed, including blocked and unreachable ones. The requests were made either way.

A note on what we do with an ai-train=no declaration

A site declaring Content-Signal: ai-train=no is stating a preference about AI training. This actor records that declaration and does not act on it. Recording a policy is not violating it, and nothing here trains on anything.

No personal data

Every field is a property of a website: file presence, byte counts, directive values, markup types, ratios. Nothing describes a person, and there is no path in the design by which a person could enter the output.

Pricing

Pay per event. One event per domain analyzed.

Apify plan	Price per domain
Free	$0.005
Bronze	$0.00475
Silver	$0.0045
Gold	$0.00425

A 500 domain audit costs $2.50 on the free tier.

Output

One flat row per domain, Clay native: scalar fields, snake_case, no nested JSON.

{
  "domain": "vercel.com",
  "has_llms_txt": true,
  "llms_txt_bytes": 207885,
  "robots_ai_policy": "none",
  "content_signal_present": true,
  "content_signal_ai_train": "no",
  "jsonld_block_count": 1,
  "jsonld_broken_count": 0,
  "text_ratio_pct": 0.6,
  "render_mode": "client_rendered",
  "endpoints_found": "/sitemap.xml, /.well-known/security.txt",
  "endpoints_soft_404": "/openapi.json, /feed, /rss.xml, /llms-full.txt, /ai.txt",
  "agent_readable_score": 77,
  "agent_readable_tier": "open",
  "fetch_status": "ok"
}

Built by Mamba Labs.