Agent Accessibility Auditor llms txt Robots AI Schema Check

🔎 What can Agent Accessibility Auditor do?

Can an AI agent read this site?

Give it a domain. It returns one flat row with 42 fields covering five families of fact: the llms.txt family, robots AI crawler policy including the newer Content Signal directives, structured data presence and health, render mode, and machine readable endpoint discovery.

Every field is a fact read off a fetch. No model is called at any point, so the same domain returns the same row today and next month unless the site changed.

📦 What you get	⚙️ Features and integrations
📄 llms.txt family, presence and real byte size 🤖 Robots AI policy, including Content Signal directives 🧩 Structured data, render mode and endpoints 🧾 42 flat fields and one 0 to 100 score, snake_case	⚡ Twelve requests per domain, 2 to 4 seconds typical 🎯 checks array, pay only for the parts you asked for 🧊 7 day cache, with a skipCache override ⬇️ Export to JSON, CSV, Excel, HTML or XML

Bought by technical SEO and growth engineers preparing a site for AI crawlers and agent traffic, and by agencies selling that work who need a before and after audit across a client list.

🚫 This actor does not ask an AI whether a site is readable, and it is not an SEO rank tracker. Every field is a fact read off a fetch, so the same domain returns the same row today and next month unless the site changed. It does not score your search rankings, audit your content, or tell you what to write.

👥 Who this is for

A technical SEO or growth engineer preparing a site for AI crawlers and agent traffic, or an agency selling that work and needing a before and after audit across a client list.

📋 What data can Agent Accessibility Auditor extract?

Twelve requests per domain. robots.txt first, then the homepage and ten probes concurrently. Typical wall clock is 2 to 4 seconds.

The llms.txt family

Field	What it holds
has_llms_txt	True only when /llms.txt returns 200 and the body is real markdown
llms_txt_bytes	File size. Measured range on real sites: 6,780 to 207,885 bytes
llms_txt_reject_reason	Why a 200 was not counted
has_llms_full_txt	The long form companion
has_ai_txt	Same validity treatment

HTTP 200 is not enough on its own. A site with a catch-all route answers 200 for /llms.txt and hands back its normal page, byte identical to the homepage. A status only check marks every such site as publishing an llms.txt. The body is validated instead, and when it fails, llms_txt_reject_reason says why.

Robots AI policy

Field	What it holds
has_robots_txt	True only when /robots.txt returns 200 with a text body
robots_ai_agents_named	AI crawler agents with an explicit rule, matched against a 26 agent list
robots_ai_agent_count	How many
robots_ai_policy	allow, block, mixed, none, or null
content_signal_present	Whether a Content-Signal line exists
content_signal_search	The declared position on search indexing
content_signal_ai_input	The declared position on use as AI inference input
content_signal_ai_train	The declared position on use as AI training data

The 26 agent list covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, Meta-ExternalFetcher, cohere-ai, Amazonbot, Diffbot, omgili, FacebookBot, YouBot, Timpibot, ImagesiftBot, AI2Bot, DuckAssistBot and MistralAI-User.

Content Signal is the newer syntax and most tools do not read it. A reader that only knows named bots reports "no AI policy" for a site that has an explicit one. In the ten domain build sample, two sites published a Content Signal policy and five named AI bots, and only one site did both.

Structured data

jsonld_block_count, jsonld_parsed_count, jsonld_broken_count, jsonld_types, has_microdata, has_open_graph, has_canonical.

Parsed and broken are separate counts on purpose. A broken JSON-LD block is invisible to an agent, so a site with three blocks where one does not parse is not the same as a site with two clean blocks.

Render mode

html_bytes, text_bytes, text_ratio_pct, render_mode.

text_ratio_pct is the extracted text divided by the raw HTML, as a percentage. Measured range on real sites: 0.5 to 7.8 percent.

Ratio	render_mode
5.0 percent or more	server_rendered
2.0 to 5.0	mixed
below 2.0	client_rendered

These thresholds are arbitrary and are published so you can re-cut them. They were set against measurements of real pages: the known JavaScript shells land in client_rendered, the known readable content pages land in server_rendered, and marketing homepages sit in between.

Machine readable endpoints

endpoints_found, endpoints_soft_404, and six flags: has_sitemap, has_openapi, has_ai_plugin, has_mcp_manifest, has_security_txt, has_feed.

Probed paths: /sitemap.xml, /openapi.json, /.well-known/ai-plugin.json, /.well-known/mcp.json, /.well-known/security.txt, /feed, /rss.xml.

endpoints_soft_404 is the trap field. Some sites answer HTTP 200 with a normal page for any URL you ask for. One site in the build sample did it for five of the eleven paths, including /openapi.json and /ai.txt. Those paths are listed here rather than counted as present, and a long list is itself a finding: that site is hard for an agent to probe at all.

The score

agent_readable_score is 0 to 100. agent_readable_tier is open at 60 or more, partial at 35 to 59, closed at 15 to 34, unreadable below 15. score_reasons names every rule that fired with its points, so you can rebuild the number from the raw fields or throw it away and use the fields directly.

Signal	Points
has_llms_txt	25
jsonld_parsed_count is 1 or more	15
has_sitemap	10
content_signal_present	10
robots_ai_policy is allow or mixed	10
render_mode is server_rendered	8
render_mode is mixed	4
robots_ai_policy is none	5
has_llms_full_txt	5
has_ai_txt	5
jsonld_broken_count is 0 with at least one block	5
has_openapi or has_mcp_manifest or has_ai_plugin	5
has_open_graph	3
has_canonical	2
has_security_txt	2

The weights are arbitrary and are labeled so. They were set against one principle: llms.txt plus valid structured data plus a sitemap should on their own clear 50, because those three are what actually determine whether an agent can consume a site today.

⚠️ False versus null, which is the whole discipline here

⚠️ How to read these values. Twenty of the 42 fields are booleans and you will read every one of them as a verdict about a company. false means the request completed and the answer was no. null means we could not ask. agent_readable_score is null, not 0, when fetch_status is blocked or unreachable: a site that refused us has no score. Check fetch_status before you conclude anything.

Twenty of the 42 fields are booleans and you will read every one of them as a verdict about a company. So the two values mean different things and they are never mixed up:

• false means the request completed and the answer was no.
• null means we could not ask.

robots_ai_policy: "none" means we read a valid robots.txt and no AI agent appears in it. robots_ai_policy: null means the robots.txt was unreadable, which is not the same thing at all. Same for jsonld_block_count: 0, which means we parsed the page and found no blocks, against null, which means we never got the page.

agent_readable_score is null, not 0, when fetch_status is blocked or unreachable. A site that refused us has no score. Check fetch_status before you conclude anything.

fetch_status	What it means
ok	Every request answered
partial	Some requests did not complete, or robots.txt told us not to make them
blocked	The site refused us. Every boolean and both verdicts are null
unreachable	The domain did not resolve. One summary row with is_summary_row: true

⌨️ Input

Field	Type	Default	What it does
domain	string	required	One domain, for example vercel.com. Protocol and path are stripped
checks	array	all	Run only these checks. See below.
check_endpoints	boolean	true	Alias for the checks array: false removes sitemap, openapi, security_txt and feeds. Ignored when checks is set
check_structured_data	boolean	true	Alias for the checks array: false removes json_ld, microdata, open_graph and canonical. Ignored when checks is set
skipCache	select	false	Leave as false to use the 7 day cache

Results are cached for 7 days. That is a short TTL by design: llms.txt adoption and Content Signal adoption are both moving fast, and a before and after audit is defeated by a stale row. Only a fully answered row is cached, so a timeout is never served back to you as a result.

Naming the checks you want

checks runs only the parts of the audit you asked for. Before this existed the actor took two coarse booleans and a caller who wanted an llms.txt check paid for the whole audit.

{ "domain": "stripe.com", "checks": ["llms_txt"] }

Check	What it covers
llms_txt	/llms.txt, /llms-full.txt, /ai.txt
robots_ai	robots.txt AI agent directives and the Content-Signal declaration
sitemap	/sitemap.xml
openapi	/openapi.json, /.well-known/ai-plugin.json, /.well-known/mcp.json
security_txt	/.well-known/security.txt
feeds	/feed and /rss.xml
json_ld	JSON-LD blocks, parse health and types
microdata	Microdata markup
open_graph	Open Graph tags
canonical	Canonical URL
render_mode	HTML to text ratio and the server versus client rendering verdict

Three things this setting does, and all three matter:

• It narrows the work, not just the answer. An llms_txt only audit makes three requests instead of ten.
• A check you did not run reports null, never false. That is the same discipline the rest of this actor keeps: false is a claim that we looked.
• The score is rescaled over the checks you selected. Without that the setting would be a trap. A site with a perfect llms.txt, audited for llms.txt alone, would score 35 out of 100 and land in the partial tier because the other 65 points came from checks nobody asked to run. Instead it scores 100, and score_reasons carries a scored_out_of: note so you can see the denominator. A check set that can earn no points at all (feeds, microdata) scores null rather than zero.

The two old booleans still work and still mean what they meant. checks wins when both are sent. checks_requested on every row records what was run.

⚠️ Known limits

• Render mode is measured on the homepage only. A marketing homepage can be almost entirely JavaScript while the documentation an agent actually wants is plain HTML. A client_rendered verdict describes the front door, not the whole site. One large payments company in the build sample reads at 2.1 percent, which reflects a heavy marketing homepage and not an unreadable site.
• The score is a composite of things we can check, not a complete picture of agent readability. The weights above are published so you can rebuild it or ignore it.
• A missing sitemap.xml at the root does not always mean there is no sitemap. One large company in the build sample serves its sitemap from a sub-path and declares it in robots.txt. We check the standard location and we also read the Sitemap: line in robots.txt, and endpoints_found records which route found it. Layouts that use neither will still read as absent.
• Some sites answer HTTP 200 to every URL you ask for. We detect that and list those paths in endpoints_soft_404 rather than counting them as present.
• A robots.txt that redirects to a normal web page is unreadable, not empty. That row comes back with a null AI policy, not "no policy". Check for nulls before concluding anything about a company's stance.
• Content Signal is new and adoption is low. Two of ten domains in the build sample publish one. Named AI crawler rules are more common at five of ten, and they are almost all publishers. Most companies have made no explicit choice at all, and robots_ai_policy: "none" is the honest way to say that.
• This actor honors robots.txt as well as reading it. If a site disallows a path for a generic crawler, we do not fetch it, and every field behind that request comes back null with robots_disallowed in score_reasons. A site that tells every crawler to stay out will therefore score low on very little data, which is the correct reading: it has told agents to stay out.
• We do not ask an AI whether a site is readable. We check facts. No model is called at any point.
• Every domain is billed, including blocked and unreachable ones. The requests were made either way.

A note on what we do with an ai-train=no declaration

A site declaring Content-Signal: ai-train=no is stating a preference about AI training. This actor records that declaration and does not act on it. Recording a policy is not violating it, and nothing here trains on anything.

No personal data

Every field is a property of a website: file presence, byte counts, directive values, markup types, ratios. Nothing describes a person, and there is no path in the design by which a person could enter the output.

💵 How much does it cost to audit a site for AI agents?

Pay per event. One event per domain analyzed.

Apify plan	Price per domain
Free	$0.005
Bronze	$0.00475
Silver	$0.0045
Gold	$0.00425

A 500 domain audit costs $2.50 on the free tier.

💳 What you are billed for. One event per domain analyzed, and every domain is billed, including blocked and unreachable ones, because the requests were made either way. The checks array does not reduce the per domain price: it reduces the work, not the event. Results are cached for 7 days and a cached row is not re-billed.

📤 Output

Exports to JSON, CSV, Excel, HTML or XML. One flat row per domain, Clay native: scalar fields, snake_case, no nested JSON.

{
  "domain": "vercel.com",
  "has_llms_txt": true,
  "llms_txt_bytes": 207885,
  "robots_ai_policy": "none",
  "content_signal_present": true,
  "content_signal_ai_train": "no",
  "jsonld_block_count": 1,
  "jsonld_broken_count": 0,
  "text_ratio_pct": 0.6,
  "render_mode": "client_rendered",
  "endpoints_found": "/sitemap.xml, /.well-known/security.txt",
  "endpoints_soft_404": "/openapi.json, /feed, /rss.xml, /llms-full.txt, /ai.txt",
  "agent_readable_score": 77,
  "agent_readable_tier": "open",
  "fetch_status": "ok"
}

💡 Tips

• Check fetch_status before you read any boolean. A blocked row has a null score and null verdicts, and reading those as "no" is the single easiest way to misuse this actor.
• Use the checks array when you only want one family. An llms.txt sweep does not need the render mode probe.
• Run it before and after the work. The 7 day cache is deliberately short so a before and after audit is not defeated by a stale row. Set skipCache when you want the after row immediately.
• Read endpoints_soft_404 before you trust endpoints_found. Some sites answer 200 to every path, and this actor separates the two rather than counting them as present.
• Treat robots_ai_policy: "none" as a real finding. Most companies have made no explicit choice at all. That is information about the account, not a gap in the data.

❓ FAQ

Does this call an AI model to judge my site? No. No model is called at any point. Every field is a fact read off a fetch, which is why the same domain returns the same row unless the site changed.

What is the difference between false and null? false means the request completed and the answer was no. null means we could not ask. They are never mixed up, and agent_readable_score is null rather than 0 on a blocked or unreachable domain.

Do you respect robots.txt? Yes, this actor honors it as well as reading it. If a site disallows a path for a generic crawler, we do not fetch it, and every field behind that request comes back null with robots_disallowed in score_reasons.

What do you do with a Content-Signal: ai-train=no declaration? We record it and do not act on it. Recording a policy is not violating it, and nothing here trains on anything.

Why is a big company scoring low on render mode? Render mode is measured on the homepage only. A marketing homepage can be almost entirely JavaScript while the documentation an agent actually wants is plain HTML. The verdict describes the front door, not the whole site.

Am I billed for a domain that blocked me? Yes. The requests were made either way, and the row tells you plainly what happened.

🧩 Want other GTM data?

Mamba Labs builds a fleet of GTM enrichment actors that share one flat, Clay-ready output convention, so their rows join on company_domain with no cleaning step:

🤖 AI Tooling Detector	📡 B2B Buying Signals Aggregator
🚀 Prospect Engine	📝 Publishing Frequency Tracker
⏩ Sequencer Lead Push	🔄 Company Change-Event Feed
🧭 Company Discovery List Builder	🏢 Company Firmographic Enricher
🪪 Company Identity Resolver	🌐 Company Social Presence Mapper
🏷️ Contact Classifier	📬 Domain Deliverability Checker
🔗 Domain to LinkedIn URL Resolver	✉️ Work Email Waterfall Finder
🎪 Event Presence Index	💰 Funding and Press Signal Scanner
🧑‍💼 GTM Hiring Signal Scraper	📋 Job Posting Monitor
🧱 Tech Stack Detector	🎯 ICP Fit Scorer
🔑 Job Board Keyword Scanner	⚖️ Legal Entity Resolver
💼 LinkedIn Company Page Mapper	💬 LinkedIn Post Tracker and Comment Capture
📸 Instagram and Facebook Brand Mapper	📮 Outbound Stack Detector
📄 Page Finder and Extractor	👤 People Finder and Email Verifier
📌 Pinterest Brand Presence Mapper	🏛️ Government Contract Award Monitor
📅 Public Company Reporting Window Finder	👥 Team Page People Extractor
🎵 TikTok Brand Presence Mapper	🏅 Workplace Program Detector
▶️ YouTube Channel Stats Extractor

Every actor in the suite takes a domain or a company and returns one flat row, so they stack in the same Clay table without reshaping anything.

🛠️ Need something custom built for you or your team? Tell us what you are trying to find and we will build it. Talk to Mamba Labs.

🆘 Support

Something wrong, or a domain the actor reads incorrectly? Open an issue on the Issues tab with the domain and the row, and we will look at it.

ℹ️ Sourcing and legal. Every field is a property of a website: file presence, byte counts, directive values, markup types and ratios. Nothing describes a person, and there is no path in the design by which a person could enter the output. Every fetch honors robots.txt. A site declaring Content-Signal: ai-train=no has that declaration recorded, not acted on, and nothing here trains on anything. You are responsible for how you use the output.

Built by Mamba Labs.