# Agent Accessibility Auditor llms txt Robots AI Schema Check (`mambalabs/agent-accessibility-auditor`) Actor

Can an AI agent read this site. Checks llms.txt, robots AI crawler rules and the Content Signal policy, structured data, render mode, and machine readable endpoints like sitemap, OpenAPI and well known files. Every field is a fact read off a fetch, never a model opinion.

- **URL**: https://apify.com/mambalabs/agent-accessibility-auditor.md
- **Developed by:** [Mamba Labs](https://apify.com/mambalabs) (community)
- **Categories:** SEO tools, Automation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.25 / 1,000 domain analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Agent Accessibility Auditor

Can an AI agent read this site?

Give it a domain. It returns one flat row with 42 fields covering five families of fact: the `llms.txt` family, robots AI crawler policy including the newer Content Signal directives, structured data presence and health, render mode, and machine readable endpoint discovery.

**Every field is a fact read off a fetch.** No model is called at any point, so the same domain returns the same row today and next month unless the site changed.

### Who this is for

A technical SEO or growth engineer preparing a site for AI crawlers and agent traffic, or an agency selling that work and needing a before and after audit across a client list.

### What it checks

Twelve requests per domain. `robots.txt` first, then the homepage and ten probes concurrently. Typical wall clock is 2 to 4 seconds.

#### The llms.txt family

| Field | What it holds |
|---|---|
| `has_llms_txt` | True only when `/llms.txt` returns 200 **and** the body is real markdown |
| `llms_txt_bytes` | File size. Measured range on real sites: 6,780 to 207,885 bytes |
| `llms_txt_reject_reason` | Why a 200 was not counted |
| `has_llms_full_txt` | The long form companion |
| `has_ai_txt` | Same validity treatment |

HTTP 200 is not enough on its own. A site with a catch-all route answers 200 for `/llms.txt` and hands back its normal page, byte identical to the homepage. A status only check marks every such site as publishing an `llms.txt`. The body is validated instead, and when it fails, `llms_txt_reject_reason` says why.

#### Robots AI policy

| Field | What it holds |
|---|---|
| `has_robots_txt` | True only when `/robots.txt` returns 200 with a text body |
| `robots_ai_agents_named` | AI crawler agents with an explicit rule, matched against a 26 agent list |
| `robots_ai_agent_count` | How many |
| `robots_ai_policy` | `allow`, `block`, `mixed`, `none`, or null |
| `content_signal_present` | Whether a `Content-Signal` line exists |
| `content_signal_search` | The declared position on search indexing |
| `content_signal_ai_input` | The declared position on use as AI inference input |
| `content_signal_ai_train` | The declared position on use as AI training data |

The 26 agent list covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, Meta-ExternalFetcher, cohere-ai, Amazonbot, Diffbot, omgili, FacebookBot, YouBot, Timpibot, ImagesiftBot, AI2Bot, DuckAssistBot and MistralAI-User.

**Content Signal is the newer syntax and most tools do not read it.** A reader that only knows named bots reports "no AI policy" for a site that has an explicit one. In the ten domain build sample, two sites published a Content Signal policy and five named AI bots, and only one site did both.

#### Structured data

`jsonld_block_count`, `jsonld_parsed_count`, `jsonld_broken_count`, `jsonld_types`, `has_microdata`, `has_open_graph`, `has_canonical`.

Parsed and broken are separate counts on purpose. **A broken JSON-LD block is invisible to an agent**, so a site with three blocks where one does not parse is not the same as a site with two clean blocks.

#### Render mode

`html_bytes`, `text_bytes`, `text_ratio_pct`, `render_mode`.

`text_ratio_pct` is the extracted text divided by the raw HTML, as a percentage. Measured range on real sites: 0.5 to 7.8 percent.

| Ratio | `render_mode` |
|---|---|
| 5.0 percent or more | `server_rendered` |
| 2.0 to 5.0 | `mixed` |
| below 2.0 | `client_rendered` |

**These thresholds are arbitrary and are published so you can re-cut them.** They were set against measurements of real pages: the known JavaScript shells land in `client_rendered`, the known readable content pages land in `server_rendered`, and marketing homepages sit in between.

#### Machine readable endpoints

`endpoints_found`, `endpoints_soft_404`, and six flags: `has_sitemap`, `has_openapi`, `has_ai_plugin`, `has_mcp_manifest`, `has_security_txt`, `has_feed`.

Probed paths: `/sitemap.xml`, `/openapi.json`, `/.well-known/ai-plugin.json`, `/.well-known/mcp.json`, `/.well-known/security.txt`, `/feed`, `/rss.xml`.

`endpoints_soft_404` is the trap field. Some sites answer HTTP 200 with a normal page for any URL you ask for. One site in the build sample did it for five of the eleven paths, including `/openapi.json` and `/ai.txt`. Those paths are listed here rather than counted as present, and a long list is itself a finding: that site is hard for an agent to probe at all.

#### The score

`agent_readable_score` is 0 to 100. `agent_readable_tier` is `open` at 60 or more, `partial` at 35 to 59, `closed` at 15 to 34, `unreadable` below 15. `score_reasons` names every rule that fired with its points, so you can rebuild the number from the raw fields or throw it away and use the fields directly.

| Signal | Points |
|---|---|
| `has_llms_txt` | 25 |
| `jsonld_parsed_count` is 1 or more | 15 |
| `has_sitemap` | 10 |
| `content_signal_present` | 10 |
| `robots_ai_policy` is `allow` or `mixed` | 10 |
| `render_mode` is `server_rendered` | 8 |
| `render_mode` is `mixed` | 4 |
| `robots_ai_policy` is `none` | 5 |
| `has_llms_full_txt` | 5 |
| `has_ai_txt` | 5 |
| `jsonld_broken_count` is 0 with at least one block | 5 |
| `has_openapi` or `has_mcp_manifest` or `has_ai_plugin` | 5 |
| `has_open_graph` | 3 |
| `has_canonical` | 2 |
| `has_security_txt` | 2 |

**The weights are arbitrary and are labeled so.** They were set against one principle: `llms.txt` plus valid structured data plus a sitemap should on their own clear 50, because those three are what actually determine whether an agent can consume a site today.

### False versus null, which is the whole discipline here

Twenty of the 42 fields are booleans and you will read every one of them as a verdict about a company. So the two values mean different things and they are never mixed up:

- **`false`** means the request completed and the answer was no.
- **`null`** means we could not ask.

`robots_ai_policy: "none"` means we read a valid `robots.txt` and no AI agent appears in it. `robots_ai_policy: null` means the `robots.txt` was unreadable, which is not the same thing at all. Same for `jsonld_block_count: 0`, which means we parsed the page and found no blocks, against `null`, which means we never got the page.

**`agent_readable_score` is null, not 0, when `fetch_status` is `blocked` or `unreachable`.** A site that refused us has no score. Check `fetch_status` before you conclude anything.

| `fetch_status` | What it means |
|---|---|
| `ok` | Every request answered |
| `partial` | Some requests did not complete, or robots.txt told us not to make them |
| `blocked` | The site refused us. Every boolean and both verdicts are null |
| `unreachable` | The domain did not resolve. One summary row with `is_summary_row: true` |

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `domain` | string | required | One domain, for example `vercel.com`. Protocol and path are stripped |
| `check_endpoints` | boolean | true | Probes sitemap, OpenAPI, well known files and feeds. Adds 7 concurrent requests |
| `check_structured_data` | boolean | true | Parses JSON-LD, microdata, Open Graph and canonical off the homepage. Costs no extra requests |
| `skipCache` | select | `false` | Leave as `false` to use the 7 day cache |

Results are cached for 7 days. That is a short TTL by design: `llms.txt` adoption and Content Signal adoption are both moving fast, and a before and after audit is defeated by a stale row. Only a fully answered row is cached, so a timeout is never served back to you as a result.

### Known limits, stated plainly

- **Render mode is measured on the homepage only.** A marketing homepage can be almost entirely JavaScript while the documentation an agent actually wants is plain HTML. A `client_rendered` verdict describes the front door, not the whole site. One large payments company in the build sample reads at 2.1 percent, which reflects a heavy marketing homepage and not an unreadable site.
- **The score is a composite of things we can check, not a complete picture of agent readability.** The weights above are published so you can rebuild it or ignore it.
- **A missing `sitemap.xml` at the root does not always mean there is no sitemap.** One large company in the build sample serves its sitemap from a sub-path and declares it in `robots.txt`. We check the standard location and we also read the `Sitemap:` line in `robots.txt`, and `endpoints_found` records which route found it. Layouts that use neither will still read as absent.
- **Some sites answer HTTP 200 to every URL you ask for.** We detect that and list those paths in `endpoints_soft_404` rather than counting them as present.
- **A `robots.txt` that redirects to a normal web page is unreadable, not empty.** That row comes back with a null AI policy, not "no policy". Check for nulls before concluding anything about a company's stance.
- **Content Signal is new and adoption is low.** Two of ten domains in the build sample publish one. Named AI crawler rules are more common at five of ten, and they are almost all publishers. Most companies have made no explicit choice at all, and `robots_ai_policy: "none"` is the honest way to say that.
- **This actor honors `robots.txt` as well as reading it.** If a site disallows a path for a generic crawler, we do not fetch it, and every field behind that request comes back null with `robots_disallowed` in `score_reasons`. A site that tells every crawler to stay out will therefore score low on very little data, which is the correct reading: it has told agents to stay out.
- **We do not ask an AI whether a site is readable. We check facts.** No model is called at any point.
- **Every domain is billed, including blocked and unreachable ones.** The requests were made either way.

### A note on what we do with an ai-train=no declaration

A site declaring `Content-Signal: ai-train=no` is stating a preference about AI training. This actor **records** that declaration and does not act on it. Recording a policy is not violating it, and nothing here trains on anything.

### No personal data

Every field is a property of a website: file presence, byte counts, directive values, markup types, ratios. Nothing describes a person, and there is no path in the design by which a person could enter the output.

### Pricing

Pay per event. One event per domain analyzed.

| Apify plan | Price per domain |
|---|---|
| Free | $0.005 |
| Bronze | $0.00475 |
| Silver | $0.0045 |
| Gold | $0.00425 |

A 500 domain audit costs $2.50 on the free tier.

### Output

One flat row per domain, Clay native: scalar fields, snake\_case, no nested JSON.

```json
{
  "domain": "vercel.com",
  "has_llms_txt": true,
  "llms_txt_bytes": 207885,
  "robots_ai_policy": "none",
  "content_signal_present": true,
  "content_signal_ai_train": "no",
  "jsonld_block_count": 1,
  "jsonld_broken_count": 0,
  "text_ratio_pct": 0.6,
  "render_mode": "client_rendered",
  "endpoints_found": "/sitemap.xml, /.well-known/security.txt",
  "endpoints_soft_404": "/openapi.json, /feed, /rss.xml, /llms-full.txt, /ai.txt",
  "agent_readable_score": 77,
  "agent_readable_tier": "open",
  "fetch_status": "ok"
}
```

Built by [Mamba Labs](https://apify.com/mambalabs).

# Actor input Schema

## `domain` (type: `string`):

One company domain, for example vercel.com. Protocol and path are stripped.

## `check_endpoints` (type: `boolean`):

Probes sitemap, OpenAPI, well known files and feeds. Adds 7 concurrent requests.

## `check_structured_data` (type: `boolean`):

Parses JSON-LD, microdata, Open Graph and canonical off the homepage. Costs no extra requests.

## `skipCache` (type: `string`):

Leave as false to use the 7 day cache. Set to true to re-audit the domain from scratch.

## Actor input object example

```json
{
  "domain": "vercel.com",
  "check_endpoints": true,
  "check_structured_data": true,
  "skipCache": "false"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domain": "vercel.com",
    "check_endpoints": true,
    "check_structured_data": true,
    "skipCache": "false"
};

// Run the Actor and wait for it to finish
const run = await client.actor("mambalabs/agent-accessibility-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domain": "vercel.com",
    "check_endpoints": True,
    "check_structured_data": True,
    "skipCache": "false",
}

# Run the Actor and wait for it to finish
run = client.actor("mambalabs/agent-accessibility-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domain": "vercel.com",
  "check_endpoints": true,
  "check_structured_data": true,
  "skipCache": "false"
}' |
apify call mambalabs/agent-accessibility-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mambalabs/agent-accessibility-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/anxbRv0lKrpQ1pnua/builds/npTWNbyaCNfwrRjpf/openapi.json