# AI Crawler & llms.txt Checker – GEO / AI Search Audit (`tesselyn/ai-search-readiness-auditor`) Actor

Check which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more) a site's robots.txt allows or blocks, validate llms.txt, sitemap & Schema.org JSON-LD, and get a 0-100 GEO readiness score with fix codes. Bulk, no login. $2 per 100 domains.

- **URL**: https://apify.com/tesselyn/ai-search-readiness-auditor.md
- **Developed by:** [Tesselyn](https://apify.com/tesselyn) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 domain auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Check in seconds whether ChatGPT, Claude, Perplexity and Google's AI can crawl a website.** This AI crawler checker and llms.txt checker reads each site's `robots.txt` rules for 15 AI and search bots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more), validates `llms.txt` / `llms-full.txt`, finds the XML sitemap, inspects homepage SEO and Schema.org JSON-LD, and returns a **0–100 GEO (Generative Engine Optimization) readiness score with fix codes** — for one domain or 1,000 in a single run.

- 💸 **$0.02 per audited domain = $2 per 100 domains.** Failed or partial audits are free.
- ⚡ ~5–10 seconds per domain, 5 domains in parallel by default. No login, no browser, no proxies.
- 🧾 Clean JSON / CSV / Excel output, ready for client reports, spreadsheets, dashboards or AI agents.

### What does the AI Crawler & llms.txt Checker do?

For every domain you submit it fetches at most five small public URLs (`/robots.txt`, `/llms.txt`, `/llms-full.txt`, the sitemap and the homepage) and reports:

| Area | What you get |
|---|---|
| **AI crawler access (robots.txt)** | `allowed` / `blocked` at the site root for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, DuckAssistBot (list is configurable, up to 50 tokens), plus whether the bot is named by an explicit rule |
| **llms.txt checker** | present? starts with a `# Title`? has a `> summary` blockquote? number of `##` sections and Markdown links; is `llms-full.txt` present? |
| **XML sitemap** | found via `robots.txt` `Sitemap:` line or `/sitemap.xml` (gzip supported), type (`sitemapindex` / `urlset`), entry count |
| **Homepage SEO for AI search** | title, meta description length, canonical, `<html lang>`, hreflang, Open Graph tags, Twitter card, **Schema.org JSON-LD types** + parse errors, H1 count, approx. word count, `noindex` (meta or `X-Robots-Tag`) |
| **GEO readiness score** | 0–100 score + machine-readable recommendations such as `ADD_LLMS_TXT` or `UNBLOCK_AI_SEARCH_BOTS:PerplexityBot` |

Works for websites in any language (output uses language-neutral codes).

#### Try one domain free first

Want to check a single site without an Apify account? Use the free [AI-search readiness checker on tesselyn.com](https://tesselyn.com/tools/ai-readiness-checker/?utm_source=apify\&utm_medium=readme\&utm_campaign=ai-readiness). It is a separate, simpler one-domain check with its own bot list and score, so its numbers can differ from this Actor's. Use this Actor for lists of domains, the full homepage SEO and Schema.org checks, scheduled monitoring and API/MCP use.

### Who is it for? Use cases

- **SEO & GEO agencies** — bulk-audit a client portfolio and turn the table into an AI-search readiness report.
- **In-house SEO / marketing teams** — check that a CDN, plugin or robots.txt change didn't accidentally block OAI-SearchBot or PerplexityBot.
- **Competitor research** — see which competitors block AI crawlers or already publish `llms.txt`.
- **Publishers & legal/compliance teams** — get an inventory of which AI training crawlers (GPTBot, CCBot, Google-Extended…) your sites opt out of.
- **Developers & AI agents** — feed structured site-readiness data into pipelines, dashboards or MCP-based agents.

### How to use it

1. Click **Try for free** / **Start**. The input is pre-filled with three example sites (apify.com, stripe.com, bbc.com), so the first run works instantly.
2. Replace them with your own domains or homepage URLs (bulk-paste or upload a list; `example.com` without `https://` works too).
3. Optionally edit the list of AI crawler user-agents, the timeout or concurrency.
4. Run it and open the **Overview**, **AI crawlers & llms.txt** or **Homepage SEO** tab of the results, or export to JSON, CSV, Excel or HTML.

#### Input example

```json
{
  "startUrls": [
    { "url": "https://apify.com" },
    { "url": "https://stripe.com" },
    { "url": "https://www.bbc.com" }
  ]
}
```

| Field | Default | Description |
|---|---|---|
| `startUrls` | – (required) | Domains or homepage URLs, max 1,000 per run. Duplicates are audited once. |
| `aiBots` | 15 tokens listed above | User-agent tokens to evaluate in robots.txt (max 50). |
| `maxSitemapUrlsToCount` | 5000 | Stop counting sitemap entries after this many. |
| `requestTimeoutSecs` | 20 | Per-request timeout. |
| `maxConcurrency` | 5 | Domains audited in parallel (max 20). |

### Output example

> ℹ️ Scores are automated indicators, not guarantees of visibility in any search or AI product.

Overview table from a real run (October 2026; results change as sites update their files):

| Domain | Score | Blocked AI bots (robots.txt) | llms.txt | Sitemap | JSON-LD types |
|---|---|---|---|---|---|
| apify.com | 100 | – | ✅ 9 sections, 73 links | ✅ index, 17 sitemaps | Organization, WebSite, SoftwareApplication… |
| stripe.com | 95 | – | ✅ 31 sections, 308 links | ✅ index, 9 sitemaps | Organization, WebSite, ContactPoint… |
| www.bbc.com | 65 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot + 4 more | ❌ | ✅ urlset | NewsMediaOrganization, WebPage |

One dataset item (shortened):

```json
{
  "input": "https://apify.com",
  "domain": "apify.com",
  "score": 100,
  "aiBotsBlocked": [],
  "robotsTxt": {
    "found": true,
    "sitemaps": ["https://apify.com/sitemap.xml"],
    "bots": { "GPTBot": "allowed", "ClaudeBot": "allowed", "PerplexityBot": "allowed", "Google-Extended": "allowed" }
  },
  "llmsTxt": { "found": true, "hasH1Title": true, "hasSummaryBlockquote": true, "sectionCount": 9, "linkCount": 73, "llmsFullTxtFound": false },
  "sitemap": { "found": true, "url": "https://apify.com/sitemap.xml", "type": "sitemapindex", "entries": 17 },
  "httpStatus": 200,
  "https": true,
  "homepage": {
    "title": "Apify: Marketplace of ready-to-run tools for AI",
    "metaDescriptionLength": 143,
    "canonical": "https://apify.com",
    "lang": "en",
    "openGraph": ["og:description", "og:image", "og:title", "og:type", "og:url"],
    "jsonLdTypes": ["Organization", "SoftwareApplication", "WebSite"],
    "h1Count": 1,
    "noindex": false
  },
  "recommendations": []
}
```

Robots.txt values look like `allowed`, `blocked`, `blocked (explicit rule)` (the bot is named in its own `User-agent` group) or `allowed (no robots.txt)`.

### How the GEO readiness score works

> ℹ️ The score is an automated indicator based on the checks below, not a guarantee of visibility, ranking or citation in any search engine or AI product.

The score is a transparent checklist (max 100):

| Check | Points |
|---|---|
| Served over HTTPS | 10 |
| robots.txt present | 10 |
| AI **search** crawlers not blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) | 15 |
| llms.txt present (+5 if it starts with an `# H1` title) | 10 + 5 |
| XML sitemap found | 10 |
| Schema.org JSON-LD on homepage | 10 |
| Title, meta description (≥ 50 chars), canonical, Open Graph, `html lang`, exactly one H1 | 5 each |
| Homepage `noindex` | −30 |

Blocking **training** crawlers such as GPTBot, CCBot or Google-Extended does **not** lower the score — that is a content-licensing decision, and the per-bot table still shows it.

Recommendation codes: `SERVE_OVER_HTTPS`, `ADD_ROBOTS_TXT`, `UNBLOCK_AI_SEARCH_BOTS:<bots>`, `ADD_LLMS_TXT`, `LLMS_TXT_SHOULD_START_WITH_H1`, `ADD_XML_SITEMAP_AND_REFERENCE_IN_ROBOTS`, `ADD_TITLE`, `ADD_META_DESCRIPTION`, `ADD_CANONICAL`, `ADD_SCHEMA_ORG_JSON_LD`, `FIX_INVALID_JSON_LD`, `ADD_OPEN_GRAPH_TAGS`, `ADD_HTML_LANG`, `HOMEPAGE_IS_NOINDEX`, `USE_EXACTLY_ONE_H1`.

### How much does it cost? Pricing

Pay-per-event pricing — you pay only for results:

| Event | Price |
|---|---|
| Domain audited successfully (`domain-audited`) | $0.02 per domain (= $2 per 100 domains, $20 per 1,000) |
| Actor start (`apify-actor-start`, per run, per GB of memory) | $0.00005 |

Platform compute is included in these prices. Rows with an `error` field are **not charged**: invalid or non-public inputs, unreachable sites, robots.txt server errors/timeouts, homepages that return an HTTP error (e.g. 403 bot protection) or that disallow this auditor in robots.txt. Such rows still contain whatever site-level data could be collected (e.g. the AI-crawler table), with `score: null`. Apify's Free plan includes monthly platform credit you can use to try it. You can cap spending per run with the *Maximum cost per run* option.

### Integrations and API

- **API:** run it from any language via the [Apify API](https://docs.apify.com/api/v2) or the Python / JavaScript clients — see the *API* tab for ready-made code.
- **No-code:** connect it to **Make, Zapier, n8n**, Google Sheets, Slack or webhooks via Apify integrations.
- **Monitoring:** use Apify **Schedules** to re-audit your sites weekly and catch newly blocked AI crawlers.
- **AI agents:** available to MCP clients (Claude, Cursor, etc.) through the Apify MCP server.

### Responsible use

- Fetches at most ~5 small public URLs per domain (robots.txt, llms.txt, llms-full.txt, sitemap, homepage), **one at a time with at least 1 s between requests** (longer if the site sets a `Crawl-delay`, capped at 10 s).
- Identifies itself honestly with the User-Agent `TesselynAuditBot/1.0 (+https://apify.com/tesselyn)`. It never impersonates GPTBot or other crawlers.
- **Obeys robots.txt (RFC 9309)** for its own token `TesselynAuditBot`: disallowed paths are skipped; if the homepage is disallowed the domain is reported and not charged; if robots.txt is unreachable (server error/timeout) the site is not crawled at all.
- Only public websites: private, loopback, link-local, cloud-metadata and other non-public addresses are refused, including after DNS resolution and on every redirect. Only ports 80/443, max 5 redirects, per-request timeouts and response-size caps (robots.txt 512 KB, llms.txt 1 MB, sitemap 10 MB, homepage 3 MB).
- Max 1,000 domains per run. No login, no CAPTCHA bypass, no proxies, no JavaScript rendering.
- Collects **no contact data** (no emails, phones or addresses are extracted; JSON-LD is reduced to `@type` names only).

### Terms of use

These usage terms supplement, and do not replace, the [Apify Standard Actor Contract](https://docs.apify.com/legal/standard-actor-contract), which governs your use of this Actor.

- Only submit domains you own, manage, or otherwise have a legitimate interest and right to audit. You are responsible for your use of the Actor and its output complying with applicable laws and the target websites' terms.
- Do not use the Actor to probe internal networks, to load-test, or to circumvent access controls. Abusive use may be blocked.
- **Sanctions and restricted countries.** You may not use this Actor if you are located in, ordinarily resident in, or acting on behalf of a person or organisation in a restricted country or region. Currently these are: Afghanistan, Belarus, Central African Republic, Cuba, Democratic Republic of the Congo, Haiti, Iran, Iraq, Lebanon, Libya, Mali, Myanmar, Nicaragua, North Korea, Russia, Somalia, South Sudan, Sudan, Syria, Venezuela, Yemen, Zimbabwe, Antarctica, and the Crimea, Sevastopol, Donetsk, Luhansk, Kherson and Zaporizhzhia regions of Ukraine. You also may not use it if you are on, or owned or controlled by someone on, any applicable sanctions list (including those of the United Nations, the United States (OFAC SDN), the European Union, the United Kingdom or Israel). We may block such use, and we may update this list when the law or our policy changes.
- Output is an automated, best-effort technical assessment provided "as is"; verify it before relying on it for consequential decisions. The score is not an endorsement or a guarantee of visibility in any search or AI product. Names of third-party crawlers/products are used only to describe compatibility; no affiliation is implied.
- **No warranty; limitation of liability.** To the maximum extent permitted by applicable law, the Actor and its output are provided "as is" and "as available", without warranties of any kind, express or implied (including accuracy, completeness, merchantability, fitness for a particular purpose and non-infringement), as set out in section 7 of the Apify Standard Actor Contract. The Actor creator (Tesselyn) is not liable for any indirect, incidental, special, consequential or punitive loss or damage, including lost traffic, rankings, revenue, profits or data, arising from use of or reliance on the Actor or its output; any other liability is limited as set out in section 8 of that Contract, and nothing here limits liability that cannot be limited under applicable law. Output is informational only and is not SEO, legal or other professional advice. Results may change as websites, crawlers and standards change.

### Privacy notice

- **What is processed:** the domains/URLs you submit and the public technical data listed above (robots.txt rules, llms.txt structure, sitemap counts, homepage metadata such as title, language, canonical URL, Open Graph tag names and Schema.org types). A homepage title can occasionally contain a person's or business name; nothing else about individuals is collected.
- **Where it is stored:** only in your own Apify run storage (default dataset, run log and input), under your Apify account and Apify's retention settings. The Actor sends nothing to third parties other than the HTTP requests to the sites you submit, and stores nothing elsewhere. It runs with Apify *limited permissions* (it can access only its own run storage).
- **Roles:** for any personal data in your input or output you are the controller and the Actor creator acts as processor, as set out in section 5 of the Apify Standard Actor Contract. Run logs contain only domain names, scores and error codes.
- **Deletion:** delete the run or dataset in Apify Console at any time. Questions or privacy requests: open an issue on the Actor's Issues tab.

### Related Tesselyn tools

- [Broken Link & Redirect Checker](https://apify.com/tesselyn/broken-link-checker): crawl a site for broken links, redirect chains and missing #anchors.
- Domain Health Checker: domain expiry, DNS, SPF/DMARC/DKIM, SSL and HSTS in bulk. Coming to the Apify Store soon. Until then, try the free [domain and email health checker on tesselyn.com](https://tesselyn.com/tools/domain-health-checker/).
- Bulk PageSpeed & Core Web Vitals Checker: Google PageSpeed Insights scores for many URLs. Coming to the Apify Store soon.
- See [all Tesselyn Actors on the Apify Store](https://apify.com/tesselyn) and the [free single-site tools on tesselyn.com](https://tesselyn.com/tools/).

### FAQ

#### What is llms.txt and does it matter?

`llms.txt` is a community proposal (2024) for a Markdown file at the site root that gives language models a curated map of a site's key content. It is not an official standard and AI search engines have not confirmed using it for ranking; it is cheap to add and used by some AI tools and agents. This Actor checks presence and basic structure (H1 title, summary, sections, links).

#### Which AI crawlers should I allow?

Search/answer crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) affect whether you can be found and cited in AI answers; training crawlers (GPTBot, CCBot, Google-Extended…) are an opt-out choice. The report shows each separately so you can decide.

#### Which bots matter for ChatGPT, Claude and Perplexity answers?

The search and answer bots. By default this Actor checks OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot. You can add other tokens, such as Claude-User or Perplexity-User, in the `aiBots` input. OpenAI and Anthropic describe GPTBot and ClaudeBot as their model-training crawlers, so blocking those is a training opt-out and is reported separately. Vendors change their crawlers, so check their current documentation before you edit robots.txt.

#### Is there a free version?

For a single domain, yes: the free [AI-search readiness checker on tesselyn.com](https://tesselyn.com/tools/ai-readiness-checker/) runs a simpler check in your browser, with no account needed. Apify's Free plan also includes monthly platform credit you can use to try this Actor.

#### Does "allowed" mean the bot can really reach my site?

It means your `robots.txt` allows that user-agent at the site root. robots.txt is advisory; firewalls/CDNs may still block or allow bots regardless, and path-specific rules below the root are not evaluated. This Actor does not impersonate crawlers, so it cannot test CDN-level blocking.

#### Does it render JavaScript or crawl the whole site?

No. It analyzes the raw homepage HTML and the root-level files only. That keeps it fast, cheap and polite.

#### Why did a domain return an error?

See the `error` field: e.g. `HOMEPAGE_HTTP_403` (bot protection), `ROBOTS_TXT_UNREACHABLE:TIMEOUT`, `HOMEPAGE_DISALLOWED_BY_ROBOTS_FOR_THIS_AUDITOR`, `INVALID_OR_NON_PUBLIC_DOMAIN:…`. Error rows are not charged.

#### Is it legal?

It only reads small public files that websites publish for crawlers (robots.txt, llms.txt, sitemap) and the public homepage, identifies itself, obeys robots.txt and extracts no contact data. You remain responsible for auditing only sites you have a right to audit (see Terms of use). This is not legal advice.

#### How fast / how polite is it?

Each domain gets ~5 sequential requests spaced ≥1 s apart (about 5–10 s per domain); different domains are processed in parallel with limited concurrency (default 5).

### Support

Built and maintained by **Tesselyn**. Found a bug, need another check or a different crawler list? Open an issue on the Actor's **Issues** tab — feedback is welcome.

Website: [Tesselyn website (tesselyn.com)](https://tesselyn.com)

# Actor input Schema

## `startUrls` (type: `array`):

Domains or homepage URLs to audit (one audit per domain, duplicates removed, max 1,000 per run). 'example.com' without https:// works too. Only public websites you are entitled to audit; private/internal addresses are rejected. Prefilled with three example sites so you can click Start right away.

## `aiBots` (type: `array`):

User-agent tokens checked against the site's robots.txt rules for the site root (/). Default: 15 AI search, AI training and classic search crawlers. Max 50.

## `maxSitemapUrlsToCount` (type: `integer`):

Stops counting sitemap entries after this many (keeps runs cheap).

## `requestTimeoutSecs` (type: `integer`):

Per-request HTTP timeout. Lower = faster runs on slow sites.

## `maxConcurrency` (type: `integer`):

How many domains are audited in parallel (each domain gets ~5 sequential requests).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://stripe.com"
    },
    {
      "url": "https://www.bbc.com"
    }
  ],
  "aiBots": [
    "GPTBot",
    "OAI-SearchBot",
    "ChatGPT-User",
    "ClaudeBot",
    "Claude-SearchBot",
    "PerplexityBot",
    "Google-Extended",
    "Googlebot",
    "Bingbot",
    "Applebot-Extended",
    "CCBot",
    "Bytespider",
    "Meta-ExternalAgent",
    "Amazonbot",
    "DuckAssistBot"
  ],
  "maxSitemapUrlsToCount": 5000,
  "requestTimeoutSecs": 20,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

All per-domain audit results stored in the default dataset (one item per unique input domain).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        },
        {
            "url": "https://stripe.com"
        },
        {
            "url": "https://www.bbc.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tesselyn/ai-search-readiness-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        { "url": "https://apify.com" },
        { "url": "https://stripe.com" },
        { "url": "https://www.bbc.com" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("tesselyn/ai-search-readiness-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://stripe.com"
    },
    {
      "url": "https://www.bbc.com"
    }
  ]
}' |
apify call tesselyn/ai-search-readiness-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tesselyn/ai-search-readiness-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kPZZaexy5Gr2f5g2D/builds/PUMZcIzjqW5tVnzeu/openapi.json