# AI Crawler & GEO Readiness Audit (`ottomath/ai-crawler-geo-audit`) Actor

Check which AI crawlers your robots.txt allows, plus llms.txt, sitemaps and homepage readiness. Get a 0-100 score and fixes. $15 per 1,000 sites.

- **URL**: https://apify.com/ottomath/ai-crawler-geo-audit.md
- **Developed by:** [Ottomath](https://apify.com/ottomath) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $15.00 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler & GEO Readiness Audit

Find out how ready a website is for AI search and AI assistants. For every site in your list the Actor reads robots.txt, reports which AI crawlers are allowed or blocked, checks llms.txt, counts sitemap URLs, inspects the homepage and returns a 0-100 score with plain-English fixes. It can also draft an llms.txt file. Plain HTTP requests only: no browser, no proxies, no personal data.

### What it does

For each site the Actor makes a few polite requests and returns one dataset item.

1. **robots.txt and AI crawlers.** The file is parsed as RFC 9309 describes: user-agent groups, the longest matching rule wins, `Allow` beats `Disallow` on a tie, `*` and `$` wildcards work and user-agent names are case-insensitive. A missing file (4xx) means everything is allowed; an unreachable one (5xx) means crawlers should stay away. Each AI crawler gets `allowed`, `partial` (typical paths such as `/blog/`, `/docs/` or `/pricing` are blocked) or `blocked`. Declared `Sitemap:` lines are listed.
2. **llms.txt and llms-full.txt.** Is the file served, with which HTTP status and content type, and is it real text rather than an HTML error page? The structure is checked against the llms.txt proposal (H1 title, blockquote summary, sections with link lists) and up to 20 internal links are tested for broken URLs.
3. **Sitemaps.** Sitemaps declared in robots.txt, or `/sitemap.xml` and `/sitemap_index.xml` when none is declared. Indexes and gzip files are followed. You get the URL count (capped by `maxSitemapUrls`), the newest `lastmod` date and any problems.
4. **Homepage.** Status, final URL, response time, HTML size, title, meta description, canonical, `robots` meta tag and `X-Robots-Tag` header (including `noai` and `noimageai`), hreflang count, Open Graph tags, JSON-LD types, and a heuristic telling whether the main text is in the HTML without running JavaScript.
5. **Score and recommendations.** A 0-100 score with the weights below and a prioritised list of fixes (high, medium, low).
6. **Optional llms.txt draft.** With `generateLlmsTxt` on, the Actor writes a draft from the sitemap URLs and the titles and descriptions of up to `maxPagesForLlmsTxt` pages, fetching only pages that robots.txt allows. The draft is extractive: no AI model writes anything, so review it before publishing.

The built-in list has 21 user-agent tokens taken from vendor documentation: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google-Extended, Applebot-Extended, CCBot, Amazon (Amazonbot, Amzn-SearchBot, Amzn-User), Meta (meta-externalagent, meta-webindexer, meta-externalfetcher), DuckAssistBot and MistralAI-User. `Bytespider` and `cohere-ai` are common in robots.txt files but not documented by their vendors, so treat them as unofficial. Use `aiBots` to check your own tokens instead.

| Score part | Points | What earns the points |
| --- | --- | --- |
| AI crawler access | 30 | Search and assistant crawlers allowed (partial counts half). Training-only crawlers are reported but not scored, because blocking them is a legitimate choice. |
| Readable without JavaScript | 20 | Homepage text is in the HTML: yes 20, partial 10, no 0. |
| Sitemap | 15 | Found with URLs 8, declared in robots.txt 3, `lastmod` on half the URLs 2, newest `lastmod` under 12 months old 2. |
| Metadata and structured data | 20 | Title 4, meta description 4, canonical 3, JSON-LD 5, complete Open Graph 2, an H1 2. |
| llms.txt | 10 | Served as a file 6, H1 plus a link 2, links checked and none broken 2. |
| Page directives | 5 | No `noindex` 3, no `nosnippet`, `noai` or `noimageai` 2. |

llms.txt is an emerging convention and AI systems differ in whether they read it, which is why it is worth 10 points only.

### Use cases

- **GEO and SEO agencies:** audit a client or prospect list in one run and sort by score.
- **Publishers:** confirm that you block the training crawlers you want to block while letting AI search crawlers in, or the reverse.
- **In-house teams:** schedule weekly runs and watch for robots.txt, sitemap or llms.txt changes after a release.
- **Developers:** check a site before launch for leftover `noindex`, empty server output and a missing sitemap.
- **Documentation teams:** generate a first llms.txt draft from the sitemap, then edit it by hand.

### Input

| Field | Default | Description |
| --- | --- | --- |
| `urls` | required | Sites to audit, up to 1,000, as full URLs. The site root of each address is audited. Duplicates run once. |
| `aiBots` | built-in 21 | Your own user-agent tokens instead of the built-in list. The score then averages over your tokens. |
| `maxSitemapUrls` | 5000 | Stop counting after this many sitemap URLs per site (1 to 50,000). The result says when the count is a lower bound. |
| `generateLlmsTxt` | false | Draft an llms.txt per site. |
| `maxPagesForLlmsTxt` | 30 | Pages fetched for each draft (1 to 200). Each fetched page is a billed event. |
| `respectRobotsTxt` | true | Apply robots.txt to this Actor's own requests. Turn off only for sites you own or may audit. |
| `maxConcurrency` | 10 | Sites in parallel (1 to 30), at most 2 requests at a time per domain. |
| `requestTimeoutSecs` | 20 | Timeout per request (1 to 120). |

```json
{
  "urls": ["https://example.com", "https://wordpress.org"],
  "generateLlmsTxt": false,
  "maxSitemapUrls": 5000
}
```

### Output example

One dataset item per site. A site that fails gets an item with `error` set, no score and no charge. This is a shortened real result for example.com (long sections and lists trimmed):

```json
{
  "url": "https://example.com",
  "domain": "example.com",
  "checkedAt": "2026-10-08T10:28:20.898Z",
  "score": 59,
  "scoreBreakdown": { "aiCrawlerAccess": { "points": 30, "max": 30 }, "jsReadability": { "points": 20, "max": 20 }, "sitemap": { "points": 0, "max": 15 }, "metadata": { "points": 4, "max": 20 }, "llmsTxt": { "points": 0, "max": 10 }, "directives": { "points": 5, "max": 5 } },
  "robots": { "url": "https://example.com/robots.txt", "found": false, "fetchStatus": "not-found", "httpStatus": 404, "groupCount": 0, "sitemapsDeclared": [], "counts": { "allowed": 21, "partial": 0, "blocked": 0, "unknown": 0 } },
  "aiBots": { "GPTBot": "allowed", "OAI-SearchBot": "allowed", "ClaudeBot": "allowed", "Claude-SearchBot": "allowed", "PerplexityBot": "allowed", "Google-Extended": "allowed" },
  "llmsTxt": { "url": "https://example.com/llms.txt", "found": false, "httpStatus": 404, "valid": false, "wellStructured": false, "brokenLinks": [] },
  "llmsFullTxt": { "url": "https://example.com/llms-full.txt", "found": false, "httpStatus": 404 },
  "sitemaps": { "found": false, "declaredInRobotsTxt": false, "urlCount": 0, "newestLastmod": null, "issues": [] },
  "homepage": { "status": 200, "finalUrl": "https://example.com/", "responseTimeMs": 48, "htmlBytes": 577, "title": "Example Domain", "metaDescription": null, "canonical": null, "hreflangCount": 0, "jsonLdTypes": [], "noindex": false, "noai": false, "readableWithoutJs": "yes", "visibleTextChars": 156 },
  "recommendations": [
    { "priority": "high", "area": "sitemap", "message": "No XML sitemap was found. Publish /sitemap.xml and add a \"Sitemap:\" line to robots.txt so crawlers can discover all your pages." }
  ],
  "generatedLlmsTxt": null,
  "error": null
}
```

Bot statuses are `allowed`, `partial`, `blocked` or `unknown` (robots.txt unreachable). The Console **AI readiness** table shows the domain, score, the status of six key bots, llms.txt found, sitemap URL count and JavaScript readability; two more views list recommendations and generated drafts.

### Pricing

Pay per event. You pay for sites that were actually audited, never for failures.

| Event | Price | When it is charged |
| --- | --- | --- |
| Actor start | $0.005 | Once per run. |
| Site audited | $0.015 | Once per site whose homepage answered with an HTTP status below 400 and was analysed. |
| Page fetched for llms.txt draft | $0.001 | Per extra page fetched, only when `generateLlmsTxt` is on. |

Invalid URLs, robots.txt blocks, timeouts, DNS or TLS errors and 4xx or 5xx homepages are free. Examples: 100 sites cost about $1.51, 1,000 sites about $15.01, and one site with a 30-page draft $0.05. The Actor never charges beyond the maximum total charge you set; it stops early and says so in the run status.

### FAQ

**Does it respect robots.txt?** Yes. With `respectRobotsTxt` on, the Actor reads each site's robots.txt first and fetches the homepage, llms files, sitemaps, link checks and draft pages only where that file allows its own user agent. If robots.txt cannot be read because of a server error, the site is skipped and not charged. It identifies itself as `OttomathGeoAudit/1.0 (+https://apify.com)`, uses no proxies and no fingerprint tricks.

**Does it run a browser or JavaScript?** No. "Readable without JavaScript" is a heuristic built from visible text, empty app containers such as `#root` or `#__next`, `noscript` warnings and the share of script in the page. It shows what a crawler that does not run JavaScript is likely to see, and it can be wrong on unusual pages.

**What does "partial" mean?** The crawler may fetch the site root but is blocked from some typical paths. The blocked paths are listed in `robots.aiBotDetails`.

**Does it collect personal data?** No. It does not extract e-mail addresses, phone numbers or names, and text taken from pages is scrubbed of anything that looks like an e-mail address or phone number.

**How is the draft written, and what is billed?** From what the pages already say: the homepage title and description, then page titles and meta descriptions grouped by the first part of the URL path. Pages that are not HTML, missing or `noindex` are left out, but every page that is actually fetched counts as one event. Pages disallowed by robots.txt are not fetched and not billed.

### Limitations

- robots.txt shows a site's stated policy, not what its servers enforce. A firewall can block a crawler that robots.txt allows, and some assistants fetch pages for a user in ways robots.txt may not govern.
- Only robots.txt, llms files, sitemaps and the homepage are analysed, plus the pages used for the optional draft. This is not a full-site crawl.
- Sites that block data-centre traffic may answer 403; they are reported with an error and are free.
- Size limits: homepage HTML up to 2 MB, llms.txt up to 1 MB, each sitemap file up to 20 MB (50 MB decompressed), at most 30 sitemap files per site. Counts above `maxSitemapUrls` are flagged as a lower bound.
- Each site has an overall time limit of at least 90 seconds, and HTML analysis stops after 20 seconds.
- The bot list reflects vendor documentation read in October 2026. Vendors add and rename crawlers, so use `aiBots` for new tokens.
- The score is a transparent checklist, not a ranking factor or a promise of visibility in any AI product. The Actor is not affiliated with any AI company or search engine.

### Changelog

- **1.0 (October 2026):** first release.

# Actor input Schema

## `urls` (type: `array`):

One site per line, for example `https://example.com` or `example.com` (https is assumed). The audit covers the whole site (its homepage, robots.txt, llms.txt and sitemaps), so a path is ignored. Up to 1,000 sites per run. Duplicates are audited once.

## `aiBots` (type: `array`):

Replace the built-in list of AI crawler user-agent tokens, for example `GPTBot` or `ClaudeBot`. Leave empty to check the built-in list of 21 tokens. When you set your own list, the AI-access part of the score is the plain average over your tokens.

## `maxSitemapUrls` (type: `integer`):

Stop counting sitemap URLs after this many per site. The result then says the count is a lower bound.

## `generateLlmsTxt` (type: `boolean`):

Also write a draft llms.txt for each audited site from its sitemap URLs and the titles and descriptions of up to the number of pages set below. Pages are fetched politely and only where robots.txt allows this Actor. Each extra page fetched is charged.

## `maxPagesForLlmsTxt` (type: `integer`):

Upper limit of extra pages fetched per site when drafting an llms.txt. Only used when the option above is on.

## `respectRobotsTxt` (type: `boolean`):

Apply each site's robots.txt rules for this Actor's own User-Agent (or `*`) before fetching the homepage, llms.txt, sitemaps and pages. Sites that disallow this Actor are reported with an error and are free. The robots.txt analysis itself always works.

## `maxConcurrency` (type: `integer`):

How many sites are audited in parallel. At most 2 requests per domain run at the same time.

## `requestTimeoutSecs` (type: `integer`):

Give up on a single HTTP request after this many seconds.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://wordpress.org"
  ],
  "maxSitemapUrls": 5000,
  "generateLlmsTxt": false,
  "maxPagesForLlmsTxt": 30,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

One item per site with AI crawler access per bot, llms.txt, sitemaps, homepage checks, score and recommendations (and free error rows), as JSON. Use the dataset tabs in the Console for tables, CSV and Excel exports.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com",
        "https://wordpress.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ottomath/ai-crawler-geo-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://example.com",
        "https://wordpress.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("ottomath/ai-crawler-geo-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com",
    "https://wordpress.org"
  ]
}' |
apify call ottomath/ai-crawler-geo-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ottomath/ai-crawler-geo-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/z6iPlSCtXoEkIS3mD/builds/lDPP8Mt9CT7ndeCcC/openapi.json