# GEO & llms.txt Readiness Auditor (`gp005/geo-llms-auditor`) Actor
Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.
- **URL**: https://apify.com/gp005/geo-llms-auditor.md
- **Developed by:** [Geo](https://apify.com/gp005) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $40.00 / 1,000 domain auditeds
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## GEO & llms.txt Readiness Auditor — which AI crawlers your site allows or blocks
Find out — per domain — whether **GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot** and 18+ other AI crawlers can reach and cite your content, whether your **llms.txt** is valid, and whether your pages carry the on-page signals AI answer engines need. You get a **GEO-readiness verdict, a score, and an exact prioritized fix list** — per domain. Non-scraper: it reads only public well-known files (`robots.txt`, `llms.txt`, `sitemap.xml`) and one page's ``, one polite request each, honoring robots.txt.
### What makes it different
Most checkers just say a token is "present/absent." This one **tiers crawler compliance**, so you learn the truth:
- A `Disallow: PerplexityBot` or `Bytespider` in robots.txt **won't actually stop them** — those operators are documented ignoring robots.txt (only a WAF will) — so we flag "ineffective block."
- It separates **AI-search/citation** bots (blocking them = you vanish from ChatGPT/Claude/Perplexity/Google AI answers — a P0 error) from **training** bots (blocking them is a legitimate choice) from **opt-out tokens** (`Google-Extended`, `Applebot-Extended` — directives, not crawlers).
### Use cases
- **Audit your own sites/subdomains** before/after a robots.txt or CDN change.
- **Agencies:** batch-audit a client portfolio and hand over a fix list.
- **Catch the silent killer:** a `noindex`, a JS-only page, or an accidental AI-search block that's costing you citations.
### Input
`domains` (required, your sites) · `auditPath` (page to check, default `/`) · `verifyToken` (optional — put `` on the page to mark `ownership_verified`) · `timeoutSecs`.
```json
{ "domains": ["yourbrand.com", "docs.yourbrand.com"], "auditPath": "/" }
````
### The 16 checks (per domain)
HTTPS/200 + redirect chain · AI-search crawlers not blocked (**P0**) · ineffective-block flag (gray-zone/non-compliant crawlers) · training opt-out tokens · sitemap present/valid/referenced · **content in raw HTML** (no-JS, **P0**) · `noindex`/`nosnippet` (**P0**) · AI opt-out meta (`noai`) · schema.org JSON-LD · canonical · title + meta description · Open Graph · semantic structure (single H1, ``/``) · freshness dates · **llms.txt** presence + validity · llms-full.txt. → weighted **score** + **READY / NEEDS\_WORK / NOT\_READY** verdict + `top_fixes`.
### Pricing (pay-per-event) & cost example
| Event | Price | When |
|---|--:|---|
| `domain-audited` (primary) | $0.04 | per domain fully audited |
| `issue-found` | $0.01 | per fix-worthy issue flagged (**capped at 8/domain**) |
A clean domain costs **$0.04**; a domain with 4 issues, **$0.08**; a badly-misconfigured one, at most **$0.12**. Measured platform compute is **<$0.001/domain** (<3% of revenue). Compare: brand-tracker SaaS runs **$25–99/mo per domain**; this is per-audit and pennies.
### Integrate
Schedule it (weekly), then use Apify **webhooks/API/MCP/Make/n8n** to alert on any domain whose `geo_ready_verdict` drops or whose `search_bots_blocked > 0`.
### Limitations
- Audits **one page per domain** (the root or your `auditPath`) — not a full crawl.
- robots.txt matching uses the standard longest-prefix rule; unusual wildcard patterns may under-report.
- Intended for domains you **own or are authorized to audit**; it reads public files with a single polite request and honors robots.txt. Compliance tiering reflects publicly documented crawler behavior and can change.
# Actor input Schema
## `domains` (type: `array`):
Domains (or full URLs) to audit — ideally your OWN sites. The auditor reads only public well-known files (robots.txt, llms.txt, sitemap.xml) and one HTML page per domain, with a single polite request each, and honors robots.txt.
## `auditPath` (type: `string`):
Which page to fetch for on-page signals (title, meta, canonical, JSON-LD, headings). Default is the site root.
## `verifyToken` (type: `string`):
If set, the auditor looks for on the page and marks ownership\_verified=true when it matches. Recommended when auditing domains to prove you own them.
## `timeoutSecs` (type: `integer`):
Timeout for each well-known-file / page fetch.
## Actor input object example
```json
{
"domains": [
"yourbrand.com",
"https://docs.yourbrand.com"
],
"auditPath": "/",
"timeoutSecs": 15
}
```
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"domains": [
"example.com"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("gp005/geo-llms-auditor").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = { "domains": ["example.com"] }
# Run the Actor and wait for it to finish
run = client.actor("gp005/geo-llms-auditor").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"domains": [
"example.com"
]
}' |
apify call gp005/geo-llms-auditor --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=gp005/geo-llms-auditor",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "GEO & llms.txt Readiness Auditor",
"description": "Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.",
"version": "0.1",
"x-build-id": "wO1bKd9ASxBvmFxrU"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/gp005~geo-llms-auditor/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-gp005-geo-llms-auditor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/gp005~geo-llms-auditor/runs": {
"post": {
"operationId": "runs-sync-gp005-geo-llms-auditor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/gp005~geo-llms-auditor/run-sync": {
"post": {
"operationId": "run-sync-gp005-geo-llms-auditor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"domains"
],
"properties": {
"domains": {
"title": "Domains to audit",
"type": "array",
"description": "Domains (or full URLs) to audit — ideally your OWN sites. The auditor reads only public well-known files (robots.txt, llms.txt, sitemap.xml) and one HTML page per domain, with a single polite request each, and honors robots.txt.",
"items": {
"type": "string"
}
},
"auditPath": {
"title": "Page path to audit",
"type": "string",
"description": "Which page to fetch for on-page signals (title, meta, canonical, JSON-LD, headings). Default is the site root.",
"default": "/"
},
"verifyToken": {
"title": "Ownership verification token (optional)",
"type": "string",
"description": "If set, the auditor looks for on the page and marks ownership_verified=true when it matches. Recommended when auditing domains to prove you own them."
},
"timeoutSecs": {
"title": "Per-request timeout (seconds)",
"minimum": 3,
"maximum": 60,
"type": "integer",
"description": "Timeout for each well-known-file / page fetch.",
"default": 15
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```