# AI Crawler Checker: robots.txt & llms.txt Audit (`aiqlabs/seo-ai-visibility-auditor`) Actor

Audits any URL for classic SEO health and for AI/LLM discoverability - AI crawler access in robots.txt, llms.txt, structured data, metadata - in a single pass.

- **URL**: https://apify.com/aiqlabs/seo-ai-visibility-auditor.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** AI, SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

**Find out whether AI answer engines can actually see your site — and fix what is blocking them.** This Actor audits any URL for classic SEO health *and* for AI/LLM discoverability in a single pass: which AI crawlers your `robots.txt` allows, whether you publish an `llms.txt`, whether your content survives without JavaScript, and whether your structured data gives models something to cite. You get one scored row per page, with a concrete fix attached to every issue.

Most SEO auditors were designed for the ten-blue-links era. They will happily give you a green score while `GPTBot`, `ClaudeBot` and `PerplexityBot` are all locked out of your site.

### Why use SEO & AI Visibility Auditor?

Search traffic increasingly arrives through answers, not links. Whether you appear in those answers depends on things a traditional audit never looks at:

- **AI crawler access.** Blocking `Google-Extended` removes you from Gemini's grounding. Blocking `OAI-SearchBot` removes you from ChatGPT Search. These are separate agents from `Googlebot`, and blocking them is usually an accident — a copied `robots.txt` snippet nobody revisited.
- **Server-rendered content.** Most AI crawlers do not execute JavaScript. A page that renders beautifully in your browser can be an empty shell to them. This Actor reports exactly how much text exists without JS.
- **Structured data.** Schema.org markup is how a model knows *what* your page is. Missing or malformed JSON-LD is silently ignored.
- **The classic basics still matter** — titles, descriptions, canonicals, headings, alt text, Open Graph — so they are all in the same report.

Try it on a publisher and the difference is immediate. `nytimes.com` blocks 21 of the 26 AI crawlers this Actor checks. `wikipedia.org` blocks none. `theatlantic.com` allows `GPTBot` while blocking `ClaudeBot` — a licensing decision, visible in one line of output.

### How to use SEO & AI Visibility Auditor

1. Paste the URLs you care about into **URLs to audit** — your home page, top landing pages, best blog posts.
2. Leave everything else at its defaults for a first run.
3. Click **Start**. A handful of pages takes seconds.
4. Open the **Scores** tab for the summary, then **AI visibility** and **Issues** for the detail.
5. Work down the issue list. Every issue carries a `fix` field telling you what to change.

To sweep a whole site instead of specific pages, switch on **Also follow internal links** and raise **Max pages**.

### Input

| Field | Type | Default | What it does |
| --- | --- | --- | --- |
| `startUrls` | array | — | Pages to audit. One report row per URL. **Required.** |
| `crawlSite` | boolean | `false` | Follow internal links and audit those too, same domain only. |
| `maxPages` | integer | `10` | Hard cap on pages audited. Stops an accidental full-site crawl. |
| `checkBrokenLinks` | boolean | `false` | HEAD-request every link on the page and report failures. |
| `maxLinksToCheck` | integer | `50` | Per-page cap when link checking is on. |
| `proxyConfiguration` | object | off | Use a proxy if the target blocks datacenter traffic. |

```json
{
    "startUrls": [{ "url": "https://example.com" }],
    "crawlSite": false,
    "maxPages": 10,
    "checkBrokenLinks": false
}
```

### Output

One dataset item per audited page. Download as JSON, CSV, Excel or HTML, or pull it from the API.

```json
{
    "url": "https://example.com/",
    "statusCode": 200,
    "scoreTotal": 48,
    "scoreSeo": 72,
    "scoreAiAccess": 50,
    "scoreStructuredData": 20,
    "scoreSocial": 10,
    "title": "Example Domain",
    "titleLength": 14,
    "metaDescription": "",
    "h1Count": 1,
    "jsonLdTypes": [],
    "robotsTxtFound": false,
    "llmsTxtFound": false,
    "sitemapFound": false,
    "aiCrawlersAllowedCount": 26,
    "aiCrawlersBlocked": [],
    "mainTextChars": 125,
    "textToHtmlRatio": 0.224,
    "issueCount": 15,
    "issues": [
        {
            "severity": "warning",
            "area": "seo",
            "message": "No meta description.",
            "fix": "Add a 120-160 character summary. Answer engines often quote it verbatim."
        }
    ]
}
```

### What it checks

| Area | Fields |
| --- | --- |
| **AI access** | Per-bot allow/deny for 26 AI crawlers, matched `robots.txt` rule, `llms.txt` presence and size, sitemap discovery, text available without JavaScript, text-to-HTML ratio |
| **SEO** | Title and length, meta description and length, canonical (and whether it is self-referential), meta robots, `X-Robots-Tag`, `lang`, charset, viewport, favicon |
| **Structure** | H1/H2 counts, heading level skips, image count, images missing `alt`, internal/external/nofollow link counts, optional broken-link report |
| **Structured data** | JSON-LD block count, every `@type` found (including inside `@graph`), JSON parse errors, microdata types, RDFa presence, and flags for Organization, Article, FAQPage, HowTo, Product and BreadcrumbList |
| **Social** | `og:title`, `og:description`, `og:image`, `og:type`, `twitter:card` |
| **Scoring** | Four 0-100 sub-scores plus a weighted total, and a severity-tagged issue list with a fix for each |

### AI crawlers covered

OpenAI (`GPTBot`, `OAI-SearchBot`, `ChatGPT-User`), Anthropic (`ClaudeBot`, `Claude-SearchBot`, `Claude-User`, `anthropic-ai`), Perplexity (`PerplexityBot`, `Perplexity-User`), Google (`Google-Extended`, `Googlebot`), Microsoft (`Bingbot`), Apple (`Applebot`, `Applebot-Extended`), Meta (`meta-externalagent`), Amazon (`Amazonbot`), ByteDance (`Bytespider`), Common Crawl (`CCBot`), plus Cohere, Mistral, You.com, Diffbot, Webz.io, ImageSift, AI2 and Timpi.

Each one is reported with its vendor, what blocking it costs you, and the exact `robots.txt` line that decided the verdict.

### Cost estimation

Auditing is cheap because it is plain HTTP — no browser. A single page is one request plus up to three small site-level files (`robots.txt`, `llms.txt`, `sitemap.xml`), and those are fetched **once per domain**, not once per page. Ten pages on one domain typically finish in seconds on the smallest memory setting.

Turning on **Check links for 404s** is the one thing that meaningfully increases run time, since it adds up to `maxLinksToCheck` extra requests per page. Leave it off for routine audits.

### Tips

- **Audit templates, not just pages.** One product page, one blog post, one category page tells you more than ten pages of the same type.
- **Check `scoreAiAccess` before anything else.** If AI crawlers are blocked, no amount of on-page work will get you cited.
- **Watch `mainTextChars`.** Under a few hundred characters on a content page means your text is being injected by JavaScript, and most AI crawlers will never see it.
- **Re-run after deploys** and schedule it. Crawler policy is the kind of thing that regresses silently when someone edits `robots.txt`.

### FAQ

**Does this modify my site?** No. Every request is a read.

**Does it respect robots.txt?** Yes. It would be strange for a tool that reports on crawler policy to ignore it. If your own `robots.txt` blocks a URL you asked to audit, that page is skipped — which is itself a useful finding.

**Can I audit a competitor?** You can audit any publicly reachable page, and the AI-access picture is often the interesting part. Only public data is read.

**Why does my JavaScript site score badly on AI access?** Because that is the honest answer for a non-JS crawler. Server-side rendering or prerendering fixes it.

**Is `llms.txt` actually used?** Adoption is early and no major model vendor has committed to it. It is scored as a small bonus, not a requirement — it costs nothing to publish and may pay off later.

### Support

Found a bug or want a check added? Open an issue on the **Issues** tab of this Actor. Requests for additional crawlers or extra checks are welcome.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to audit. Each URL produces one report row. Enter the exact pages you care about - the home page, key landing pages, top blog posts.

## `crawlSite` (type: `boolean`):

Follow links found on the pages above and audit those too, staying on the same domain. Leave off to audit only the URLs you entered.

## `maxPages` (type: `integer`):

Hard cap on how many pages get audited in total, protecting you from an accidental full-site crawl. It only limits pages discovered by following links - every URL you list above is always audited, even if there are more of them than this number.

## `checkBrokenLinks` (type: `boolean`):

Send a HEAD request to each link found on the page and report the ones that fail. Slower, but finds dead links.

## `maxLinksToCheck` (type: `integer`):

Only used when link checking is on.

## `proxyConfiguration` (type: `object`):

Optional. Use a proxy if the target site blocks datacenter traffic.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "crawlSite": false,
  "maxPages": 10,
  "checkBrokenLinks": false,
  "maxLinksToCheck": 50,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/seo-ai-visibility-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/seo-ai-visibility-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call aiqlabs/seo-ai-visibility-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=aiqlabs/seo-ai-visibility-auditor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/GuuMKUiWcaUUqGGhG/builds/4aXhN46eiNwrUc3fU/openapi.json