# GEO Website Auditor & AI Crawler Checker (`frere527/geo-ai-visibility-auditor`) Actor

Audit one page or discover up to 100 same-origin URLs from XML sitemaps. Check AI crawler access, robots.txt, llms.txt, JSON-LD, indexability, citations, and answer-ready content. Export scored GEO reports with transparent evidence and prioritized fixes for AI search visibility.

- **URL**: https://apify.com/frere527/geo-ai-visibility-auditor.md
- **Developed by:** [Frere](https://apify.com/frere527) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$25.00 / 1,000 successful audit reports

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## GEO Website Auditor & AI Crawler Checker

![GEO & AI Visibility Auditor logo](assets/geo-ai-auditor-logo.png)

[![Run on Apify](https://apify.com/ext/run-on-apify.png)](https://apify.com/frere527/geo-ai-visibility-auditor)
[![GitHub](https://img.shields.io/badge/source-GitHub-181717?logo=github)](https://github.com/Frere527/geo-ai-visibility-auditor)
[![MIT license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Turn any public web page into a scored, actionable Generative Engine Optimization (GEO) report. This Actor checks whether major AI crawlers can retrieve the site, whether it publishes useful AI discovery files, and whether each page is structured so answer engines can understand and cite it. It produces explainable JSON results for SEO and content teams—without browser automation, logins, or proxy configuration.

### Common use cases

- Check whether GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended are allowed by `robots.txt`
- Audit a landing page before an AI-search or GEO campaign
- Find missing `llms.txt`, JSON-LD, canonical, sitemap, and indexability signals
- Audit exact URLs or automatically discover up to 100 same-origin pages from XML sitemaps and `robots.txt` directives
- Export GEO scores and recommendations to a spreadsheet, dashboard, API, or MCP workflow
- Compare the answer readiness of product, documentation, and editorial pages

### What you get

- A 0–100 GEO readiness score and A–F grade for every URL
- `robots.txt` analysis for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended
- Detection of `llms.txt`, `llms-full.txt`, and `sitemap.xml`
- Optional sitemap and sitemap-index discovery with a buyer-controlled total-page cap
- JSON-LD validation and discovered Schema.org types
- Indexability, canonical, metadata, heading, content-depth, freshness, FAQ, list, and table checks
- Prioritized recommendations with a plain-language reason for every action
- Structured JSON results ready for dashboards, spreadsheets, APIs, and MCP clients

### Quick start

1. Open the Actor in [Apify Store](https://apify.com/frere527/geo-ai-visibility-auditor).
2. Add one or more public HTTP(S) pages in **URLs to audit**. To expand them automatically, enable **Discover pages from sitemaps**.
3. Click **Start**. Most single-page audits finish in seconds.
4. Open the **Dataset** tab to review scores, crawler access, evidence, and prioritized fixes.

The default input audits `https://example.com`, so you can run a safe demonstration before changing any settings.

### Input

```json
{
  "startUrls": [
    { "url": "https://example.com" },
    { "url": "https://example.com/blog/guide" }
  ],
  "discoverFromSitemaps": false,
  "sitemapUrls": [],
  "maxUrls": 25,
  "timeoutSecs": 20,
  "maxConcurrency": 3
}
```

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `startUrls` | array | `https://example.com` | One to 100 public HTTP(S) pages to audit. |
| `discoverFromSitemaps` | boolean | `false` | Read each seed origin's `/sitemap.xml` and `Sitemap:` directives, including same-host sitemap indexes. |
| `sitemapUrls` | array | empty | Up to 10 custom public XML sitemap URLs. Providing one enables discovery automatically. |
| `maxUrls` | integer | `25` | Maximum total seed plus discovered pages, capped at 100. |
| `timeoutSecs` | integer | `20` | Per-request timeout from 5 to 60 seconds. |
| `maxConcurrency` | integer | `3` | Concurrent audits from 1 to 10. Keep this low for polite crawling. |
| `userAgent` | string | Actor user agent | User-Agent sent with audit requests. |

Authentication and private pages are intentionally out of scope.

To audit a managed site's sitemap, keep one page from the site as the seed and enable discovery:

```json
{
  "startUrls": [{ "url": "https://your-site.example" }],
  "discoverFromSitemaps": true,
  "maxUrls": 25
}
```

Only pages with the same origin as a seed URL are added. Custom sitemap files may be hosted on another public origin, but URLs inside them still cannot expand the audit to unrelated sites.

### Run through the API

Replace `<APIFY_TOKEN>` with an Apify API token. The synchronous endpoint returns dataset items directly:

```bash
curl -X POST "https://api.apify.com/v2/acts/frere527~geo-ai-visibility-auditor/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"maxConcurrency":1}'
```

With the Apify CLI:

```bash
apify call frere527/geo-ai-visibility-auditor \
  --input '{"startUrls":[{"url":"https://example.com"}]}'
```

### Output example

```json
{
  "url": "https://example.com/blog/guide",
  "score": 74,
  "grade": "C",
  "indexable": true,
  "robotsTxt": {
    "present": true,
    "allowedCrawlers": ["GPTBot", "ClaudeBot", "PerplexityBot"],
    "blockedCrawlers": []
  },
  "structuredDataTypes": ["Article", "Organization"],
  "topRecommendations": [
    "Publish /llms.txt with a concise site summary and links to canonical, useful pages.",
    "Add concise question-answer sections, lists, or comparison tables."
  ]
}
```

Each successful URL creates one dataset item. The overview exposes the requested and final URL, HTTP status, response time, 0–100 score, grade, indexability, AI crawler access, discovery files, structured data types, and top actions. The full item also includes every weighted check and its evidence, so the score is auditable instead of opaque.

### Scoring

The weighted score covers reachability, indexability, titles and descriptions, canonical URLs, headings, substantive content, semantic HTML, structured data, social metadata, crawler policy, AI discovery files, sitemap availability, answer-shaped content, freshness, and response time. The full `checks` array makes every point auditable.

Scores are diagnostic signals, not a promise of ranking or inclusion in an AI answer. Search and answer engines use independent systems that can change over time.

### Pricing

The live price is **$0.025 per successfully audited page**, including each page found through a sitemap. The `maxUrls` input gives the buyer a predictable report-count cap before starting. Failed audits return an error item and do not emit the billable `audit-report` event. There is no separate subscription or minimum run charge; normal Apify platform usage rules still apply to the person running the Actor.

### What this Actor does not do

- It does not guarantee citations, rankings, traffic, or model inclusion.
- It does not log in, bypass access controls, solve CAPTCHAs, or audit private networks.
- It follows declared XML sitemaps only; it does not spider arbitrary links or expand beyond the seed origins.
- It does not modify the audited website.

### Troubleshooting

**A URL is rejected before the audit starts.** The Actor blocks credentials in URLs and private, loopback, link-local, reserved, or local-network destinations, including unsafe redirects. Submit a public HTTP(S) URL.

**The result says `robots.txt` is missing.** This is a valid audit finding, not a run failure. Review the recommendation in the output before publishing a crawler policy.

**A page times out or returns an error item.** Increase `timeoutSecs` up to 60, lower `maxConcurrency`, and confirm the page is publicly reachable. Failed audits are not charged as reports.

**The reported crawler policy differs from a browser visit.** The site may vary responses by network, region, or user agent. Check the returned evidence and verify the live `robots.txt` from the same environment.

### Responsible use

This Actor only requests URLs supplied by the user, declared sitemaps, and four conventional discovery paths on each seed origin. Discovered pages are restricted to the seed origins, sitemap indexes are capped at 10 documents, total pages are capped at 100, and concurrency is capped at 10. No login, evasion, fingerprint spoofing, or access-control bypass is attempted. Private, loopback, link-local, reserved, credential-bearing, and local-network URLs are rejected, including after redirects. Only audit sites you are authorized to assess and follow applicable laws and site terms.

### Support and source

Report reproducible problems through the [GitHub issue tracker](https://github.com/Frere527/geo-ai-visibility-auditor/issues). Include the public URL, run ID, and the unexpected field, but never post API tokens or private data.

The source is available on [GitHub](https://github.com/Frere527/geo-ai-visibility-auditor) under the [MIT License](LICENSE). Focused bug fixes, test cases, and documentation improvements are welcome.

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) pages. Each page produces one scored report.

## `discoverFromSitemaps` (type: `boolean`):

Read sitemap.xml and Sitemap directives in robots.txt for each seed origin, then audit same-origin page URLs up to the maximum total.

## `sitemapUrls` (type: `array`):

Optional public XML sitemap or sitemap-index URLs. Adding one automatically enables sitemap discovery.

## `maxUrls` (type: `integer`):

Maximum number of seed plus sitemap-discovered pages audited in one run.

## `timeoutSecs` (type: `integer`):

Maximum time allowed for each HTTP request.

## `maxConcurrency` (type: `integer`):

Keep this low for polite, reliable auditing.

## `userAgent` (type: `string`):

User-Agent sent while auditing public pages.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "discoverFromSitemaps": false,
  "sitemapUrls": [],
  "maxUrls": 25,
  "timeoutSecs": 20,
  "maxConcurrency": 3,
  "userAgent": "GEO-AI-Visibility-Auditor/1.0 (+https://apify.com/)"
}
```

# Actor output Schema

## `reports` (type: `string`):

Structured report items, one per requested URL.

## `summary` (type: `string`):

Compact JSON summary with counts, scores, grades, and top actions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("frere527/geo-ai-visibility-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("frere527/geo-ai-visibility-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call frere527/geo-ai-visibility-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,frere527/geo-ai-visibility-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FkOcbpu5HtghPS7fk/builds/y02yDSlkJu0TgCMXO/openapi.json