# Sitemap Indexability & AI Crawler Audit API (`dobus/sitemap-indexability-ai-crawler-audit`) Actor

Audit sitemap URLs, robots.txt, HTTP status, canonical/noindex signals, and AI crawler access as structured JSON for technical SEO and agents.

- **URL**: https://apify.com/dobus/sitemap-indexability-ai-crawler-audit.md
- **Developed by:** [Afonso Fernandes](https://apify.com/dobus) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 audited url or site summaries

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Indexability & AI Crawler Audit API

Audit sitemaps, robots.txt, HTTP status, canonical tags, noindex directives, and AI crawler access for domains or URL lists. The Actor returns structured JSON rows in the default dataset, ready for SEO workflows, site migrations, content operations, and AI-agent checks.

### What it checks

- Discovers `robots.txt` and sitemap URLs from a domain.
- Parses sitemap indexes and nested sitemaps.
- Audits sitemap URLs for HTTP status, redirects, final URL, and content type.
- Parses HTML for `meta robots`, `noindex`, canonical URL, and page title.
- Evaluates robots.txt access for `*`, `Googlebot`, and configurable AI crawler user agents such as `GPTBot`, `ChatGPT-User`, `ClaudeBot`, `PerplexityBot`, `Google-Extended`, and `CCBot`.
- Returns one `siteSummary` row per site and one `url` row per audited URL.

### Common use cases

- Technical SEO audits before site launches and migrations.
- Detecting sitemap URLs that redirect, 404, or are noindexed.
- Checking whether important pages are blocked by robots.txt.
- Monitoring whether AI crawlers can access public documentation or content hubs.
- Feeding clean crawlability data into AI agents and MCP workflows.

### Input example

```json
{
  "domains": ["example.com"],
  "maxUrlsPerSite": 100,
  "checkHttpStatus": true,
  "checkIndexability": true,
  "checkRobotsTxt": true,
  "checkAiCrawlerAccess": true,
  "requestsPerSecond": 2
}
```

### Output example

```json
{
  "recordType": "url",
  "domain": "example.com",
  "siteUrl": "https://example.com",
  "url": "https://example.com/product/widget",
  "sourceType": "sitemap",
  "sitemapUrl": "https://example.com/sitemap.xml",
  "httpStatus": 200,
  "finalUrl": "https://example.com/product/widget",
  "robotsAllowed": true,
  "googlebotAllowed": true,
  "aiCrawlerAllowed": false,
  "aiCrawlerAccess": {
    "GPTBot": false,
    "ClaudeBot": true
  },
  "metaRobots": "index,follow",
  "noindex": false,
  "canonicalUrl": "https://example.com/product/widget",
  "canonicalMatches": true,
  "indexable": true
}
```

### Pricing

Launch pricing is prepared for pay-per-event monetization:

- USD 0.01 per Actor start.
- USD 0.003 per audited URL or site summary dataset item.

This keeps small checks inexpensive while matching the recurring value of technical SEO and AI crawler monitoring.

### Responsible usage

Use this Actor for websites you own, manage, or are allowed to audit. It reads public `robots.txt`, sitemap, and page metadata, and includes rate limiting so runs can be kept conservative.

# Actor input Schema

## `domains` (type: `array`):

Domains or website homepages to audit. The Actor discovers robots.txt and sitemap.xml automatically.

## `sitemapUrls` (type: `array`):

Optional sitemap.xml or sitemap index URLs to audit directly.

## `urls` (type: `array`):

Optional individual URLs to audit even if they are not present in a sitemap.

## `maxUrlsPerSite` (type: `integer`):

Safety limit for sitemap URLs checked per website.

## `includeNestedSitemaps` (type: `boolean`):

Follow nested sitemaps found inside sitemap index files.

## `sameHostOnly` (type: `boolean`):

Keep only URLs whose host matches the audited domain or sitemap host.

## `checkHttpStatus` (type: `boolean`):

Fetch each URL and return status code, final URL, redirect flag, and content type.

## `checkIndexability` (type: `boolean`):

Parse HTML for meta robots noindex and canonical tags when HTML is returned.

## `checkRobotsTxt` (type: `boolean`):

Read robots.txt and report whether standard crawlers are allowed to fetch each URL.

## `checkAiCrawlerAccess` (type: `boolean`):

Report whether common AI crawler user agents are allowed by robots.txt.

## `aiCrawlerUserAgents` (type: `array`):

AI-related user agents to evaluate against robots.txt.

## `requestsPerSecond` (type: `number`):

Per-run rate limit. Keep this conservative for third-party sites.

## `timeoutSecs` (type: `integer`):

Timeout for robots.txt, sitemap, and URL fetches.

## `customUserAgent` (type: `string`):

Optional identifiable User-Agent for HTTP requests. Include a contact URL or email when auditing sites you control.

## Actor input object example

```json
{
  "domains": [
    "apify.com"
  ],
  "maxUrlsPerSite": 100,
  "includeNestedSitemaps": true,
  "sameHostOnly": true,
  "checkHttpStatus": true,
  "checkIndexability": true,
  "checkRobotsTxt": true,
  "checkAiCrawlerAccess": true,
  "aiCrawlerUserAgents": [
    "GPTBot",
    "ChatGPT-User",
    "ClaudeBot",
    "PerplexityBot",
    "Google-Extended",
    "CCBot"
  ],
  "requestsPerSecond": 2,
  "timeoutSecs": 20,
  "customUserAgent": "Sitemap Indexability AI Crawler Audit (+https://apify.com)"
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset containing one item per audited URL plus one site summary item per website.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("dobus/sitemap-indexability-ai-crawler-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("dobus/sitemap-indexability-ai-crawler-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com"
  ]
}' |
apify call dobus/sitemap-indexability-ai-crawler-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dobus/sitemap-indexability-ai-crawler-audit"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LwYtwkH5kWggkXiYT/builds/zTFCyHrmvYtLgHqef/openapi.json