# Sitemap & Robots.txt Analyzer - AI Crawler Check 💰$10/1k (`flyingnimbus/sitemap-robots-analyzer`) Actor

Audit any website's robots.txt and sitemaps in seconds: which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked, how many URLs are listed, how fresh they are, and which technical SEO issues to fix.

- **URL**: https://apify.com/flyingnimbus/sitemap-robots-analyzer.md
- **Developed by:** [Rakshit Solanki](https://apify.com/flyingnimbus) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$10.00 / 1,000 website analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap & Robots.txt Analyzer

Audit any website's robots.txt and sitemaps in seconds: which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked, how many URLs are listed, how fresh they are, and which technical SEO issues to fix.

### What does Sitemap & Robots.txt Analyzer do?

- AI crawler access report for 16 bots
- Recursive sitemap index parsing with gzip support
- Freshness stats from lastmod and top site sections
- Detects sitemap URLs blocked by robots.txt and missing sitemap declarations
- Checks for llms.txt

### How to use it

1. Click **Try for free** and open the **Input** tab.
2. Fill in the fields below, or keep the defaults for a quick test.
3. Press **Start**. Results appear in the **Output** tab within seconds to a few minutes.
4. Download as JSON, CSV or Excel, or call the actor from the API, a schedule, Make, Zapier or an AI agent through the Apify MCP server.

### Input

| Field | Type | Description |
|---|---|---|
| `websites` | array | Domains or URLs, one per line. |
| `maxSitemapFiles` | integer | How many sitemap files to fetch per site. |
| `maxUrlsToCheckAgainstRobots` | integer | Sample size for robots conflicts. |

#### Example input

```json
{
  "websites": [
    "apify.com",
    "nytimes.com"
  ],
  "maxSitemapFiles": 20,
  "maxUrlsToCheckAgainstRobots": 5000
}
```

### Output

Each result is a flat JSON object that you can open in the **Output** tab, export to CSV or Excel, or read through the API. See the dataset views for the main columns.

### How much does it cost to use Sitemap & Robots.txt Analyzer?

Pricing is pay per event: **$0.01 per website**. You pay only for results saved to the dataset, plus nothing extra for platform usage on the free plan limits. Apify's free monthly credit lets you test it before paying.

### FAQ

**Does it crawl the pages?**  No. It only reads robots.txt, llms.txt and sitemap files.

**What counts as blocked?**  A bot is blocked if robots.txt disallows the site root for that user agent.

**Can I run it for many domains?**  Yes. Add as many as you like; each domain is one billable result.

### Is it legal?

The actor reads public, open-licence or government data only and does not collect private information. You are responsible for using the results in line with the source's terms and applicable laws.

### Use it with AI agents (MCP)

This actor is agent-ready: it is a plain tool with a documented JSON input and a flat JSON output, so Claude, ChatGPT, Cursor and other agents can call it through the [Apify MCP server](https://mcp.apify.com).

Add it to an MCP client with this URL (Apify account or agentic payment required):

```
https://mcp.apify.com/?tools=flyingnimbus/sitemap-robots-analyzer
```

Example prompt: "Use the Sitemap & Robots.txt Analyzer tool with the example input below and summarise the results."

Calls are pay per event ($0.01 per website), so an agent can estimate the cost before it runs. Empty or failed lookups return a status record instead of failing the run.

### More actors by flyingnimbus

- [Property Details by Address API](https://apify.com/flyingnimbus/property-details-by-address-api)
- [Schools Near Address Finder API](https://apify.com/flyingnimbus/schools-near-address-finder)
- [School Rating Converter & Equivalence API](https://apify.com/flyingnimbus/school-rating-normalizer)
- [Building Permits Multi-City API](https://apify.com/flyingnimbus/building-permits-multi-city)
- [Company Registry Lookup (LEI + SEC)](https://apify.com/flyingnimbus/company-registry-lookup)
- [Restaurant Inspections API - NYC, Chicago, Toronto](https://apify.com/flyingnimbus/restaurant-inspections-multi-city)
- [CVE Risk Enricher - NVD, CISA KEV & EPSS](https://apify.com/flyingnimbus/cve-risk-enricher)
- [SEC Insider Trades (Form 4) - Buys & Sells](https://apify.com/flyingnimbus/sec-insider-trades)
- [SEC 13F Holdings Tracker](https://apify.com/flyingnimbus/sec-13f-holdings)
- [EU Company Registry Search](https://apify.com/flyingnimbus/eu-company-registry-search)
- [Canada Recalls & Safety Alerts Monitor](https://apify.com/flyingnimbus/canada-recalls-monitor)
- [Email Validator - MX, SPF & DMARC](https://apify.com/flyingnimbus/email-mx-validator)
- [Website Tech Stack Detector](https://apify.com/flyingnimbus/tech-stack-detector)
- [PDF Text & Table Extractor](https://apify.com/flyingnimbus/pdf-table-extractor)
- [URL to Markdown + Metadata](https://apify.com/flyingnimbus/url-to-markdown-metadata)
- [SSL Certificate Expiry Checker](https://apify.com/flyingnimbus/ssl-certificate-checker)
- [VIN Decoder + Recalls API](https://apify.com/flyingnimbus/vin-decoder-recalls)
- [ClinicalTrials.gov Search API](https://apify.com/flyingnimbus/clinical-trials-search)
- [US Federal Contracts & Grants (USAspending)](https://apify.com/flyingnimbus/federal-contract-awards)
- [Federal Register Search API](https://apify.com/flyingnimbus/federal-register-search)
- [FDA Recalls & Drug Labels API](https://apify.com/flyingnimbus/openfda-recalls-labels)
- [Eurostat Data API](https://apify.com/flyingnimbus/eurostat-data-api)
- [UK Government Tenders (Contracts Finder)](https://apify.com/flyingnimbus/uk-tenders-search)

### Feedback

Found a bug or need another field or city? Open an issue on the actor's **Issues** tab and it will be looked at quickly.

# Actor input Schema

## `websites` (type: `array`):

Domains or URLs, one per line.

## `maxSitemapFiles` (type: `integer`):

How many sitemap files to fetch per site.

## `maxUrlsToCheckAgainstRobots` (type: `integer`):

Sample size for robots conflicts.

## Actor input object example

```json
{
  "websites": [
    "apify.com",
    "nytimes.com"
  ],
  "maxSitemapFiles": 20,
  "maxUrlsToCheckAgainstRobots": 5000
}
```

# Actor output Schema

## `results` (type: `string`):

One row per result.

## `overview` (type: `string`):

Key columns.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "apify.com",
        "nytimes.com"
    ],
    "maxSitemapFiles": 20,
    "maxUrlsToCheckAgainstRobots": 5000
};

// Run the Actor and wait for it to finish
const run = await client.actor("flyingnimbus/sitemap-robots-analyzer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websites": [
        "apify.com",
        "nytimes.com",
    ],
    "maxSitemapFiles": 20,
    "maxUrlsToCheckAgainstRobots": 5000,
}

# Run the Actor and wait for it to finish
run = client.actor("flyingnimbus/sitemap-robots-analyzer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "apify.com",
    "nytimes.com"
  ],
  "maxSitemapFiles": 20,
  "maxUrlsToCheckAgainstRobots": 5000
}' |
apify call flyingnimbus/sitemap-robots-analyzer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,flyingnimbus/sitemap-robots-analyzer"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/04OAbhDPbNjvKL7wl/builds/b1uJNe965q96bH7Sy/openapi.json