# Robots & Sitemap Analyzer (`ib4ngz/robots-sitemap-analyzer`) Actor

Analyze robots.txt and XML sitemaps into normalized crawlability intelligence.

- **URL**: https://apify.com/ib4ngz/robots-sitemap-analyzer.md
- **Developed by:** [Iqbal R](https://apify.com/ib4ngz) (community)
- **Categories:** Automation, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 website analyzeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### What does Robots & Sitemap Analyzer do?

**Robots & Sitemap Analyzer checks robots.txt and XML sitemaps for one or many websites and returns structured crawlability data.** Use it to discover sitemap files, evaluate robots rules for a selected user-agent, inspect sitemap coverage, and detect common technical problems without crawling website pages.

Run the Actor on the [Apify platform](https://apify.com/) and connect its results to your workflows through the API, schedules, webhooks, and integrations.

### Why use Robots & Sitemap Analyzer?

- Audit multiple websites in one run.
- Check whether important website areas are restricted by robots.txt.
- Discover declared and common sitemap locations.
- Follow sitemap indexes and summarize their child files.
- Identify missing files, invalid XML, duplicate URLs, invalid dates, and cross-domain entries.
- Feed normalized crawlability data into SEO monitoring, migration checks, lead enrichment, or Website Intelligence products.
- Route requests through Apify Proxy or your own proxies when needed.

### How to use Robots & Sitemap Analyzer

1. Add one or more websites in the **Websites** field.
2. Keep the default user-agent or enter the crawler you want to evaluate.
3. Adjust the per-website sitemap and URL limits if needed.
4. Optionally enable Apify Proxy or provide custom proxies.
5. Start the Actor and open the Dataset to view one result for each website.

### Input

| Field                | Type    | Default             | Description                                                                    |
| -------------------- | ------- | ------------------- | ------------------------------------------------------------------------------ |
| `startUrls`          | array   | `https://apify.com` | Websites to analyze. There is no Actor-defined limit on the number of entries. |
| `userAgent`          | string  | `*`                 | User-agent used to evaluate robots.txt rules.                                  |
| `maxSitemaps`        | integer | `50`                | Maximum sitemap files processed per website.                                   |
| `maxUrls`            | integer | `10000`             | Maximum sitemap URL entries analyzed in detail per website.                    |
| `followSitemapIndex` | boolean | `true`              | Follow child files referenced by sitemap indexes.                              |
| `requestTimeoutSecs` | integer | `30`                | Timeout for each request.                                                      |
| `proxyConfiguration` | object  | proxy disabled      | Apify Proxy or custom proxy settings.                                          |

Example input:

```json
{
    "startUrls": [{ "url": "https://apify.com" }, { "url": "https://crawlee.dev" }],
    "userAgent": "*",
    "maxSitemaps": 50,
    "maxUrls": 10000,
    "followSitemapIndex": true,
    "requestTimeoutSecs": 30,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
}
```

URLs without a protocol are interpreted as HTTPS. Limits apply separately to every website.

### Output

Each website produces one Dataset item:

```json
{
    "inputUrl": "https://apify.com",
    "normalizedUrl": "https://apify.com/",
    "origin": "https://apify.com",
    "domain": "apify.com",
    "robots": {
        "url": "https://apify.com/robots.txt",
        "exists": true,
        "statusCode": 200,
        "evaluatedUserAgent": "*",
        "rules": {
            "allow": [],
            "disallow": []
        },
        "declaredSitemaps": []
    },
    "sitemaps": {
        "found": true,
        "sitemapCount": 1,
        "urlCount": 250,
        "uniqueUrlCount": 248,
        "duplicateUrlCount": 2,
        "sampleUrls": []
    },
    "summary": {
        "robotsTxtExists": true,
        "sitemapExists": true,
        "crawlability": "open",
        "issueCount": 0
    },
    "flags": {
        "robotsMissing": false,
        "robotsBlocksRoot": false,
        "sitemapMissing": false
    },
    "issues": []
}
```

The default key-value store record `OUTPUT` contains the run results under `results`. Dataset results can be downloaded as JSON, CSV, Excel, XML, RSS, or JSONL.

### Data table

| Field           | Description                                                                          |
| --------------- | ------------------------------------------------------------------------------------ |
| `inputUrl`      | URL supplied in `startUrls`.                                                         |
| `normalizedUrl` | Normalized website URL.                                                              |
| `robots`        | Robots file status, groups, applicable rules, crawl delay, and sitemap declarations. |
| `sitemaps`      | Sitemap files, URL statistics, metadata coverage, and representative URL samples.    |
| `summary`       | Main crawlability status and totals.                                                 |
| `flags`         | Boolean indicators for common conditions.                                            |
| `issues`        | Findings with a stable code, severity, message, and optional context.                |

### How much does it cost to analyze website robots and sitemaps?

This Actor uses lightweight HTTP requests instead of a browser. Cost depends mainly on the number of websites, sitemap files, sitemap URLs, response times, and proxy selection. Start with the defaults and lower the per-website limits for large monitoring lists. See the current Apify pricing page for platform allowances and compute pricing.

### Tips and advanced options

- Increase `maxSitemaps` for websites with large sitemap indexes.
- Increase `maxUrls` when detailed coverage statistics matter.
- Lower the limits for fast portfolio-wide checks.
- Enable Apify Proxy when direct requests are blocked or when geographic routing is required.
- Use a specific `userAgent` to inspect the rules relevant to your crawler.
- Schedule recurring runs to detect crawlability changes over time.

Crawlability values are:

- `open` - no applicable crawl restriction was detected.
- `partially_restricted` - applicable robots rules restrict some paths.
- `blocked` - applicable robots rules block the website root.
- `unknown` - robots.txt could not be evaluated reliably.

Issue codes include `ROBOTS_NOT_FOUND`, `ROBOTS_HTTP_ERROR`, `ROBOTS_INVALID`, `ROOT_BLOCKED`, `SITEMAP_NOT_DECLARED`, `SITEMAP_NOT_FOUND`, `SITEMAP_HTTP_ERROR`, `SITEMAP_INVALID_XML`, `SITEMAP_EMPTY`, `SITEMAP_LIMIT_REACHED`, `URL_LIMIT_REACHED`, `SITEMAP_RECURSION_LOOP`, `SITEMAP_CROSS_DOMAIN`, `SITEMAP_INVALID_URL`, and `SITEMAP_INVALID_LASTMOD`.

Run locally:

```bash
npm install
npm run start
```

Run a deployed Actor with the Apify CLI:

```bash
apify call your-username/robots-sitemap-analyzer --input='{"startUrls":[{"url":"https://apify.com"}]}'
```

### FAQ, disclaimers, and support

#### Does the Actor crawl website pages?

No. It analyzes robots.txt and sitemap resources only.

#### Can one run process multiple websites?

Yes. Add all websites to `startUrls`. The Dataset contains one result per website.

#### Does it support sitemap indexes?

Yes. Child sitemaps are processed when `followSitemapIndex` is enabled.

#### Is the crawlability result equivalent to a search engine decision?

No. Search engines can apply provider-specific behavior. The result is structured technical guidance for monitoring and automation.

Only analyze websites when your use complies with applicable laws, terms, and policies. If you find an issue or need a custom integration, use the Actor support channel on Apify Console.

# Actor input Schema

## `startUrls` (type: `array`):

Website URLs to analyze. URLs without a protocol use HTTPS.

## `userAgent` (type: `string`):

User-agent used to evaluate robots.txt rules.

## `maxSitemaps` (type: `integer`):

Maximum sitemap files processed for each website, including sitemap-index children.

## `maxUrls` (type: `integer`):

Maximum sitemap URL entries retained and analyzed in detail for each website.

## `followSitemapIndex` (type: `boolean`):

Recursively process child sitemaps listed in sitemap indexes.

## `requestTimeoutSecs` (type: `integer`):

Maximum duration of each HTTP request.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy or custom proxy configuration.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "userAgent": "*",
  "maxSitemaps": 50,
  "maxUrls": 10000,
  "followSitemapIndex": true,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `output` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ib4ngz/robots-sitemap-analyzer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("ib4ngz/robots-sitemap-analyzer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call ib4ngz/robots-sitemap-analyzer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ib4ngz/robots-sitemap-analyzer"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fZtseFMlV0wO5rZLD/builds/ymVbFmvhj0BVgRz1e/openapi.json