# Robots.txt Auditor | Crawl Rules & Sitemap Discovery (`sbio-apps/robots-txt-auditor`) Actor

Read robots.txt for supplied websites and export user-agent groups, allow/disallow rules, sitemap declarations, extension directives and parsing warnings. Identify missing files and compare content hashes in your crawl configuration workflows.

- **URL**: https://apify.com/sbio-apps/robots-txt-auditor.md
- **Developed by:** [Austin Aryain](https://apify.com/sbio-apps) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$5.00 / 1,000 completed checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### Robots.txt Auditor | Crawl Rules & Sitemap Discovery

Read robots.txt for supplied websites and export user-agent groups, allow/disallow rules, sitemap declarations, extension directives and parsing warnings. Identify missing files and compare content hashes in your crawl configuration workflows.

### How it works

Supply a website URL. The Actor replaces the path with /robots.txt on that origin, follows up to four HTTP redirects, and parses the returned UTF-8 text. It groups consecutive user-agent declarations with their allow and disallow rules, collects Sitemap directives, and preserves unknown directives such as Crawl-delay or Content-Signal separately. The content hash lets your own workflow detect edits between runs. Missing files returning 404 or 410 are useful completed results and are charged.

### Quick start

1. Enter one or more public URLs in the Input tab, beginning with the supplied example.
2. Set a maximum run charge. A completed check costs $0.005; checking 10 sources once costs $0.05.
3. Start the Actor and inspect the dataset. Download JSON, CSV or Excel, or consume results through the Apify API.
4. Inspect the OUTPUT run summary as well as the dataset: failed or unprocessed inputs appear there. Save the input as a task if you want to schedule future runs.

### Pricing

**$0.005 per completed check ($5 per 1,000), with platform usage included.** There are no separate Actor-start or dataset-item fees. Empty and unchanged successful checks are charged. The maximum charge is checked before each source request and again before output. Failed network or format checks are free; see the specific HTTP-response cases below. Billing is per completed source check, not per nested array item, extracted URL, change or schema block.

### Limits and interpretation

The robots file must fit within 500 KB and 10,000 directives. This is a directive inventory, not a crawler permission decision or a guarantee of search-engine behavior. It does not evaluate a target path against allow/disallow patterns, interpret extensions, visit sitemap links, or crawl the website. HTTP 401, 403, 429, 5xx, HTML masquerading as robots text, and network failures are free errors. There is no retained cross-run state.

A run accepts 1-50 unique input URLs and requests them sequentially. Each check has an 18-second network deadline; new checks stop after 160 seconds. Use a 240-second run timeout and 512 MB memory. If the time or charge limit stops a batch, OUTPUT lists uncheckedUrls for a later run. No response exceeding the configured byte limit is accepted, and a complete record must fit within 6 MB. The Actor permits only public HTTP(S) destinations on standard ports, pins a validated DNS address per request, and refuses redirects into private networks or from HTTPS to HTTP.

The Actor uses direct HTTP requests, without a browser, residential proxy, login, CAPTCHA solving or access-control bypass. Rate limits and blocks may prevent checks. Avoid secret-bearing URLs. Results describe the source and network observed at check time.

### Integrations and support

Connect the dataset and OUTPUT summary to your own n8n, Make, Zapier or API workflow. This Actor produces data; it does not automatically send email, Slack messages or webhooks to third parties. No external account credentials are needed for the supplied public examples. Report reproducible issues in the Actor Issues tab, including a non-sensitive input and run link. This is an independent utility and is not endorsed by the websites, standards bodies or services it reads.

### Input example

```json
{
  "urls": [
    "https://www.wikipedia.org/"
  ]
}
```

See the Input tab for all supported fields. Results are available through the dataset API and can be downloaded as JSON, CSV or Excel.

### Output fields

| Field | Meaning |
|---|---|
| inputUrl | Normalized supplied URL. |
| checkedAt | Check time in ISO format. |
| robotsUrl | Final robots.txt URL. |
| httpStatus | Observed HTTP status. |
| exists | False for HTTP 404 or 410. |
| groups | User-agent groups and their ordered allow/disallow rules. |
| sitemaps | Unique declared sitemap values; not downloaded or validated. |
| extensions | Other directives such as crawl-delay with observed group context. |
| warnings | Line numbers and structural parsing warnings. |
| contentHash | SHA-256 of decoded robots text; null for absent files. |

### Output example

Example from a public source check; live values vary. Long items, changes, groups and blocks arrays are shortened to two entries here for readability; the actual record contains the complete arrays within the documented limits.

```json
{
  "inputUrl": "https://www.wikipedia.org/",
  "checkedAt": "2026-09-07T21:05:12.189Z",
  "robotsUrl": "https://en.wikipedia.org/robots.txt",
  "httpStatus": 200,
  "exists": true,
  "groups": [
    {
      "userAgents": [
        "MJ12bot"
      ],
      "rules": [
        {
          "directive": "disallow",
          "path": "/"
        }
      ]
    },
    {
      "userAgents": [
        "Mediapartners-Google*"
      ],
      "rules": [
        {
          "directive": "disallow",
          "path": "/"
        }
      ]
    }
  ],
  "sitemaps": [
    "https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0"
  ],
  "extensions": [
    {
      "directive": "crawl-delay",
      "value": "5",
      "userAgents": [
        "SemrushBot"
      ]
    }
  ],
  "warnings": [],
  "contentHash": "62ede7ec5bff7cd9bdbd97ee37273e0162db5b1930db11ee0665b1fa1254fcc3"
}
```

# Actor input Schema

## `urls` (type: `array`):

1-50 explicit public URLs on standard ports. Exact duplicate input URLs are checked once.

## Actor input object example

```json
{
  "urls": [
    "https://www.wikipedia.org/"
  ]
}
```

# Actor output Schema

## `checks` (type: `string`):

Priced dataset records, including observed unchanged checks or HTTP audit errors as documented.

## `summary` (type: `string`):

Free errors, spending/time stops and unprocessed inputs.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.wikipedia.org/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("sbio-apps/robots-txt-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://www.wikipedia.org/"] }

# Run the Actor and wait for it to finish
run = client.actor("sbio-apps/robots-txt-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.wikipedia.org/"
  ]
}' |
apify call sbio-apps/robots-txt-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sbio-apps/robots-txt-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eljVGGgVp852aTmcD/builds/zYF1mugHVBNRuE0cb/openapi.json