# Robots Txt Inspector (`eichan/robots-txt-inspector`) Actor

- **URL**: https://apify.com/eichan/robots-txt-inspector.md
- **Developed by:** [Eichan Jolly](https://apify.com/eichan) (community)
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.01 / actor start

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Robots.txt Inspector

**One job:** Retrieve the root `/robots.txt` from a single public website and output a structured record for each user-agent **group**. This is a parser, **not** a sitemap comparison tool and **not** a crawler-permissions simulation.

### Who is it for?

Website owners, technical SEO practitioners, developers, and QA teams who need a faithful, auditable summary of rules in one public robots.txt file.

### Input

```json
{"siteUrl":"https://www.python.org"}
```

`siteUrl` accepts either a public `http://` / `https://` site URL or a bare domain such as `www.python.org`. The Actor fetches **only** the `/robots.txt` at that site's origin. It ignores any input page path. No login, key, cookies or custom headers are needed.

The editor prefill is `https://www.python.org`, which has a nonempty public robots.txt with user-agent groups, Allow and Disallow directives. Public availability was checked at preparation time, but can change.

### Output

One dataset item per **consecutive user-agent group**. `userAgent` is deliberately a **list** to preserve a group shared by multiple crawler names without duplicating rules. Examples below are illustrative, not a claim about the live python.org rules:

```json
{
  "userAgent": ["ExampleBot", "OtherBot"],
  "allowRules": ["/public/"],
  "disallowRules": ["/private/"],
  "crawlDelaySeconds": 2.5,
  "sitemaps": ["https://example.org/sitemap.xml"],
  "sourceUrl": "https://example.org/robots.txt",
  "httpStatus": 200,
  "otherDirectives": [],
  "issues": [],
  "status": "ok",
  "note": null,
  "error": null
}
```

`allowRules` and `disallowRules` preserve directive values in their original order and do not try to rank or merge them. `otherDirectives` preserves unrecognized named directives as `{name, value}` objects. Global `Sitemap:` URLs are repeated on every group item. `crawlDelaySeconds` is a numeric value (including decimals) or `null`. `issues` contains plain-English line-numbered warnings; `status` is `ok` or `warning`. If no groups can be parsed, one `invalid` item is emitted. HTTP errors, redirects and network errors each generate one structured error item instead of a traceback.

A missing file (HTTP **404**) produces **one** item with `status: "not_found"`, empty rule arrays, and the note: *No robots.txt file was found. A missing robots.txt imposes no crawling restrictions by this file.* It does **not** mean other access controls or policies do not exist.

### How to run

On Apify, provide `siteUrl` in the input editor. Apify's default key-value store INPUT is fetched over Apify REST using its automatically supplied runtime token, and dataset items are pushed to the default dataset. The Actor never asks the buyer to provide a token.

Locally (from this folder):

```bash
ACTOR_INPUT_JSON='{"siteUrl":"https://www.python.org"}' python src/main.py
```

Alternatively put the input at `storage/key_value_stores/default/INPUT.json`; output appears as JSON item files in `storage/datasets/default/`. `ACTOR_INPUT_JSON` takes precedence over the Apify REST key-value-store input and the local file.

### Limits and safety

- Exactly one public site per run. HTTP and HTTPS **default ports only**. Resolved IP addresses must all be public.
- One GET to `/robots.txt`, 12-second timeout, maximum **512,000 bytes**. No redirect is followed (even if same-host); a redirect is reported instead.
- Does not access or test paths listed in rules, does not retrieve sitemaps, and does not discover subdomains.
- Recognizes `User-agent`, `Allow`, `Disallow`, `Crawl-delay`, `Sitemap` (case insensitive). Comments are removed, unrecognized directive names are retained, malformed lines are noted, not fatal.
- Robots exclusion behavior is crawler-specific; this Actor reports the file's contents rather than asserting Googlebot, Bingbot, etc. obey any particular rule. It does not merge noncontiguous groups by user-agent.
- TLS failures and other connection errors are plain-language dataset errors. Neither live cloud builds nor live Apify storage writes have been tested in the provided offline smoke run.

### Reproducible verification

```bash
python tests/run_smoke.py
python -m unittest discover -s tests -p 'test_*.py' -v
```

The `tests/` smoke runner executes **`python src/main.py` as a subprocess** with valid prefilled-style input and deterministic fixture network responses. See `verification/script-mode-smoke.txt` and `verification/dataset-items.json`. The smoke transport is **simulated**, not a claim of live python.org HTTP execution.

### Store listing

Category: `DEVELOPER_TOOLS` (only). Suggested price: **$0.003 per successful run**, advisory only. No publishing or billing changes were performed.

# Actor input Schema

## `siteUrl` (type: `string`):

A public site URL or bare domain whose root robots.txt will be inspected.

## Actor input object example

```json
{
  "siteUrl": "https://www.python.org"
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing the Actor's result items

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://www.python.org"
};

// Run the Actor and wait for it to finish
const run = await client.actor("eichan/robots-txt-inspector").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "siteUrl": "https://www.python.org" }

# Run the Actor and wait for it to finish
run = client.actor("eichan/robots-txt-inspector").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://www.python.org"
}' |
apify call eichan/robots-txt-inspector --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eichan/robots-txt-inspector"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/GxC68rsYTa3IgBLI2/builds/UIBWgapAan3Nvjb4O/openapi.json
