# Robots.txt Tester — Is a URL Allowed to Crawl? (`hipersoft/robots-txt-checker`) Actor

Test whether URLs are allowed or blocked by a site's robots.txt for any crawler user-agent. Returns allowed/blocked, the matched rule, crawl-delay and the site's sitemaps. Great for SEO audits and crawler compliance. One row per URL as JSON, CSV or Excel.

- **URL**: https://apify.com/hipersoft/robots-txt-checker.md
- **Developed by:** [hiper soft](https://apify.com/hipersoft) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.00015 / url checked

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Robots.txt Tester — Is a URL Allowed to Crawl?

Check whether URLs are **allowed or blocked by a site's robots.txt** with the **Robots.txt Tester**. Pass a list of URLs and a crawler user-agent, and get back — for each URL — whether it's crawlable, the exact rule that matched, the crawl-delay and the site's sitemaps. Export as **JSON, CSV or Excel**.

Built for **SEO audits, crawler and scraper compliance, indexing checks and site migrations**, it applies real robots.txt matching rules so you know exactly what a bot is allowed to fetch.

### What it does

- ✅ **Allowed / blocked** — a clear verdict for each URL and user-agent.
- 🎯 **Matched rule** — the exact `Allow`/`Disallow` line that decided it (longest-match, Allow-wins precedence).
- 🤖 **Any user-agent** — test as `*`, `Googlebot`, `Bingbot`, `GPTBot` or any bot.
- ⏱️ **Crawl-delay** — the crawl-delay declared for that agent, if any.
- 🗺️ **Sitemaps** — the sitemap URLs listed in the site's robots.txt.
- ⚡ **Bulk & cached** — tests many URLs and caches each site's robots.txt for speed.
- 📤 **Export anywhere** — JSON, CSV or Excel, or straight into an SEO tool or report.

### Example output

```json
{
  "url": "https://en.wikipedia.org/wiki/Special:Search",
  "userAgent": "*",
  "allowed": false,
  "matchedRule": "disallow: /wiki/Special:",
  "robotsFound": true,
  "sitemaps": ["https://en.wikipedia.org/sitemap.xml"]
}
```

### How to use it

1. Add the **URLs** you want to test.
2. Set the **crawler user-agent** (e.g. `Googlebot`).
3. Run, and download the results as **JSON, CSV or Excel**.

### Input fields

| Field | Description |
|-------|-------------|
| **URLs to test** | Full URLs (or bare domains) to test. |
| **Crawler user-agent** | The bot to test as (`*`, `Googlebot`, `GPTBot`…). |
| **Max URLs** | Maximum URLs to test (0 = no limit). |

### Output fields

`url`, `userAgent`, `allowed`, `matchedRule`, `crawlDelay`, `robotsUrl`, `robotsFound`, `sitemaps`, `error`.

### Popular use cases

- **SEO audits** — confirm important pages aren't accidentally blocked.
- **Crawler compliance** — respect robots.txt before scraping a site.
- **Indexing checks** — verify what search bots can and can't fetch.
- **Migrations** — catch robots.txt mistakes after a site move.
- **AI crawler policy** — see whether bots like GPTBot are allowed.

### FAQ

**Do I need an account or key?**
No. Add URLs and run.

**How is the verdict decided?**
By standard robots.txt matching — the most specific (longest) matching rule wins, and `Allow` beats `Disallow` on a tie, for the selected user-agent.

**Can I test different bots?**
Yes — set any user-agent (Googlebot, Bingbot, GPTBot, etc.) to see rules that apply to it.

**Can I export to Excel or Google Sheets?**
Yes — results download as JSON, CSV or Excel and integrate with SEO tools and reports.

***

Test **robots.txt allow/block rules** for any URL and crawler — with the matched rule, crawl-delay and sitemaps — clean and export-ready for SEO and compliance.

# Actor input Schema

## `urls` (type: `array`):

Full URLs to test against each site's robots.txt — e.g. 'https://example.com/page', 'example.com/admin'. Bare domains are treated as HTTPS.

## `userAgent` (type: `string`):

The crawler user-agent to test as — e.g. '\*', 'Googlebot', 'Bingbot', 'GPTBot'.

## `maxItems` (type: `integer`):

Maximum number of URLs to test (0 = no limit).

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Special:Search",
    "https://example.com/"
  ],
  "userAgent": "*",
  "maxItems": 0
}
```

# Actor output Schema

## `results` (type: `string`):

The results as dataset items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Special:Search",
        "https://example.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hipersoft/robots-txt-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Special:Search",
        "https://example.com/",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("hipersoft/robots-txt-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Special:Search",
    "https://example.com/"
  ]
}' |
apify call hipersoft/robots-txt-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hipersoft/robots-txt-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sT6eG72iaB6oMM4XP/builds/zJzhoXYqDxCGr5Hh4/openapi.json