# Robots.txt Checker – Test URLs, Googlebot & AI Crawlers (`rowfeed/robots-txt-checker`) Actor

Check robots.txt of any website in bulk: is it there, which sitemaps it lists, broken lines, and whether Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and other AI crawlers are blocked. Test your own URLs with Google's rules. $0.50 per 1,000 sites.

- **URL**: https://apify.com/rowfeed/robots-txt-checker.md
- **Developed by:** [Rowfeed](https://apify.com/rowfeed) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 website checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Robots.txt Checker – Test URLs, Googlebot & AI Crawlers

Check the robots.txt of one website or 10,000 at once. For each site you get, in plain words:

- **Is there a robots.txt**, and what Google does when it is missing, broken or the server errors.
- **Which search bots are blocked**: Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider.
- **Which AI crawlers are blocked or allowed**: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot.
- **Test your own URLs**: is `/blog/` or `/search?q=shoes` blocked, for which bot, and which rule decides it.
- **The sitemaps it lists**, lines Google cannot read, crawl-delay, files over Google's 500 KiB limit, and whether the site has an `llms.txt`.

The rules are read the way Google reads them (RFC 9309): the most specific user-agent group, the longest matching rule, Allow wins a tie, `*` and `$` wildcards, and the common typos Google forgives.

Built for SEO audits before and after a launch, checking that a deploy did not block Google by mistake, monitoring competitors' AI crawler policy, publishers deciding what to allow, and AI agents that must respect robots.txt.

### Example output row

```json
{"website": "https://www.nytimes.com", "status": 200, "found": true, "verdict": "robots.txt found", "sitemap_count": 25, "blocks_whole_site": false,
 "search_bots_blocked": [], "ai_bots_blocked": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot", "Google-Extended", "CCBot", "..."],
 "ai_bots_allowed": ["Amazonbot"], "invalid_lines": 0, "has_llms_txt": false,
 "url_tests": [{"url": "https://www.nytimes.com/section/technology", "blocked_for": ["GPTBot"], "rules": {"GPTBot": "Disallow: /"}}]}
```

### How to use

1. Click **Try for free** (or **Start**) at the top of this page to open the input form.
2. Fill in **Websites**. Domains or URLs, up to 10,000. The form comes with a working example, so you can also run it as it is.
3. Optional: adjust **URLs or paths to test (optional)**, **Bots for the URL tests (optional)**, **Also check for llms.txt**. Every field has a sensible default.
4. Click **Start** and wait for the run to finish.
5. Download the results as Excel, CSV or JSON from the **Output** tab, or get them through the Apify API, a schedule or an integration (Make, Zapier, Google Sheets ...).

### Pricing

Pay per event, no subscription:

- **$0.0005 per website** checked ($0.50 per 1,000).
- $0.001 per run start.

A site that does not answer is still a result (it tells you Google sees it as blocked). Text that is not a website is free. Works on the free Apify plan.

### Input

| Field | What it does |
|---|---|
| `websites` | Domains or URLs, up to 10,000. `www` and non-`www` are different sites, as for Google. |
| `testUrls` | Optional paths (`/blog/`, tested on every site) or full URLs. |
| `bots` | Optional user agents for the URL tests. Empty = Googlebot, Bingbot and the 12 AI crawlers. |
| `checkLlmsTxt` | Also look for `/llms.txt` (default on). |

### Good to know

- **What Google does without a file:** a 404 or other 4xx means "crawl everything"; a 5xx, 429 or no answer means Google treats the whole site as blocked until it can read the file. The `verdict` says which applies.
- **Refused:** some sites block automated checkers or whole countries (HTTP 401/403). Then the row says `refused` and draws no conclusion, because Googlebot may get a different answer.
- robots.txt controls crawling, not indexing: a blocked page can still show up in Google if other sites link to it. To check indexing, use the [Google Index Checker](https://apify.com/rowfeed/google-index-checker).
- AI crawler names change; the list is updated when the big AI companies add new ones. Use `bots` to test any other user agent.

### More tools from Rowfeed

- [SEO Site Audit](https://apify.com/rowfeed/seo-site-audit) – every page's on-page SEO, noindex and broken links.
- [Sitemap URL Extractor](https://apify.com/rowfeed/sitemap-url-extractor) – every URL in the sitemaps found here.
- [AI Search Visibility Tracker](https://apify.com/rowfeed/ai-search-visibility-tracker) – whether ChatGPT, Gemini and Perplexity mention your brand.
- [Redirect Checker](https://apify.com/rowfeed/redirect-checker) – status codes and redirect chains in bulk.

# Actor input Schema

## `websites` (type: `array`):

Domains or URLs, up to 10,000. Each site's /robots.txt is read; www and non-www count as different sites, as for Google.

## `testUrls` (type: `array`):

Paths like /blog/ (tested on every site) or full URLs. Each gets the bots it is blocked for and the rule that decides it. Up to 100.

## `bots` (type: `array`):

User agents to test the URLs for. Empty = Googlebot, Bingbot and 12 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot ...).

## `checkLlmsTxt` (type: `boolean`):

One extra request per site to /llms.txt.

## Actor input object example

```json
{
  "websites": [
    "https://www.nytimes.com",
    "https://www.amazon.com",
    "https://apify.com"
  ],
  "testUrls": [
    "/",
    "/blog/",
    "/search?q=shoes"
  ],
  "bots": [
    "Googlebot",
    "GPTBot",
    "ClaudeBot"
  ],
  "checkLlmsTxt": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "https://www.nytimes.com",
        "https://www.amazon.com",
        "https://apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("rowfeed/robots-txt-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "websites": [
        "https://www.nytimes.com",
        "https://www.amazon.com",
        "https://apify.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("rowfeed/robots-txt-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "https://www.nytimes.com",
    "https://www.amazon.com",
    "https://apify.com"
  ]
}' |
apify call rowfeed/robots-txt-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rowfeed/robots-txt-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/OKLNt2DzxeGn6huSY/builds/MvX6PnBEgAkWdmJZL/openapi.json