# Robots.txt Checker (`gridlinetools/robots-txt-checker`) Actor

Test a list of URLs against each site's robots.txt for a chosen user agent (RFC 9309). One row per URL with the allow or disallow verdict, the rule that decided it, sitemaps and crawl-delay. robots.txt is fetched once per host and pages are never downloaded.

- **URL**: https://apify.com/gridlinetools/robots-txt-checker.md
- **Developed by:** [Gridline Tools](https://apify.com/gridlinetools) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 url checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Robots.txt Checker: test URLs against robots.txt for a chosen crawler and see the rule that decided

Provide a list of URLs and the product token of a crawler (`Googlebot`, `bingbot`, a token of your own, or `*` for the rules every robot gets) and the Actor returns one record per URL: may this URL be crawled, which group and which rule decided, and what sitemaps and crawl-delay the host declares. Evaluation follows RFC 9309: a group for the exact token wins over the `*` group, the longest matching pattern wins, and a tie resolves to allow. `robots.txt` is fetched once per host (scheme, host and port) per run with up to 5 redirects; the pages themselves are never requested, so no browser, no proxy and no page content are involved.

### What it is good for

- **SEO checks after a migration or a redesign:** confirm that the new sections are not disallowed by accident and that the pages you want indexed are open to `Googlebot` and `bingbot`.
- **Crawler operators:** audit your own crawler's compliance before you run it. Pass your product token as `userAgent` and see, per URL, the rule your crawler must obey and the `Crawl-delay` the host asks for.
- **Link directories and content teams:** check hundreds of listed URLs in one run and find the ones a search engine is not allowed to fetch.
- **Feeding other tools:** one flat JSON record per URL with the decision, the matched rule and the host's sitemaps, easy to join with your own spreadsheets or to schedule as a recurring run.

### How to use it

1. Open the Actor and paste your list of URLs into the `urls` field, one per line (or pass a JSON array through the API).
2. Enter the robot's product token in `userAgent` (`Googlebot`, `bingbot`, `MyCrawler`) or leave `*` to check the rules that apply to every robot. The other options can stay at their defaults.
3. Click Start. When the run finishes, the status message reads `Processed N of M URLs (X with errors, Y not charged)`.
4. Download the results from the Dataset tab as JSON, CSV or Excel, or read them through the API. Switch the dataset view to "All fields" to see every column.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | list of strings | small self-test list | Absolute, publicly reachable http(s) URLs, one per line; exact duplicates skipped; max 1,000 per run. Private, loopback and `localhost` addresses, embedded credentials and blocked ports are reported as `INVALID_URL` and not charged |
| `userAgent` | string | `*` | Product token of the robot to check, as it appears in `User-agent:` lines of robots.txt. Matched as a whole token, case-insensitively. `*` checks the default group. Allowed: `*` or 1 to 100 characters from `A-Z`, `a-z`, `0-9`, `_` and `-`; any other value is rejected by the input validation (API calls included), and a value that still reaches the Actor unusable falls back to `*` |
| `timeoutSecs` | integer 1 to 60 | 15 | Maximum time for fetching `robots.txt` of one URL, redirects included; on timeout the record gets `error = TIMEOUT` |
| `maxConcurrency` | integer 1 to 50 | 10 | URLs processed in parallel; lower it if many URLs point to one host |

### Output

One dataset record per input URL. Example for `https://www.google.com/search?q=anvil` checked with `userAgent = *`:

```json
{
  "inputValue": "https://www.google.com/search?q=anvil",
  "error": null,
  "errorMessage": null,
  "checkedAt": "2026-09-09T10:12:31.482Z",
  "billable": true,
  "userAgent": "*",
  "robotsUrl": "https://www.google.com/robots.txt",
  "robotsStatus": 200,
  "robotsState": "ok",
  "allowed": false,
  "matchedGroup": "*",
  "matchedRule": { "type": "disallow", "pattern": "/search" },
  "crawlDelay": null,
  "sitemaps": ["https://www.google.com/sitemap.xml"],
  "robotsTruncated": false,
  "redirectCount": 0,
  "robotsFromCache": false
}
```

| Field | Type | Meaning |
|---|---|---|
| `inputValue` | string | URL as provided (credentials masked) |
| `error` | string or null | `INVALID_URL` (not charged), `TIMEOUT`, `NETWORK_ERROR`, `TOO_MANY_REDIRECTS` or `null` |
| `errorMessage` | string or null | Human-readable reason |
| `checkedAt` | string | ISO 8601 timestamp |
| `billable` | boolean | `true` when the record was charged; `false` for records written free of charge (`INVALID_URL`) |
| `userAgent` | string | The product token that was checked, after normalization (`*` when the input value was invalid) |
| `robotsUrl` | string or null | `<scheme>://<host>[:port]/robots.txt` derived from the URL; `null` only for `INVALID_URL` |
| `robotsStatus` | integer or null | HTTP status code of the final response for `robots.txt`; `null` when there was no final response (timeout, network error, too many redirects) |
| `robotsState` | string or null | `ok` (2xx, parsed), `missing` (4xx, including 401, 403, 404 and 410), `unreachable` (5xx, timeout, network error, too many redirects, redirect without a usable target), `invalid` (2xx but the body is not text), `null` for `INVALID_URL` |
| `allowed` | boolean or null | Decision for `userAgent`: from the rules when `robotsState` is `ok`; `true` when the file is `missing` or `invalid` (RFC 9309 section 2.3.1.3); `false` when it is `unreachable` (section 2.3.1.4); `null` for `INVALID_URL` |
| `matchedGroup` | string or null | Token of the group that decided, spelled as in the file, or `*`; for a group declared with several `User-agent:` lines, the first token of that group; `null` when the file has no matching group or was not used |
| `matchedRule` | object or null | `{ "type": "allow" or "disallow", "pattern": "..." }` of the longest matching rule; `null` means no rule in the group matched, so the URL is allowed |
| `crawlDelay` | number or null | `Crawl-delay` of the matched group in seconds, when declared |
| `sitemaps` | list of strings | All `Sitemap:` lines of the file with an absolute http(s) URL, in order, without duplicates; empty when there are none or the file was not used |
| `robotsTruncated` | boolean | `true` when the file exceeded 512,000 bytes and only the complete lines before that limit were processed |
| `redirectCount` | integer | Redirects followed while fetching `robots.txt` (0 to 5) |
| `robotsFromCache` | boolean | `true` when `robots.txt` of this host had already been fetched earlier in the same run |

A `SUMMARY` record in the default key-value store holds the run counters (`inputEntries`, `inputItems`, `skippedDuplicates`, `droppedEmptyOrNonString`, `truncatedOverLimit`, `resumedSkipped`, `processed`, `checked` = charged, `failed`, `notBilled`, `notSavedAfterLimit`, `unexpectedErrors`, `stoppedByChargeLimit`).

### Pricing

This Actor uses Apify pay per event pricing. Each run costs a small fixed start fee plus a price per URL checked; the exact amounts are shown in the Pricing tab. A URL is charged when it received a decision, which includes `unreachable` hosts (the work was done and "unreachable" is the answer RFC 9309 gives). Invalid entries (`INVALID_URL`) are written to the dataset but not charged. There are no additional platform usage fees.

A URL is charged only when its record is written, and every charged record is written. When the run's "Maximum cost per run" is reached, the Actor stops and reports how many URLs were left unprocessed.

### Limits and behavior

- Up to 1,000 URLs per run; exact duplicates are skipped.
- Private, loopback, link-local and `localhost` addresses, URLs with embedded credentials and blocked ports are reported as `INVALID_URL` without any request and without charge. Credentials are masked in `inputValue` (`user:***@host`). A user name without a password is masked as well (`https://***@host/`); a password containing a raw `/`, `\`, `?` or `#` is not recognized as credentials, so percent-encode such characters.
- A run interrupted by a platform migration resumes without processing or charging the same URL twice.
- One `robots.txt` request per host per run: all URLs of the same scheme, host and port share the fetched file (and share its error, if any); `robotsFromCache` marks the records that reused it.
- Up to 5 redirects per `robots.txt`; the sixth gives `TOO_MANY_REDIRECTS` and `robotsState = unreachable`. A redirect to another host is followed (the file at the target counts as the file of the original host); a redirect to a private address or without a `Location` header makes the file `unreachable`.
- The file is read up to 512,000 bytes (RFC 9309 section 2.5); a longer file is cut at the last complete line before that limit and `robotsTruncated` is `true`, so rules that appear later in the file are not applied.
- No retries: `403`, `429` and `5xx` answers for `robots.txt` are reported through `robotsStatus` and `robotsState`, not retried.
- Matching follows RFC 9309: the group for the exact token (case-insensitive) beats `*`, and no matching group means everything is allowed; several groups with the same token are merged; `*` in a pattern matches any string, `$` anchors the end of the path; the path compared is path plus query string, percent-encoding normalized on both sides; the rule with the longest pattern wins and a tie goes to `allow`; an empty `Disallow:` is no rule; line keys are case-insensitive, comments start with `#`, unknown keys and a leading BOM are ignored; `Sitemap:` lines are read regardless of the group; `Crawl-delay:` is taken from the matched group.
- The Actor sends a descriptive `User-Agent` (`GridlineTools-RobotstxtChecker`) so site owners can identify it.

### FAQ

**Why is a URL `allowed` when the site has no robots.txt?** A `404`, like any other 4xx answer (401, 403, 410), counts as "unavailable" under RFC 9309, and a robot may then fetch everything. The record shows `robotsState = missing` and `allowed = true`.

**Why is every URL of a host disallowed after a server error?** When `robots.txt` answers with 5xx, times out or cannot be reached, RFC 9309 tells robots to assume they may fetch nothing. The record shows `robotsState = unreachable`, `allowed = false`, and it is charged because the answer is a real result.

**Does `Googlebot` match `Googlebot-Image`?** No. Tokens are matched as a whole, so `Googlebot-Image` only matches its own group. When the file has no group for your token, the `*` group decides, and without a `*` group everything is allowed.

**What does `matchedRule = null` with `allowed = true` mean?** The matching group has no rule that covers this path, so the URL may be fetched. `matchedGroup` still tells you which group was consulted.

**Is `Crawl-delay` part of the standard?** No, but many hosts declare it and many crawlers honor it, so the Actor reports the value from the matched group. `null` means the group does not declare one or declares a value that is not a number.

**Can I check servers on my own network or on `localhost`?** No. The Actor runs in Apify's cloud and only checks publicly reachable hosts; private, loopback and link-local addresses are rejected as `INVALID_URL` without a request.

### Other Actors by Gridline Tools

Other checks from the same publisher, each with one row per address and the same pricing rule (start fee plus price per checked item):

- [HTTP Status and Headers Checker](https://apify.com/gridlinetools/http-status-headers-checker): Audit a list of URLs in one run: status code, every hop of the redirect chain and the final URL, response time, server and content type, and six key security headers.
- [RDAP Domain Lookup and Expiry Checker](https://apify.com/gridlinetools/rdap-domain-lookup): Query RDAP registries for a list of domains and get registrar, creation and expiry dates, EPP statuses, name servers and DNSSEC in one table.
- [AI Crawler Access Audit](https://apify.com/gridlinetools/ai-crawler-access-audit): See which AI and search crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and 150+ more) each site allows or blocks in robots.txt, and whether llms.txt is published.

### Disclaimer

The Actor sends plain HTTP requests only for the `robots.txt` file of the hosts you list and reads that file; it never requests the pages themselves and never stores page content. You are responsible for having the right to query those URLs and for complying with the terms of service of the sites you check. Gridline Tools does not store the checked URLs or results beyond your own Apify account.

# Actor input Schema

## `urls` (type: `array`):

Absolute, publicly reachable URLs starting with http:// or https://, one per line. Exact duplicates are skipped. Maximum 1,000 per run. Private/localhost addresses, embedded credentials and blocked ports are reported as INVALID_URL and not charged. The default list is a small self-test (Apify also runs it automatically every day to verify the Actor works).

## `userAgent` (type: `string`):

Product token of the robot to check, as it appears in robots.txt "User-agent:" lines (e.g. Googlebot, MyBot). Matched exactly, case-insensitively. Use \* to check the rules that apply to every robot. Allowed: \* or 1-100 characters from A-Z, a-z, 0-9, \_ and -. An invalid value is replaced by \* (reported in the log).

## `timeoutSecs` (type: `integer`):

Maximum time for one URL. On timeout the record gets error = TIMEOUT.

## `maxConcurrency` (type: `integer`):

How many URLs are processed in parallel. Lower it if many URLs point to one host.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://example.com/does-not-exist"
  ],
  "userAgent": "*",
  "timeoutSecs": 15,
  "maxConcurrency": 10
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset: one record per input URL.

## `summary` (type: `string`):

SUMMARY record in the default key-value store: input, processed, checked, failed and invalid counters, charge-limit flags.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("gridlinetools/robots-txt-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("gridlinetools/robots-txt-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call gridlinetools/robots-txt-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gridlinetools/robots-txt-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BGReeNtb0O17YjUTV/builds/3BBpoRQuEhgDmMHy9/openapi.json