# Robots Rule Tester - Would Google Crawl This URL? (`eliai/robots-rule-tester`) Actor

Test URLs against robots.txt with real Google semantics: longest-match, Allow wins ties, UA group selection (test Googlebot, GPTBot, ClaudeBot...), wildcards. Verdict + the exact rule that decided it + sitemaps. $0.0008 per URL, invalid URLs never charged — vs $0.001-$0.15 measured incumbents.

- **URL**: https://apify.com/eliai/robots-rule-tester.md
- **Developed by:** [Broke to Built](https://apify.com/eliai) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.64 / 1,000 tested urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Robots Rule Tester — Would Google Crawl This URL?

Test any URL against its site's robots.txt with real Google semantics — longest-match rule, Allow wins ties, most-specific user-agent group, `*` and `$` wildcards — and get the verdict plus the exact rule that decided it. Up to 100 URLs per run (robots.txt fetched once per origin), online, by API, or as an agent tool via Apify MCP.

"Is this page blocked for Googlebot?" is answered wrong constantly, because robots.txt evaluation has real rules people eyeball incorrectly: `Allow: /search/about` beats `Disallow: /search` for `/search/about` (longer match), ties go to Allow, and a `GPTBot` group overrides `*` for GPTBot. This actor implements the spec and shows its work.

### What you get

- **allowed** — the verdict for your chosen `userAgent` (default `*`; try `Googlebot`, `GPTBot`, `ClaudeBot`...)
- **matchedRule** — the exact `allow`/`disallow` line that decided it (null = no rule matched)
- **matchedUserAgentGroup** — which UA group applied
- **sitemaps** — every Sitemap: URL declared in robots.txt
- **crawlDelay**, **robotsTxtUrl**, **robotsStatus** — plus honest notes when robots.txt is missing (HTTP 4xx = everything allowed, stated explicitly)
- **Fail-soft**: an invalid URL never fails the run — `{ok: false, error}`, **never charged**.

### Input

```json
{ "urls": ["https://yoursite.com/private/page", "https://yoursite.com/blog/post"], "userAgent": "Googlebot" }
```

Or a single URL via `url`.

### Output (real run)

```json
{
  "url": "https://www.google.com/search?q=test",
  "ok": true,
  "userAgent": "*",
  "allowed": false,
  "matchedRule": { "type": "disallow", "path": "/search" },
  "matchedUserAgentGroup": ["*"],
  "sitemaps": ["https://www.google.com/sitemap.xml"],
  "robotsStatus": 200,
  "testedPath": "/search?q=test"
}
```

In the same run, `https://www.google.com/search/about` comes back `allowed: true` via the longer `Allow: /search/about` rule — the longest-match subtlety, demonstrated live and asserted by our release test.

### Pricing

**$0.0008 per URL tested.** No start fee. Invalid URLs are never charged; bulk URLs on one site share a single robots.txt fetch.

Measured against store incumbents (2026-08-07): scrapeworks/robots-txt charges $0.001 per item, apage/robots-txt-audit $0.005 start + $0.005 per item, alizarin robots\_txt\_analysis $0.1 start + $0.05 per analysis.

### Honest limits

- This evaluates the published robots.txt exactly as a spec-following crawler would. It cannot tell you whether a given bot actually honors robots.txt (some don't), nor does it check meta-robots/X-Robots-Tag — pair with our webpage-meta-robots-auditor for the on-page half of indexability.
- UA matching is substring-based on the group token (per de-facto crawler behavior); pass the product token (e.g. `Googlebot`, not a full UA string) for clearest results.
- `crawl-delay` is reported when declared; Google ignores it, others honor it — reported, not judged.

### FAQ

**Why do I need a tester instead of reading robots.txt?**
Because of precedence: wildcards, longest-match, Allow-vs-Disallow ties, and UA group specificity interact. The matchedRule field shows exactly which line won, so you can fix the file with confidence.

**Can I check AI-crawler access?**
Yes — run the same URLs with `userAgent: "GPTBot"`, `"ClaudeBot"`, `"CCBot"`, etc., and compare verdicts against `*`.

**What happens when there's no robots.txt?**
HTTP 4xx on /robots.txt means everything is allowed — the record says so explicitly in `note` rather than silently passing.

**Does it validate my whole robots.txt?**
It answers per-URL verdicts, which is what audits actually need: feed it your critical URLs and assert `allowed` in CI before every deploy.

**Why did some rows come back `ok: false`?**
The input could not be parsed as a URL. Recorded, never charged.

### Use from code or AI agents

```bash
curl -s "https://api.apify.com/v2/acts/EliAI~robots-rule-tester/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST -H 'Content-Type: application/json' \
  -d '{"urls": ["https://yoursite.com/checkout", "https://yoursite.com/blog"], "userAgent": "Googlebot"}'
```

Agents: connect [Apify MCP](https://mcp.apify.com) and call the `EliAI/robots-rule-tester` tool.

- **Capability:** test one or many URLs against robots.txt with Google longest-match semantics for any user agent
- **Required input:** `url` (string) or `urls` (array); optional `userAgent` (default `*`)
- **Returns:** one record per URL; `allowed` + `matchedRule` are the verdict
- **Bounded:** 100 URLs per run, one robots.txt fetch per origin; failures isolate per URL
- **Side effects:** none (GET robots.txt only — the tested URLs are never fetched)

### For AI agents

This Actor is built to be called by software, not just by people.

- **Mount it directly as an MCP tool** — no Store search, no ranking, just this one tool:
  `https://mcp.apify.com/?actors=eliai/robots-rule-tester`
- **Or call it over HTTP** and get the results in the same request:
  `POST https://api.apify.com/v2/acts/eliai~robots-rule-tester/run-sync-get-dataset-items`
- **Pay with x402, without an Apify account.** This Actor is whitelisted for agentic payments, so an agent holding USDC on Base can buy a prepaid token and spend it here. The minimum purchase is $1, the token balance is an absolute spending cap, and it expires 14 days after purchase.
- **Costs are predictable before you call.** Pricing is pay-per-event (see Pricing above), so an agent can budget a run in advance instead of discovering the bill afterwards.
- **Send only the field you mean.** If you pass the bulk field, it is used on its own; the single-value field is a fallback, never merged into your request. You are charged for the items you sent and nothing else.

# Actor input Schema

## `url` (type: `string`):

The URL to test against the site's robots.txt rules.

## `urls` (type: `array`):

Optional list of URLs to test in one run When you set this, it is used on its own and the single-value field above is ignored, so you are only charged for the items you sent.

## `userAgent` (type: `string`):

The crawler name to evaluate as (e.g. Googlebot, Bingbot). Use \* for the default group.

## `maxUrls` (type: `integer`):

Maximum number of URLs to test in a single run.

## Actor input object example

```json
{
  "url": "https://www.google.com/search",
  "urls": [
    "https://apify.com/store",
    "https://apify.com/admin"
  ],
  "userAgent": "Googlebot",
  "maxUrls": 25
}
```

# Actor output Schema

## `results` (type: `string`):

Every item this run produced, as JSON.

## `resultsCsv` (type: `string`):

The same items as a spreadsheet-ready CSV.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://www.google.com/search",
    "urls": [
        "https://apify.com/store",
        "https://apify.com/admin"
    ],
    "userAgent": "Googlebot"
};

// Run the Actor and wait for it to finish
const run = await client.actor("eliai/robots-rule-tester").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://www.google.com/search",
    "urls": [
        "https://apify.com/store",
        "https://apify.com/admin",
    ],
    "userAgent": "Googlebot",
}

# Run the Actor and wait for it to finish
run = client.actor("eliai/robots-rule-tester").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://www.google.com/search",
  "urls": [
    "https://apify.com/store",
    "https://apify.com/admin"
  ],
  "userAgent": "Googlebot"
}' |
apify call eliai/robots-rule-tester --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eliai/robots-rule-tester"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/SCEhgaByNGUsHc9aK/builds/ewj2ybt0uXiR2PdeQ/openapi.json