# Robots.txt and AI Crawler Access Checker API (`rowcraft/robots-ai-crawler-checker`) Actor

Check which AI and search crawlers a website lets in: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, read from robots.txt by the RFC 9309 rules. Also llms.txt, sitemaps and meta robots. Pay only per website checked.

- **URL**: https://apify.com/rowcraft/robots-ai-crawler-checker.md
- **Developed by:** [Rowcraft](https://apify.com/rowcraft) (community)
- **Categories:** SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 website checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Robots.txt and AI Crawler Access Checker API

Find out what a website's **robots.txt** says to AI and search crawlers. Give the actor a list of domains or addresses; for each one it reads robots.txt exactly as the robots exclusion standard (RFC 9309) defines, works out which of 27 named crawlers may fetch the paths you care about, and returns one row per website: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and the others as true/false columns, a verdict on AI training, AI search and classic search, the sitemaps the file declares, and whether the site publishes `llms.txt`. JSON, CSV or Excel, one polite request at a time.

Price and limits at a glance: **$2.00 per 1,000 websites** on the Free plan and $1.50 on the Business plan, charged only for websites that were read and delivered; up to 100,000 websites and 20 paths per run; about four requests and five seconds per website at the default speed; no personal data.

![Input form of Robots.txt and AI Crawler Access Checker API](https://api.apify.com/v2/key-value-stores/7hbQFQdGq61v2OZ2h/records/robots-ai-crawler-checker-input.png)

### What you get

One row per website, 65 fields, with `null` where something does not apply:

- **The file itself:** the address of robots.txt that was read (after redirects), its HTTP status, what came of it (`found`, `empty`, `missing`, `not_a_robots_file` or `server_error`), its size in bytes and whether it was longer than the 500 KiB a crawler must read
- **A column per crawler:** `gptBot`, `chatGptUser`, `oaiSearchBot`, `claudeBot`, `claudeUser`, `claudeSearchBot`, `perplexityBot`, `perplexityUser`, `googlebot`, `googleExtended`, `googleOther`, `applebot`, `applebotExtended`, `bingbot`, `ccBot`, `amazonbot`, `amznSearchBot`, `amznUser`, `metaExternalAgent`, `metaWebIndexer`, `duckAssistBot`, `duckDuckBot`, `mistralAiUser`, `mistralAiIndex`, `mistralAiTraining`, `yandexBot` and `baiduspider`. `true` means the crawler may fetch every path you checked, `false` means at least one is closed to it
- **Lists and counts:** `allowedBots`, `blockedBots`, `partlyBlockedBots` (allowed on some of your paths, not on all), `explicitlyListedBots` (crawlers the file names in a group of their own, as opposed to those that fall under the `*` rules), `allowedCount`, `blockedCount`
- **Three verdicts:** `aiTrainingPolicy` (`blocks all AI training bots`, `allows all` or `mixed`), `aiSearchPolicy` (the same for AI search bots) and `searchVisibility` (`all search crawlers allowed`, `some search crawlers blocked` or `all search crawlers blocked`)
- **Path by path:** `pathAccess` lists, for every path you gave, which crawlers are allowed and which are blocked
- **Your own crawlers:** any user-agent name you add is checked with the same rules and shown in `customBots`
- **What the site declares:** `sitemaps` and `sitemapCount`, the `crawlDelay` of the `*` group, and `robotsUserAgents`, the crawler names the file mentions (so you also see crawlers that are not on the list)
- **AI files:** `llmsTxtPresent`, `llmsTxtStatus`, `llmsTxtBytes` and the same three for `llms-full.txt`
- **The home page signals:** the `homeMetaRobots` tag and the `homeXRobotsTag` header, read only when robots.txt lets our own crawler in
- **Change tracking:** `policyHash`, which changes only when the policy changes, `notes` with anything that could not be read and why, and the time of the check

The run also writes a `COVERAGE` record with one line per website you asked for and a `RUN_SUMMARY` with counts, how many websites block all AI training bots, and why the run stopped.

### Why this actor

- **The standard, not a guess.** Rules are matched the way RFC 9309 says: lines for several user agents form one group, the longest matching rule wins, `Allow` wins a tie, `*` and a final `$` work inside patterns, percent-encoded paths are compared in normal form, and only the first 500 KiB are read. A missing file means everything is allowed and a server error means everything is closed.
- **Crawler names from the operators.** A name is on the list only when the company that runs the crawler documents it. The source of every one is written in the Data source section, and names that could not be confirmed that way are left out rather than guessed.
- **Your paths, not only the home page.** A site can let GPTBot into `/blog/` and keep it out of `/members/`. Give the paths and every crawler is evaluated on each of them.
- **Your own crawler.** Add the user-agent name of a bot you run, or one you saw in a server log, and get the same answer for it.
- **The related files in one row.** robots.txt, `llms.txt`, `llms-full.txt`, sitemaps, the meta robots tag and the X-Robots-Tag header, so an AI-readiness check is one run.
- **Documented fallbacks.** When robots.txt has no group for Applebot, Bingbot or YandexBot, the actor applies the group those operators say the crawler follows instead (Googlebot, msnbot and Yandex), and says so in the data through `explicitlyListedBots`.
- **Pay only for websites read.** Websites that block the actor, do not exist, are invalid or repeated, or whose policy did not change since your last run are never charged.
- **Polite by design.** robots.txt is read first on every host, our own crawler obeys it, the request rate is one per second by default, and a Crawl-delay the site sets for us is kept.

### How to use it

1. Open the Actor and paste your domains or addresses into **Websites**, one per line. A column copied from a spreadsheet works, and so do entries separated by commas.
2. For a first try keep the list as it is: it holds three large websites with different policies.
3. To check a section of a site, add paths such as `/blog/` or `/products/shoes?page=2` under **Paths to check**. To check only some crawlers, or one of your own, list their names under **Crawlers to check**.
4. Click **Start**. The status line shows progress, and the run stops cleanly at your maximum or at the charge limit you set in the run options.
5. Open the **Dataset** tab: the **Overview**, **AI crawlers**, **Search and archive crawlers**, **Allowed and blocked lists** and **robots.txt, llms.txt and meta tags** views show the main columns. Download JSON, CSV, Excel, XML or HTML, or send the rows to Google Sheets, Slack, Make, n8n or Zapier with the Integrations tab.
6. Open the `COVERAGE` record in the key-value store to see which websites could not be read and why.
7. To watch a list over time, run it on a schedule with **Only new rows since the last run** turned on: a website is delivered, and charged, again only when its policy changed.

### Input

| Field | What it does |
|---|---|
| `websites` | Domains (`example.com`) or addresses (`https://www.example.com/shop`), one per line. robots.txt is read from the top of the site; the path of an address is ignored, use `paths` for that. Up to 100,000 per run. |
| `paths` | The paths the access is calculated for, default `/`. Up to 20, each with an optional query string. A crawler counts as allowed only when every path is allowed. |
| `bots` | Empty = all 27 listed crawlers. Or list names, for example `GPTBot` and `ClaudeBot`. A name that is not on the list is checked as your own crawler; the word `all` adds the whole list to your own names. |
| `maxItems` | Stop after this many delivered rows. 0 = the whole list. |
| `onlyNew`, `stateName` | Deliver a website only when its policy changed since the last run with the same state name |
| `notifyUrl`, `notifyWhenEmpty` | A webhook that receives one message when the run ends; see **Automation** below |
| `requestIntervalMs` | Pause between requests, default 1000 ms, minimum 250 ms |

Example: three websites, all listed crawlers, the home page.

```json
{
  "websites": ["bbc.com", "reuters.com", "etsy.com"]
}
```

Example: two paths on two stores, and a crawler of your own.

```json
{
  "websites": ["etsy.com", "walmart.com"],
  "paths": ["/", "/search?q=shoes"],
  "bots": ["GPTBot", "ClaudeBot", "Googlebot", "MyCompanyBot"]
}
```

Example: a weekly watch on competitors that reports only the websites whose policy changed, with a Slack message.

```json
{
  "websites": ["competitor-one.com", "competitor-two.com", "competitor-three.com"],
  "onlyNew": true,
  "stateName": "competitors",
  "notifyUrl": "https://hooks.slack.com/services/..."
}
```

### Output

65 fields per row. Empty values are `null`, never missing, so CSV and Excel columns are stable.

| Field | Type | Example |
|---|---|---|
| `id` | string | example.com |
| `domain` | string | example.com |
| `inputUrl` | string | `https://www.example.com/shop` |
| `robotsUrl` | string | `https://www.example.com/robots.txt` |
| `robotsStatus` | integer | 200 |
| `robotsOutcome` | string | found |
| `robotsBytes` | integer | 3412 |
| `robotsTruncated` | boolean | False |
| `checkedPaths` | string | / |
| `botsChecked` | integer | 27 |
| `allowedCount` | integer | 12 |
| `blockedCount` | integer | 15 |
| `allowedBots` | string | Googlebot, Bingbot |
| `blockedBots` | string | GPTBot, ClaudeBot |
| `partlyBlockedBots` | string | Googlebot, Bingbot |
| `explicitlyListedBots` | string | GPTBot, CCBot |
| `aiTrainingPolicy` | string | blocks all AI training bots |
| `aiSearchPolicy` | string | allows all |
| `searchVisibility` | string | all search crawlers allowed |
| `pathAccess` | array | \[{"path": "/", "allowedBots": "Googlebot, Bingbot", "blockedBots": … |
| `gptBot` | boolean | True |
| `chatGptUser` | boolean | True |
| `oaiSearchBot` | boolean | True |
| `claudeBot` | boolean | True |
| `claudeUser` | boolean | True |
| `claudeSearchBot` | boolean | True |
| `perplexityBot` | boolean | True |
| `perplexityUser` | boolean | True |
| `googlebot` | boolean | True |
| `googleExtended` | boolean | True |
| `googleOther` | boolean | True |
| `applebot` | boolean | True |
| `applebotExtended` | boolean | True |
| `bingbot` | boolean | True |
| `ccBot` | boolean | True |
| `amazonbot` | boolean | True |
| `amznSearchBot` | boolean | True |
| `amznUser` | boolean | True |
| `metaExternalAgent` | boolean | True |
| `metaWebIndexer` | boolean | True |
| `duckAssistBot` | boolean | True |
| `duckDuckBot` | boolean | True |
| `mistralAiUser` | boolean | True |
| `mistralAiIndex` | boolean | True |
| `mistralAiTraining` | boolean | True |
| `yandexBot` | boolean | True |
| `baiduspider` | boolean | True |
| `customBots` | object | {"MyCrawler": true} |
| `sitemaps` | string | `https://www.example.com/sitemap.xml` |
| `sitemapCount` | integer | 1 |
| `crawlDelay` | number | 10 |
| `robotsUserAgents` | string | gptbot, ccbot |
| `llmsTxtPresent` | boolean | False |
| `llmsTxtStatus` | integer | 404 |
| `llmsTxtBytes` | integer | 15010 |
| `llmsFullTxtPresent` | boolean | False |
| `llmsFullTxtStatus` | integer | 404 |
| `llmsFullTxtBytes` | integer | 482113 |
| `homeUrl` | string | `https://www.example.com/` |
| `homeStatusCode` | integer | 200 |
| `homeMetaRobots` | string | max-image-preview:large |
| `homeXRobotsTag` | string | noindex, noarchive |
| `notes` | string | home page not read: HTTP 403: the site answered with a Cloudflare a… |
| `policyHash` | string | 3f2a9c0d11b7e845 |
| `checkedAt` | string | 2026-10-16T07:03:12.000Z |

#### Reading the three verdicts

`aiTrainingPolicy` covers the crawlers whose operators say they collect content for model training or offer a robots.txt opt-out from it: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, Meta-ExternalAgent and MistralAI-Training. `aiSearchPolicy` covers the crawlers that build an AI search or answer product: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, Meta-WebIndexer, DuckAssistBot and MistralAI-Index. `searchVisibility` covers classic search: Googlebot, Applebot, Bingbot, DuckDuckBot, YandexBot and Baiduspider. Each verdict looks only at the crawlers of its kind that are in your list, and is `null` when none of them is. The remaining crawlers have a column of their own but no verdict: ChatGPT-User, Claude-User, Perplexity-User, Amzn-User and MistralAI-User, which fetch a page when a person asks for it, CCBot, the open Common Crawl archive, and GoogleOther, a general Google crawler.

#### Example rows

Two real rows from a run (long texts shortened here, complete in the dataset):

```json
[
  {
    "id": "bbc.com",
    "domain": "bbc.com",
    "inputUrl": "bbc.com",
    "robotsUrl": "https://www.bbc.com/robots.txt",
    "robotsStatus": 200,
    "robotsOutcome": "found",
    "robotsBytes": 6169,
    "robotsTruncated": false,
    "checkedPaths": "/",
    "botsChecked": 27,
    "allowedCount": 14,
    "blockedCount": 13,
    "allowedBots": "Claude-User, Claude-SearchBot, Googlebot, GoogleOther, Applebot, Bingbot, Meta-WebIndexer, DuckAssistBot, DuckDuckBot, MistralAI-User, Mi…",
    "blockedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Amz…",
    "partlyBlockedBots": null,
    "explicitlyListedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Amz…",
    "aiTrainingPolicy": "mixed",
    "aiSearchPolicy": "mixed",
    "searchVisibility": "all search crawlers allowed",
    "pathAccess": [
      {
        "path": "/",
        "allowedBots": "Claude-User, Claude-SearchBot, Googlebot, GoogleOther, Applebot, Bingbot, Meta-WebIndexer, DuckAssistBot, DuckDuckBot, MistralAI-User, Mi…",
        "blockedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Amz…"
      }
    ],
    "gptBot": false,
    "chatGptUser": false,
    "oaiSearchBot": false,
    "claudeBot": false,
    "claudeUser": true,
    "claudeSearchBot": true,
    "perplexityBot": false,
    "perplexityUser": false,
    "googlebot": true,
    "googleExtended": false,
    "googleOther": true,
    "applebot": true,
    "applebotExtended": false,
    "bingbot": true,
    "ccBot": false,
    "amazonbot": false,
    "amznSearchBot": false,
    "amznUser": false,
    "metaExternalAgent": false,
    "metaWebIndexer": true,
    "duckAssistBot": true,
    "duckDuckBot": true,
    "mistralAiUser": true,
    "mistralAiIndex": true,
    "mistralAiTraining": true,
    "yandexBot": true,
    "baiduspider": true,
    "customBots": null,
    "sitemaps": "https://www.bbc.com/afrique/sitemap.xml, https://www.bbc.com/arabic/sitemap.xml, https://www.bbc.com/bbcx/audio_archive_sitemap.xml, http…",
    "sitemapCount": 38,
    "crawlDelay": null,
    "robotsUserAgents": "amazonbot, magpie-crawler, ccbot, omgili, omgilibot, claude-web, claudebot, anthropic-ai, cohere-ai, bytespider, petalbot, scrapy, appleb…",
    "llmsTxtPresent": false,
    "llmsTxtStatus": 404,
    "llmsTxtBytes": null,
    "llmsFullTxtPresent": false,
    "llmsFullTxtStatus": 404,
    "llmsFullTxtBytes": null,
    "homeUrl": "https://www.bbc.com/",
    "homeStatusCode": 200,
    "homeMetaRobots": "NOODP, NOYDIR",
    "homeXRobotsTag": "bingbot: noarchive",
    "notes": null,
    "policyHash": "3df2ac7ae1fffeb5",
    "checkedAt": "2026-10-15T20:37:09.965Z"
  },
  {
    "id": "nodejs.org",
    "domain": "nodejs.org",
    "inputUrl": "nodejs.org",
    "robotsUrl": "https://nodejs.org/robots.txt",
    "robotsStatus": 200,
    "robotsOutcome": "found",
    "robotsBytes": 199,
    "robotsTruncated": false,
    "checkedPaths": "/",
    "botsChecked": 27,
    "allowedCount": 27,
    "blockedCount": 0,
    "allowedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended…",
    "blockedBots": null,
    "partlyBlockedBots": null,
    "explicitlyListedBots": null,
    "aiTrainingPolicy": "allows all",
    "aiSearchPolicy": "allows all",
    "searchVisibility": "all search crawlers allowed",
    "pathAccess": [
      {
        "path": "/",
        "allowedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended…",
        "blockedBots": null
      }
    ],
    "gptBot": true,
    "chatGptUser": true,
    "oaiSearchBot": true,
    "claudeBot": true,
    "claudeUser": true,
    "claudeSearchBot": true,
    "perplexityBot": true,
    "perplexityUser": true,
    "googlebot": true,
    "googleExtended": true,
    "googleOther": true,
    "applebot": true,
    "applebotExtended": true,
    "bingbot": true,
    "ccBot": true,
    "amazonbot": true,
    "amznSearchBot": true,
    "amznUser": true,
    "metaExternalAgent": true,
    "metaWebIndexer": true,
    "duckAssistBot": true,
    "duckDuckBot": true,
    "mistralAiUser": true,
    "mistralAiIndex": true,
    "mistralAiTraining": true,
    "yandexBot": true,
    "baiduspider": true,
    "customBots": null,
    "sitemaps": "https://nodejs.org/sitemap.xml, https://nodejs.org/learn/sitemap.xml",
    "sitemapCount": 2,
    "crawlDelay": null,
    "robotsUserAgents": null,
    "llmsTxtPresent": true,
    "llmsTxtStatus": 200,
    "llmsTxtBytes": 15010,
    "llmsFullTxtPresent": false,
    "llmsFullTxtStatus": 404,
    "llmsFullTxtBytes": null,
    "homeUrl": "https://nodejs.org/en",
    "homeStatusCode": 200,
    "homeMetaRobots": "index, follow",
    "homeXRobotsTag": null,
    "notes": null,
    "policyHash": "81d3b3106cbb6572",
    "checkedAt": "2026-10-15T20:47:04.363Z"
  }
]
```

Results can be downloaded as JSON, CSV, Excel, XML or HTML from the Dataset tab, or read through the Apify API.

![Results of Robots.txt and AI Crawler Access Checker API in the Apify dataset view](https://api.apify.com/v2/key-value-stores/7hbQFQdGq61v2OZ2h/records/robots-ai-crawler-checker-results.png)

### Pricing

**Pay per result.** You pay for each website read and delivered to your dataset, nothing else: no subscription, no minimum, no charge for websites that could not be read (anti-bot page, a server that refuses robots.txt, not found), for invalid or repeated entries, or for websites skipped by **Only new rows** because their policy did not change.

| Your Apify plan | Price per 1,000 rows | Price per row |
|---|---|---|
| Free plan | **$2.00** | $0.002 |
| Starter plan | **$1.80** | $0.0018 |
| Scale plan | **$1.70** | $0.0017 |
| Business plan and above | **$1.50** | $0.0015 |

Example: 1,000 websites cost $2.00 on the Free plan and $1.50 on the Business plan, however many crawlers and paths each row covers. The Apify platform usage of a run (compute and storage) is billed by Apify separately, as for every Actor; the actor runs in 256 MB and spends most of its time waiting between requests.
A start event of $0.00005 is charged once per run (Apify default).
**Cap your spend:** set *Maximum total charge* in the run options. The actor stops cleanly when the cap is reached and you keep everything delivered until then. The run summary (`RUN_SUMMARY` in the key-value store) says why a run stopped.

### Automation: only new rows, notifications, schedules

**Only new rows since the last run.** Turn on `onlyNew` and the actor remembers, for every website it delivered to you, a fingerprint of its policy (in a key-value store named `rowcraft-robots-ai-crawler-checker-state` in your Apify account). The fingerprint covers the answer for every crawler on every path, the declared sitemaps, the Crawl-delay, the presence of `llms.txt` and `llms-full.txt`, and the meta robots tag and X-Robots-Tag header of the home page. It does not cover the size of the file, comments or the order of the lines. The next run with the same `stateName` delivers a website again only when something in that list changed; websites with the same policy are skipped and not charged. The first run delivers every website it can read. Use a different `stateName` for each list so they keep separate memories.

**Notification when the run ends.** Put a webhook address in `notifyUrl` and the actor sends one JSON POST with the number of delivered rows, links to the dataset (JSON and CSV) and the run. It works with Slack and Discord incoming webhooks (the message is in `text` and `content`), with Make, n8n and Zapier catch hooks, and with your own endpoint. By default it is sent only when at least one row was delivered; `notifyWhenEmpty` sends it every time.

**Schedules.** Create a schedule in Apify Console (daily, weekly, monthly or cron) with `onlyNew` on, and you get a feed of the websites in your list that changed their crawler policy, for the price of those rows only.

**Integrations.** Send the results to Make, n8n, Zapier, Google Sheets, Slack, Google Drive, Airtable or any webhook from the Integrations tab of the Actor, with no code.

**API, SDKs and AI agents.** Every run is available through the Apify API and the official JavaScript and Python clients; the Actor can also be called by AI agents through the Apify MCP server. The example input in the API tab is a working call.

### Use cases

- **AI visibility audit for your own sites:** check that the search crawlers you want, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, are let in, and see in one row whether `llms.txt` exists and a sitemap is declared.
- **Publisher and brand policy:** list your websites or those of a client, see which AI training bots each one blocks, and find the sites that block none by mistake, for example after a CMS or CDN change.
- **Agency reporting:** a table per client with the crawlers allowed and blocked on the sections that matter, such as `/blog/` or `/products/`.
- **Competitor and market research:** which publishers, retailers or software companies block AI training bots, and who lets AI search crawlers in. The `aiTrainingPolicy` column answers it for thousands of domains.
- **Change alerts:** a weekly schedule with **Only new rows** tells you when a website you follow starts blocking GPTBot or ClaudeBot, adds a sitemap or publishes an `llms.txt`.
- **Before you crawl:** check what a site allows for the crawler you operate, with its own name, before you send it there.

Ready-made examples, each with the input filled in (open one and press Start, or save it as your own task):

- [Which news sites block AI crawlers like GPTBot and ClaudeBot?](https://apify.com/rowcraft/robots-ai-crawler-checker/examples/which-news-sites-block-ai-crawlers)
- [AI crawler access of big online stores: who is blocked](https://apify.com/rowcraft/robots-ai-crawler-checker/examples/ai-crawler-access-of-big-online-stores)

### What this actor does not do

- Enforce or test anything: robots.txt is a request to crawlers, not a lock. The actor reports what the file says, not whether a crawler obeys it.
- Tell you what a crawler did: there is no access-log analysis, no crawl statistics and no history; each row is the policy seen at the time of the run.
- Collect contacts: no e-mail addresses, phone numbers, names of people or social profiles, and no page text.
- Crawl a website: only robots.txt, `llms.txt`, `llms-full.txt` and the home page are read, never other pages, the sitemap files, the shop or the checkout.
- Judge the content of `llms.txt`: it reports whether a file exists and how large it is, not whether it follows the proposed format.
- Read rules written in meta tags or headers for individual crawlers: the home page tags are reported as the site sends them, and your own reading decides what they mean for a given crawler.
- Run JavaScript, log in, accept cookie banners, use proxies or solve CAPTCHAs: websites that answer with an anti-bot page are reported as blocked and not charged.

### Limits and fair use

- **One request per second** by default (minimum 250 ms). A website takes four requests: robots.txt, the home page, `llms.txt` and `llms-full.txt`, more when redirects are involved. In our test runs that came to about five seconds per website, roughly 700 websites an hour; slow servers take longer, and a request that gets no answer is given up after 15 seconds (robots.txt) or 25 seconds (other pages).
- **Up to 100,000 websites and 20 paths per run.** The run stops 90 seconds before its timeout with the summary, the coverage and the **Only new rows** memory saved; for long lists raise the timeout or split the list.
- **robots.txt is obeyed** for every address the actor visits, including redirect targets. If it closes the home page, `llms.txt` or `llms-full.txt` to our crawler (`Rowcraft`), that file is not read and the row says so in `notes`. The policy for the other crawlers is still reported, because it needs only robots.txt.
- **Crawl-delay** is not part of RFC 9309. The row reports the value of the `*` group, and the actor itself keeps the Crawl-delay that applies to its own crawler (up to 15 seconds) between its requests to that site.
- **Anti-bot pages.** Websites that answer robots.txt with a bot challenge (Cloudflare, Akamai, DataDome and similar) or refuse the request with an error such as 403, 418 or 429 are reported as `blocked` and not charged, because the file was not read. After 10 blocked websites in a row the run stops and says why in the run summary, because the rest of the list would most likely be blocked too. A home page that is blocked while robots.txt was read still gives a row, with the reason in `notes`.
- **File sizes:** the first 500 KiB of robots.txt, the first 1 MB of the home page and the first 256 KiB of each llms file are read.
- **Lists in a row** are cut at 25 sitemaps and 25 crawler names; `sitemapCount` always holds the full number.

### Data source and licence

The data is what each website sends to every visitor: its robots.txt file, its `llms.txt` and `llms-full.txt` files if it has them, and the response headers and meta tags of its home page. The actor reads each of these once, after checking robots.txt, and delivers the conclusions (allowed or blocked per crawler), the declared sitemap addresses, short header and tag values and file sizes. The text of robots.txt, the files and the page is not stored or delivered.

How robots.txt is read: the Robots Exclusion Protocol, RFC 9309 (https://www.rfc-editor.org/rfc/rfc9309). The `llms.txt` file is a proposal described at https://llmstxt.org/ and is not part of that standard.

Where each crawler name comes from. A name is on the list only when the page of the operator named below documents it. The pages were read on 15 October 2026. The operators change their lists, so a crawler that appears in a later version of a page is not covered until the actor is updated.

| Operator | Crawler names | Source |
|---|---|---|
| OpenAI | GPTBot, ChatGPT-User, OAI-SearchBot | https://platform.openai.com/docs/bots |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot | https://support.anthropic.com/en/articles/8896518 |
| Perplexity | PerplexityBot, Perplexity-User | https://docs.perplexity.ai/guides/bots |
| Google | Googlebot, Google-Extended, GoogleOther | https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers |
| Apple | Applebot, Applebot-Extended | Apple support article 119829 ("About Applebot"), on `support.apple.com/en-us/119829` |
| Microsoft | Bingbot, with the msnbot group as its documented fallback | Bing Webmaster Blog, the posts on the bingbot crawler and on crawl directives, on `blogs.bing.com/webmaster` |
| Common Crawl | CCBot | The CCBot page on `commoncrawl.org/ccbot` |
| Amazon | Amazonbot, Amzn-SearchBot, Amzn-User | The Amazonbot page on `developer.amazon.com/amazonbot` |
| Meta | Meta-ExternalAgent, Meta-WebIndexer | Meta's web crawlers page for webmasters, on `developers.facebook.com/docs/sharing/webmasters/web-crawlers` |
| DuckDuckGo | DuckAssistBot, DuckDuckBot | The DuckAssistBot and DuckDuckBot help pages, on `duckduckgo.com/duckduckgo-help-pages/results` |
| Mistral AI | MistralAI-User, MistralAI-Index, MistralAI-Training | The robots page of the Mistral documentation, on `docs.mistral.ai/robots` |
| Yandex | YandexBot, with the Yandex group as its documented fallback | The Yandex Webmaster help on the User-agent directive, on `yandex.com/support/webmaster` |
| Baidu | Baiduspider | The English robots.txt page on `baidu.com/search/robots_english.html` |

What the operators say about robots.txt, as far as it matters here: OpenAI notes that rules may not apply to ChatGPT-User, Perplexity says that Perplexity-User generally ignores robots.txt, and Amazon says that Amzn-User may not follow all directives, because these three fetch a page at the request of a person. The columns for them show what robots.txt says, not what the crawler will do. Google describes Google-Extended as a robots.txt control token with no effect on Google Search, and Apple describes Applebot-Extended as an opt-out from training its models; neither is a separate crawler that visits pages.

Rowcraft is independent and not affiliated with or endorsed by OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Common Crawl, Amazon, Meta, DuckDuckGo, Mistral AI, Yandex or Baidu. Product names belong to their owners and are used only to say which crawler a rule applies to.

### Privacy: no personal data

Rowcraft actors are built not to collect personal data:

- Only robots.txt, `llms.txt`, `llms-full.txt` and the home page of a website are read, and the row holds crawler names, true/false answers, sitemap addresses, file sizes and two short robots tag values.
- A web address that would contain an e-mail address or a long token is reduced before it is delivered.
- The actor sends no cookies, fills no forms and never logs in.
- No name, e-mail address, phone number or profile is ever requested or stored.

You are responsible for using the data in line with the law that applies to you.

### Other Rowcraft actors

All Rowcraft actors read official or public sources, deliver flat rows without personal data and support only-new runs and notifications:

- [Ashby Jobs Scraper – Job Postings with Salary Ranges](https://apify.com/rowcraft/ashby-jobs-scraper) — Job postings from Ashby job boards (jobs.ashbyhq.com) by company: title, department, team, locations with country, employment and workplace type, published salary range, full text.
- [Broken Link Checker – Find 404s and Dead Links API](https://apify.com/rowcraft/broken-link-checker) — Pay per crawled page, not per link: crawl a site within robots.txt and check every link on each page for 404s, 5xx errors, DNS and TLS failures and redirects.
- [Career Site Jobs Scraper: Greenhouse, Lever, Ashby, Workday](https://apify.com/rowcraft/career-site-jobs-scraper) — Job listings from the career pages of the companies you choose on Greenhouse, Lever, Ashby, Workday, Workable, Personio, Recruitee, Teamtailor and Gem, in one run and one format: title, locations, department, salary when published, full text.
- [Chrome Web Store Scraper: Users, Ratings & Versions](https://apify.com/rowcraft/chrome-web-store-scraper) — Reads public Chrome Web Store listing pages with no login, no proxy and no personal data: users, rating, category, version, update date, permissions and company developers.
- [Domain Email and DNS Checker API: MX, SPF, DMARC, Providers](https://apify.com/rowcraft/domain-email-dns-checker) — Find which email provider, sending services and DNS host each domain uses, from public DNS records only.
- [SIRENE France Company Scraper: SIREN, NAF, Revenue, Status](https://apify.com/rowcraft/france-company-register) — French companies from the official Recherche d'entreprises API (INSEE SIRENE + INPI): SIREN and SIRET, legal form, NAF activity, head-office address, employee band, revenue and net income, labels, creation date and VAT number.
- [Gem Jobs Scraper – Job Board Listings from jobs.gem.com](https://apify.com/rowcraft/gem-jobs-scraper) — Job listings from Gem job boards (jobs.gem.com) by company: title, department, offices with country, employment type, remote, hybrid or on-site, requisition ID, publication date and full ad text, from the public Gem Job Board API.
- [Google Play Apps Scraper: App Details, Installs, Ratings](https://apify.com/rowcraft/google-play-apps-scraper) — Public Google Play app details as flat rows: installs, rating, number of ratings, price, version, last update, minimum Android, ads and in-app purchases.
- [Greenhouse Jobs Scraper – Job Listings by Company Board](https://apify.com/rowcraft/greenhouse-jobs-scraper) — Job listings from Greenhouse job boards by company: title, departments, offices, publication date, full ad text, published pay ranges.
- [Himalayas Jobs Scraper: Remote Jobs API with Salaries](https://apify.com/rowcraft/himalayas-jobs-scraper) — Remote jobs from Himalayas through its public jobs API: title, company, salary range, seniority, categories, the countries and time zones open to candidates, full description and job link.
- [KRS Poland Companies Scraper: Register Data, PKD, Status](https://apify.com/rowcraft/krs-poland-companies) — Polish companies from the official KRS register API of the Ministry of Justice: name, KRS, NIP and REGON numbers, legal form, status, address, PKD activity codes, share capital, branches and registration dates.
- [LEI Records Scraper: Legal Entity Identifier Lookup API](https://apify.com/rowcraft/lei-records-scraper) — Legal Entity Identifier (LEI) records from the Global LEI Index, read through the public GLEIF API: legal name, legal form, register number, addresses, status, renewal date and parent companies.
- [Lever Jobs Scraper – Job Postings from jobs.lever.co](https://apify.com/rowcraft/lever-jobs-scraper) — Job postings from Lever career pages (jobs.lever.co) by company: title, team, department, location, commitment, workplace type, salary range when published, description, requirements and benefits.
- [Norway Company Register Scraper: Brønnøysund Business Data](https://apify.com/rowcraft/norway-company-register-scraper) — Norwegian companies from the Central Coordinating Register for Legal Entities (Brønnøysund, NLOD open data): organisation number, legal form, industry, employees, share capital, addresses, website, VAT and bankruptcy status.
- [Personio Jobs Scraper: Job Listings, Salaries, Departments](https://apify.com/rowcraft/personio-jobs-scraper) — Job ads from any company's Personio careers feed: title, department, office, employment type, schedule, seniority, salary when published, keywords and the full description in the language you choose.
- [Platsbanken Scraper: Sweden Jobs, Employers, Org Numbers](https://apify.com/rowcraft/platsbanken-jobs-scraper) — Every job ad on Platsbanken (Arbetsförmedlingen) from the official JobTech API: headline, employer and organisation number, occupation codes, municipality, coordinates, salary type, deadline and full text.
- [Recruitee Jobs Scraper – Job Listings by Company Career Site](https://apify.com/rowcraft/recruitee-jobs-scraper) — Job listings from Recruitee career sites by company: title, department, locations with country, employment type, remote flag, published salary and full text, one request per company.
- [RSS Feed Finder and Reader – Blog and News Article API](https://apify.com/rowcraft/rss-feed-finder-reader) — Find the RSS, Atom and JSON feeds of any website and read their newest articles, from a domain name alone.
- [SEO Audit Tool: Titles, Meta Tags, Headings and Links API](https://apify.com/rowcraft/seo-audit-tool) — Audit the on-page SEO of any list of pages or websites: titles, meta descriptions, headings, canonicals, indexing, links, images and structured data, with a score and the issues found per page.
- [Shopify Products Scraper: Store Catalog Export to JSON, CSV](https://apify.com/rowcraft/shopify-products-scraper) — Products from any Shopify store's public catalog: title, vendor, type, tags, description, variants with price, compare-at price, SKU and stock, images, options, store currency.
- [Sitemap URL Extractor: Every Page with Last-Modified Dates](https://apify.com/rowcraft/sitemap-url-extractor) — Get every page a site lists in its sitemaps, with last-modified dates, from robots.txt through sitemap indexes and .gz files.
- [SSL Certificate and Security Headers Checker API](https://apify.com/rowcraft/ssl-security-headers-checker) — Check the TLS certificate and the HTTP security headers of every domain in one run, with an A+ to F grade and the exact issues to fix.
- [Store Products Scraper – Shopify, WooCommerce, VTEX](https://apify.com/rowcraft/store-products-scraper) — Export the product catalogue of any Shopify, WooCommerce or VTEX store from its public catalogue endpoint: title, brand, prices, discounts, stock, categories, images and links, one row per product, the same columns on every platform.
- [Structured Data Extractor: JSON-LD, Schema.org, Open Graph](https://apify.com/rowcraft/structured-data-extractor) — Pull the schema.org JSON-LD, Microdata, RDFa and Open Graph of any page list into flat rows: types, product price, availability, ratings, organization, breadcrumbs, with validation issues.
- [Teamtailor Jobs Scraper – Career Site Jobs by Company](https://apify.com/rowcraft/teamtailor-jobs-scraper) — Job listings from Teamtailor career sites by company: title, department, role, locations with city and country, remote or hybrid status, publication date and full ad text, from the official job feed.
- [TED Tenders Scraper: EU Public Procurement Notices API](https://apify.com/rowcraft/ted-tenders-scraper) — EU public procurement notices from TED (Tenders Electronic Daily), the official EU tenders portal, through its public Search API: title, buyer, country, CPV codes, values, deadlines, lots and document links.
- [UK Companies House Scraper: Company Search & Profiles API](https://apify.com/rowcraft/uk-companies-house-scraper) — UK companies from the official Companies House API with your free key: search by name, status, type, SIC code, date or location, or look up company numbers.
- [UK Tenders Scraper: Find a Tender + Contracts Finder API](https://apify.com/rowcraft/uk-tenders-scraper) — UK public procurement notices from Find a Tender and Contracts Finder, the official UK government portals, through their open OCDS APIs: title, buyer, region, CPV codes, values, deadlines, notice type and links.
- [Bulk URL Status and Redirect Chain Checker API](https://apify.com/rowcraft/url-status-redirect-checker) — Check the status code, redirect chain, canonical and indexing signals of every URL in a list, with the final destination and a loop flag.
- [VTEX Products Scraper – Public Catalog Export](https://apify.com/rowcraft/vtex-products-scraper) — Export the catalogue of any VTEX store (Auchan, Chedraui and thousands of retailers in Europe and Latin America) from its public search endpoint: title, brand, EAN, prices, list prices, stock, categories, specifications and images, one row per product.
- [Website Tech Stack Detector – Technology Lookup API](https://apify.com/rowcraft/website-tech-stack-detector) — Find the technologies behind a list of websites: CMS, e-commerce platform, analytics, tag managers, ad pixels, consent tools, chat, CDN and payments, from one polite home-page request per domain.
- [WooCommerce Products Scraper – Store API Export](https://apify.com/rowcraft/woocommerce-products-scraper) — Export every product of a WooCommerce store through its public Store API: title, SKU, prices and sale prices, stock, categories, tags, attributes, ratings, images and links, one row per product.
- [Workable Jobs Scraper: Job Listings from Career Pages](https://apify.com/rowcraft/workable-jobs-scraper) — Job listings from any company's Workable career page: title, department, location, remote or on-site, employment type, description, requirements, benefits and salary when published.
- [Workday Jobs Scraper – Jobs from myworkdayjobs.com Sites](https://apify.com/rowcraft/workday-jobs-scraper) — Job listings from Workday career sites (myworkdayjobs.com) by company: title, locations, country, posting date, time type, requisition ID and the full description.

### FAQ

**Which crawlers does it check?** 27, from 13 operators: OpenAI (GPTBot, ChatGPT-User, OAI-SearchBot), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot), Perplexity (PerplexityBot, Perplexity-User), Google (Googlebot, Google-Extended, GoogleOther), Apple (Applebot, Applebot-Extended), Microsoft (Bingbot), Common Crawl (CCBot), Amazon (Amazonbot, Amzn-SearchBot, Amzn-User), Meta (Meta-ExternalAgent, Meta-WebIndexer), DuckDuckGo (DuckAssistBot, DuckDuckBot), Mistral AI (MistralAI-User, MistralAI-Index, MistralAI-Training), Yandex (YandexBot) and Baidu (Baiduspider).

**Why are Bytespider, cohere-ai, anthropic-ai and AI2Bot not on the list?** On 15 October 2026 the operators' own documentation did not name them as robots.txt user agents in a way we could confirm, and the list holds only names we could confirm. You can still check any of them: type the name under **Crawlers to check** and it is evaluated like the others, and `robotsUserAgents` shows every name a site's robots.txt mentions, so a site that blocks Bytespider is visible either way.

**How do I check a crawler of my own?** Add its user-agent product token, for example `MyCompanyBot`, to **Crawlers to check**. The token is the part before any slash in the user-agent string: letters, digits, underscores and hyphens. The answer appears in `customBots`, in `allowedBots` or `blockedBots`, and in the lists of `pathAccess`.

**What does "allowed" mean when I give several paths?** A crawler is allowed only when every path is allowed. If it can fetch `/` but not `/search`, it is shown as blocked in `blockedBots` and also in `partlyBlockedBots`, and `pathAccess` shows the paths one by one.

**A crawler is not named in robots.txt. Why is it allowed or blocked?** Under RFC 9309 a crawler obeys the group that names it; when no group names it, it obeys the `*` group; when there is none of those either, everything is allowed. `explicitlyListedBots` shows which crawlers the file names in a group of their own.

**Does blocking a crawler in robots.txt really stop it?** Robots.txt is a convention, and the crawlers of the large operators say they follow it, with the exceptions of the user-requested fetchers described in the Data source section. The actor reports what the file says. It cannot tell you whether a crawler that is not run by a well-known operator obeys it.

**Is it legal?** The actor reads a few public files and one page per website, the same ones any visitor or crawler gets, obeys robots.txt and delivers the policy that a website itself publishes, not personal data or page content. Whether and how you may use the results depends on the law that applies to you.

**Does it collect personal data?** No personal data: no e-mail addresses, phone numbers or people's names are read or delivered. See the Privacy section.

**Why is the result empty?** Every website in the list could not be read or was skipped. The `COVERAGE` record gives the reason for each: an anti-bot page or a refusal on robots.txt (`blocked`), no website at that address (`not_found`), an invalid entry or a connection problem (`error`). With **Only new rows** on, websites whose policy did not change are skipped too. Nothing is charged for those, apart from the start event.

**Why is a website with a robots.txt answer such as 403 or 418 not reported as "everything allowed"?** RFC 9309 says a crawler may treat any 4xx answer as "no rules". The actor treats only 404 and 410 that way. A 403, 418 or 429 means the site refused our request, so we do not know its rules, and saying "everything allowed" would be false. Those websites go to `COVERAGE` as `blocked` and are not charged. A server error (5xx) is different: the standard says to treat the site as closed, and the row shows every crawler as blocked with an explanation in `notes`.

**The home page shows no meta robots tag, but the row has `null`. Is something wrong?** No. `homeMetaRobots` and `homeXRobotsTag` are empty when the page has no such tag or header, and also when the home page was not read; `notes` tells which of the two it is.

**Can I see the robots.txt lines behind a verdict?** The row holds the verdict for each crawler, not the lines of the file. The file is one click away: `robotsUrl` is the exact address that was read, and `explicitlyListedBots` tells you which crawlers have a group of their own in it.

**Do I pay for websites that block the actor?** No. Blocked, not found and invalid entries are never charged, and neither are duplicates in your list.

**How does Only new rows work here?** The actor remembers the policy fingerprint of each website delivered to you. The next run with the same state name delivers a website only when its policy changed, so a weekly schedule costs only the websites that changed.

### Support and reviews

**Problems, missing fields, questions:** open an issue on the **Issues** tab of this Actor. We answer within one working day and fix confirmed bugs in the next release; the changelog lists every change.

If this Actor saved you time, a short review on its page helps other users find it and helps us keep improving it.

# Changelog

This Actor's version history is a separate document: https://apify.com/rowcraft/robots-ai-crawler-checker/changelog.md

# Actor input Schema

## `websites` (type: `array`):

Websites to check, one per line: a domain (example.com) or an address (https://www.example.com/shop). robots.txt is read from the top of each site; the path in an address is ignored (use "Paths" below). Duplicates and invalid entries are skipped and never charged.

## `paths` (type: `array`):

The paths the access is calculated for, one per line, for example / or /blog/ or /products/shoes?page=2. A crawler counts as allowed only when every path is allowed. The default is the home page.

## `bots` (type: `array`):

Leave empty to check all 27 listed crawlers (OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Amazon, Meta, Mistral, DuckDuckGo, Common Crawl, Yandex, Baidu). Or list the names you want, for example GPTBot and ClaudeBot. Any other name is checked as your own user-agent token, and the word "all" adds the whole list to your own tokens.

## `maxItems` (type: `integer`):

Stop after this many rows are delivered. 0 = the whole list.

## `onlyNew` (type: `boolean`):

Remember the rows already delivered to you (in a key-value store in your Apify account) and deliver only rows that appeared since the last run with the same state key. The first run delivers everything that matches; later runs cost only the new rows. Ideal for schedules.

## `stateName` (type: `string`):

Name of the memory used by "Only new rows". Use a different name for each schedule or search (for example `stockholm-python`) so they do not share one memory.

## `notifyUrl` (type: `string`):

A webhook address that receives one JSON POST with the number of new rows and links to the dataset: a Slack or Discord incoming webhook, a Make, n8n or Zapier catch hook, or your own endpoint. The message is also in the `text` (Slack) and `content` (Discord) fields.

## `notifyWhenEmpty` (type: `boolean`):

By default the notification is sent only when at least one row was delivered.

## `requestIntervalMs` (type: `integer`):

Default 1000 ms (one request per second). Each website takes four requests: robots.txt, llms.txt, llms-full.txt and the home page (plus redirects), about five seconds at the default. Minimum 250.

## Actor input object example

```json
{
  "websites": [
    "bbc.com",
    "reuters.com",
    "etsy.com"
  ],
  "paths": [
    "/"
  ],
  "bots": [],
  "maxItems": 0,
  "onlyNew": false,
  "stateName": "default",
  "notifyWhenEmpty": false,
  "requestIntervalMs": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

One row per website with its crawler policy, in the default dataset (JSON, CSV, Excel and other formats via the dataset API).

## `runSummary` (type: `string`):

Counts, how many websites allow or block AI training bots, warnings and why the run stopped (RUN_SUMMARY record in the default key-value store).

## `coverage` (type: `string`):

One line per website you asked for: ok, blocked, not_found, error or not_checked, with the reason (COVERAGE record).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "bbc.com",
        "reuters.com",
        "etsy.com"
    ],
    "paths": [
        "/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("rowcraft/robots-ai-crawler-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websites": [
        "bbc.com",
        "reuters.com",
        "etsy.com",
    ],
    "paths": ["/"],
}

# Run the Actor and wait for it to finish
run = client.actor("rowcraft/robots-ai-crawler-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "bbc.com",
    "reuters.com",
    "etsy.com"
  ],
  "paths": [
    "/"
  ]
}' |
apify call rowcraft/robots-ai-crawler-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rowcraft/robots-ai-crawler-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/9Aq8jfmRaxLhUfnjH/builds/XsZDsaGOv0PAq55aA/openapi.json