Pricing

$11.00 / 1,000 robots.txt ai policy rows

Try for free

Go to Apify Store

robots.txt Parser & AI Crawler Block Checker

Try for free

Check AI crawler access. No API key / no login. Open Input → leave defaults → Start (1 domain ≈ USD 0.011/row): https://console.apify.com/actors/Mkf2MMndaW7UdtuOP/input

Pricing

$11.00 / 1,000 robots.txt ai policy rows

Rating

0.0

(0)

Developer

naoki anzai

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

SEO tools

Developer tools

You can access the robots.txt Parser & AI Crawler Block Checker programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "PlvXZlzbzpYMEey9H"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/taroyamada~robotstxt-ai-checker/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-taroyamada-robotstxt-ai-checker",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/taroyamada~robotstxt-ai-checker/runs": {
      "post": {
        "operationId": "runs-sync-taroyamada-robotstxt-ai-checker",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/taroyamada~robotstxt-ai-checker/run-sync": {
      "post": {
        "operationId": "run-sync-taroyamada-robotstxt-ai-checker",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "domains"
        ],
        "properties": {
          "domains": {
            "title": "Domains to Check",
            "type": "array",
            "description": "Leave the 4 prefilled domains and click Start for the first run. Or enter public domains to check robots.txt policies for GPTBot, ClaudeBot, Google-Extended, and other AI crawlers.",
            "items": {
              "type": "string"
            }
          },
          "delivery": {
            "title": "Delivery Mode",
            "enum": [
              "dataset",
              "webhook"
            ],
            "type": "string",
            "description": "How to deliver results. 'dataset' saves to Apify Dataset, 'webhook' sends to a URL. In demoMode, delivery is always dataset.",
            "default": "dataset"
          },
          "webhookUrl": {
            "title": "Webhook URL",
            "type": "string",
            "description": "Webhook URL to send results to (only used when delivery is 'webhook'). Works with Slack, Discord, or any HTTP endpoint."
          },
          "snapshotKey": {
            "title": "Snapshot Key",
            "type": "string",
            "description": "Record key for change snapshots inside the fixed robotstxt-ai-checker-state Key-Value Store.",
            "default": "robotstxt-snapshots"
          },
          "datasetMode": {
            "title": "Dataset Mode",
            "enum": [
              "all",
              "changes_only"
            ],
            "type": "string",
            "description": "all emits every result. changes_only emits only changed results and emits zero rows and zero charges when nothing changed.",
            "default": "changes_only"
          },
          "concurrency": {
            "title": "Concurrency",
            "minimum": 1,
            "maximum": 10,
            "type": "integer",
            "description": "Maximum number of parallel requests. Higher = faster but may trigger rate limits.",
            "default": 5
          },
          "dryRun": {
            "title": "Dry Run",
            "type": "boolean",
            "description": "If true, runs without saving results or sending webhooks. Useful for testing.",
            "default": false
          },
          "demoMode": {
            "title": "Demo Mode",
            "type": "boolean",
            "description": "If true, checks only 1 domain, returns compact policy fields, and disables webhook/snapshot writes.",
            "default": false
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

robots.txt AI Crawler Checker: Rules. No API key or login OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for robots.txt Parser & AI Crawler Block Checker from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

robots.txt Parser & AI Crawler Block Checker API in Python

robots.txt Parser & AI Crawler Block Checker API in JavaScript

robots.txt Parser & AI Crawler Block Checker API through CLI

robots.txt Parser & AI Crawler Block Checker API

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

168K

4.6

(235)

Website Content Extractor — Docs & Pages to Markdown

taroyamada/website-content-extractor

Extract website pages to Markdown. No API key / no login. Open Input → leave defaults → Start (1 URL ≈ USD 0.01/run): https://console.apify.com/actors/2SEQ0bWRQ39RCeI0P/input From USD 9/1k.

naoki anzai

Wayback Machine Checker — Snapshot Diff Alerts

taroyamada/wayback-machine-checker

Compare Wayback snapshots and detect changes. No API key / no login. Open Input → leave defaults → Start (1 URL ≈ USD 0.008): https://console.apify.com/actors/fy1UnkQNIzGyy8HIs/input From USD 8/1k.

naoki anzai

DMARC & Email Security Checker

taroyamada/dns-dmarc-security-checker

Validate email-domain infrastructure. No API key / no login. Open Input → leave defaults → Start (1 domain ≈ USD 0.008 total): https://console.apify.com/actors/PUumFXvaFfJrkaMF3/input

naoki anzai

Bulk URL Health Checker — Status and Redirects

taroyamada/bulk-url-health-checker

Check HTTP status, redirects, and response time across URLs. No API key / no login. Open Input → leave defaults → Start: https://console.apify.com/actors/hHvfTzycQXbARYlZs/input

naoki anzai

Short URL Resolver & Scraper

taroyamada/url-shortener-resolver

Resolve short links. No API key / no login. Open Input → leave defaults → Start (1 short URL ≈ USD 0.008/row): https://console.apify.com/actors/NHTWuZjw0n2kAQuOW/input From USD 8/1k.

naoki anzai

Apple Podcasts Reviews Scraper — Ratings and Listener Feedback

taroyamada/apple-podcast-reviews-monitor

Apple Podcasts reviews from iTunes RSS. No API key / no login. Open Input → leave defaults → Start (1 show + 1 page): https://console.apify.com/actors/ulldRldNdzzThgg7J/input From $3 / 1,000 results.

naoki anzai

Sitemap Analyzer — Nested XML URL Audit

taroyamada/sitemap-analyzer

Analyze XML sitemaps. No API key / no login. Open Input → leave defaults → Start (1 sitemap + maxUrls 3 ≈ USD 0.005 total): https://console.apify.com/actors/wJILhGwhAXWPfqBA2/input

naoki anzai

Structured Data Scraper & Validator

taroyamada/structured-data-validator

Validate JSON-LD and Microdata on public pages. No API key / no login. Open Input → leave defaults → Start (1 URL ≈ USD 0.01 total): https://console.apify.com/actors/LupMh9Kdj6vgwlHMb/input

naoki anzai

HHS Healthcare Data Breach Change Scraper

taroyamada/data-breach-disclosure-monitor

Track HHS OCR breach disclosures. No API key / no login. Open Input → leave defaults → Start (evidence window ≈ USD 0.01/row): https://console.apify.com/actors/iKd7RyZ5vXJgbj6V5/input From USD 10/1k.

naoki anzai