Robots.txt and AI Crawler Access Checker API
Pricing
from $1.50 / 1,000 website checkeds
Check which AI and search crawlers a website lets in: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, read from robots.txt by the RFC 9309 rules. Also llms.txt, sitemaps and meta robots. Pay only per website checked.
Websites to check, one per line: a domain (example.com) or an address (https://www.example.com/shop). robots.txt is read from the top of each site; the path in an address is ignored (use "Paths" below). Duplicates and invalid entries are skipped and never charged.
The paths the access is calculated for, one per line, for example / or /blog/ or /products/shoes?page=2. A crawler counts as allowed only when every path is allowed. The default is the home page.
[ "/"]Leave empty to check all 27 listed crawlers (OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Amazon, Meta, Mistral, DuckDuckGo, Common Crawl, Yandex, Baidu). Or list the names you want, for example GPTBot and ClaudeBot. Any other name is checked as your own user-agent token, and the word "all" adds the whole list to your own tokens.
[]Stop after this many rows are delivered. 0 = the whole list.
Remember the rows already delivered to you (in a key-value store in your Apify account) and deliver only rows that appeared since the last run with the same state key. The first run delivers everything that matches; later runs cost only the new rows. Ideal for schedules.
Name of the memory used by "Only new rows". Use a different name for each schedule or search (for example stockholm-python) so they do not share one memory.
A webhook address that receives one JSON POST with the number of new rows and links to the dataset: a Slack or Discord incoming webhook, a Make, n8n or Zapier catch hook, or your own endpoint. The message is also in the text (Slack) and content (Discord) fields.
By default the notification is sent only when at least one row was delivered.