Robots.txt and AI Crawler Access Checker API

Find out what a website's robots.txt says to AI and search crawlers. Give the actor a list of domains or addresses; for each one it reads robots.txt exactly as the robots exclusion standard (RFC 9309) defines, works out which of 27 named crawlers may fetch the paths you care about, and returns one row per website: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and the others as true/false columns, a verdict on AI training, AI search and classic search, the sitemaps the file declares, and whether the site publishes llms.txt. JSON, CSV or Excel, one polite request at a time.

Price and limits at a glance: $2.00 per 1,000 websites on the Free plan and $1.50 on the Business plan, charged only for websites that were read and delivered; up to 100,000 websites and 20 paths per run; about four requests and five seconds per website at the default speed; no personal data.

What you get

One row per website, 65 fields, with null where something does not apply:

• The file itself: the address of robots.txt that was read (after redirects), its HTTP status, what came of it (found, empty, missing, not_a_robots_file or server_error), its size in bytes and whether it was longer than the 500 KiB a crawler must read
• A column per crawler: gptBot, chatGptUser, oaiSearchBot, claudeBot, claudeUser, claudeSearchBot, perplexityBot, perplexityUser, googlebot, googleExtended, googleOther, applebot, applebotExtended, bingbot, ccBot, amazonbot, amznSearchBot, amznUser, metaExternalAgent, metaWebIndexer, duckAssistBot, duckDuckBot, mistralAiUser, mistralAiIndex, mistralAiTraining, yandexBot and baiduspider. true means the crawler may fetch every path you checked, false means at least one is closed to it
• Lists and counts: allowedBots, blockedBots, partlyBlockedBots (allowed on some of your paths, not on all), explicitlyListedBots (crawlers the file names in a group of their own, as opposed to those that fall under the * rules), allowedCount, blockedCount
• Three verdicts: aiTrainingPolicy (blocks all AI training bots, allows all or mixed), aiSearchPolicy (the same for AI search bots) and searchVisibility (all search crawlers allowed, some search crawlers blocked or all search crawlers blocked)
• Path by path: pathAccess lists, for every path you gave, which crawlers are allowed and which are blocked
• Your own crawlers: any user-agent name you add is checked with the same rules and shown in customBots
• What the site declares: sitemaps and sitemapCount, the crawlDelay of the * group, and robotsUserAgents, the crawler names the file mentions (so you also see crawlers that are not on the list)
• AI files: llmsTxtPresent, llmsTxtStatus, llmsTxtBytes and the same three for llms-full.txt
• The home page signals: the homeMetaRobots tag and the homeXRobotsTag header, read only when robots.txt lets our own crawler in
• Change tracking: policyHash, which changes only when the policy changes, notes with anything that could not be read and why, and the time of the check

The run also writes a COVERAGE record with one line per website you asked for and a RUN_SUMMARY with counts, how many websites block all AI training bots, and why the run stopped.

Why this actor

• The standard, not a guess. Rules are matched the way RFC 9309 says: lines for several user agents form one group, the longest matching rule wins, Allow wins a tie, * and a final $ work inside patterns, percent-encoded paths are compared in normal form, and only the first 500 KiB are read. A missing file means everything is allowed and a server error means everything is closed.
• Crawler names from the operators. A name is on the list only when the company that runs the crawler documents it. The source of every one is written in the Data source section, and names that could not be confirmed that way are left out rather than guessed.
• Your paths, not only the home page. A site can let GPTBot into /blog/ and keep it out of /members/. Give the paths and every crawler is evaluated on each of them.
• Your own crawler. Add the user-agent name of a bot you run, or one you saw in a server log, and get the same answer for it.
• The related files in one row. robots.txt, llms.txt, llms-full.txt, sitemaps, the meta robots tag and the X-Robots-Tag header, so an AI-readiness check is one run.
• Documented fallbacks. When robots.txt has no group for Applebot, Bingbot or YandexBot, the actor applies the group those operators say the crawler follows instead (Googlebot, msnbot and Yandex), and says so in the data through explicitlyListedBots.
• Pay only for websites read. Websites that block the actor, do not exist, are invalid or repeated, or whose policy did not change since your last run are never charged.
• Polite by design. robots.txt is read first on every host, our own crawler obeys it, the request rate is one per second by default, and a Crawl-delay the site sets for us is kept.

How to use it

1. Open the Actor and paste your domains or addresses into Websites, one per line. A column copied from a spreadsheet works, and so do entries separated by commas.
2. For a first try keep the list as it is: it holds three large websites with different policies.
3. To check a section of a site, add paths such as /blog/ or /products/shoes?page=2 under Paths to check. To check only some crawlers, or one of your own, list their names under Crawlers to check.
4. Click Start. The status line shows progress, and the run stops cleanly at your maximum or at the charge limit you set in the run options.
5. Open the Dataset tab: the Overview, AI crawlers, Search and archive crawlers, Allowed and blocked lists and robots.txt, llms.txt and meta tags views show the main columns. Download JSON, CSV, Excel, XML or HTML, or send the rows to Google Sheets, Slack, Make, n8n or Zapier with the Integrations tab.
6. Open the COVERAGE record in the key-value store to see which websites could not be read and why.
7. To watch a list over time, run it on a schedule with Only new rows since the last run turned on: a website is delivered, and charged, again only when its policy changed.

Input

Field	What it does
websites	Domains (example.com) or addresses (https://www.example.com/shop), one per line. robots.txt is read from the top of the site; the path of an address is ignored, use paths for that. Up to 100,000 per run.
paths	The paths the access is calculated for, default /. Up to 20, each with an optional query string. A crawler counts as allowed only when every path is allowed.
bots	Empty = all 27 listed crawlers. Or list names, for example GPTBot and ClaudeBot. A name that is not on the list is checked as your own crawler; the word all adds the whole list to your own names.
maxItems	Stop after this many delivered rows. 0 = the whole list.
onlyNew, stateName	Deliver a website only when its policy changed since the last run with the same state name
notifyUrl, notifyWhenEmpty	A webhook that receives one message when the run ends; see Automation below
requestIntervalMs	Pause between requests, default 1000 ms, minimum 250 ms

Example: three websites, all listed crawlers, the home page.

{
  "websites": ["bbc.com", "reuters.com", "etsy.com"]
}

Example: two paths on two stores, and a crawler of your own.

{
  "websites": ["etsy.com", "walmart.com"],
  "paths": ["/", "/search?q=shoes"],
  "bots": ["GPTBot", "ClaudeBot", "Googlebot", "MyCompanyBot"]
}

Example: a weekly watch on competitors that reports only the websites whose policy changed, with a Slack message.

{
  "websites": ["competitor-one.com", "competitor-two.com", "competitor-three.com"],
  "onlyNew": true,
  "stateName": "competitors",
  "notifyUrl": "https://hooks.slack.com/services/..."
}

Output

65 fields per row. Empty values are null, never missing, so CSV and Excel columns are stable.

Field	Type	Example
id	string	example.com
domain	string	example.com
inputUrl	string	https://www.example.com/shop
robotsUrl	string	https://www.example.com/robots.txt
robotsStatus	integer	200
robotsOutcome	string	found
robotsBytes	integer	3412
robotsTruncated	boolean	False
checkedPaths	string	/
botsChecked	integer	27
allowedCount	integer	12
blockedCount	integer	15
allowedBots	string	Googlebot, Bingbot
blockedBots	string	GPTBot, ClaudeBot
partlyBlockedBots	string	Googlebot, Bingbot
explicitlyListedBots	string	GPTBot, CCBot
aiTrainingPolicy	string	blocks all AI training bots
aiSearchPolicy	string	allows all
searchVisibility	string	all search crawlers allowed
pathAccess	array	[{"path": "/", "allowedBots": "Googlebot, Bingbot", "blockedBots": …
gptBot	boolean	True
chatGptUser	boolean	True
oaiSearchBot	boolean	True
claudeBot	boolean	True
claudeUser	boolean	True
claudeSearchBot	boolean	True
perplexityBot	boolean	True
perplexityUser	boolean	True
googlebot	boolean	True
googleExtended	boolean	True
googleOther	boolean	True
applebot	boolean	True
applebotExtended	boolean	True
bingbot	boolean	True
ccBot	boolean	True
amazonbot	boolean	True
amznSearchBot	boolean	True
amznUser	boolean	True
metaExternalAgent	boolean	True
metaWebIndexer	boolean	True
duckAssistBot	boolean	True
duckDuckBot	boolean	True
mistralAiUser	boolean	True
mistralAiIndex	boolean	True
mistralAiTraining	boolean	True
yandexBot	boolean	True
baiduspider	boolean	True
customBots	object	{"MyCrawler": true}
sitemaps	string	https://www.example.com/sitemap.xml
sitemapCount	integer	1
crawlDelay	number	10
robotsUserAgents	string	gptbot, ccbot
llmsTxtPresent	boolean	False
llmsTxtStatus	integer	404
llmsTxtBytes	integer	15010
llmsFullTxtPresent	boolean	False
llmsFullTxtStatus	integer	404
llmsFullTxtBytes	integer	482113
homeUrl	string	https://www.example.com/
homeStatusCode	integer	200
homeMetaRobots	string	max-image-preview:large
homeXRobotsTag	string	noindex, noarchive
notes	string	home page not read: HTTP 403: the site answered with a Cloudflare a…
policyHash	string	3f2a9c0d11b7e845
checkedAt	string	2026-10-16T07:03:12.000Z

Reading the three verdicts

aiTrainingPolicy covers the crawlers whose operators say they collect content for model training or offer a robots.txt opt-out from it: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, Meta-ExternalAgent and MistralAI-Training. aiSearchPolicy covers the crawlers that build an AI search or answer product: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, Meta-WebIndexer, DuckAssistBot and MistralAI-Index. searchVisibility covers classic search: Googlebot, Applebot, Bingbot, DuckDuckBot, YandexBot and Baiduspider. Each verdict looks only at the crawlers of its kind that are in your list, and is null when none of them is. The remaining crawlers have a column of their own but no verdict: ChatGPT-User, Claude-User, Perplexity-User, Amzn-User and MistralAI-User, which fetch a page when a person asks for it, CCBot, the open Common Crawl archive, and GoogleOther, a general Google crawler.

Example rows

Two real rows from a run (long texts shortened here, complete in the dataset):

[
  {
    "id": "bbc.com",
    "domain": "bbc.com",
    "inputUrl": "bbc.com",
    "robotsUrl": "https://www.bbc.com/robots.txt",
    "robotsStatus": 200,
    "robotsOutcome": "found",
    "robotsBytes": 6169,
    "robotsTruncated": false,
    "checkedPaths": "/",
    "botsChecked": 27,
    "allowedCount": 14,
    "blockedCount": 13,
    "allowedBots": "Claude-User, Claude-SearchBot, Googlebot, GoogleOther, Applebot, Bingbot, Meta-WebIndexer, DuckAssistBot, DuckDuckBot, MistralAI-User, Mi…",
    "blockedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Amz…",
    "partlyBlockedBots": null,
    "explicitlyListedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Amz…",
    "aiTrainingPolicy": "mixed",
    "aiSearchPolicy": "mixed",
    "searchVisibility": "all search crawlers allowed",
    "pathAccess": [
      {
        "path": "/",
        "allowedBots": "Claude-User, Claude-SearchBot, Googlebot, GoogleOther, Applebot, Bingbot, Meta-WebIndexer, DuckAssistBot, DuckDuckBot, MistralAI-User, Mi…",
        "blockedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Amz…"
      }
    ],
    "gptBot": false,
    "chatGptUser": false,
    "oaiSearchBot": false,
    "claudeBot": false,
    "claudeUser": true,
    "claudeSearchBot": true,
    "perplexityBot": false,
    "perplexityUser": false,
    "googlebot": true,
    "googleExtended": false,
    "googleOther": true,
    "applebot": true,
    "applebotExtended": false,
    "bingbot": true,
    "ccBot": false,
    "amazonbot": false,
    "amznSearchBot": false,
    "amznUser": false,
    "metaExternalAgent": false,
    "metaWebIndexer": true,
    "duckAssistBot": true,
    "duckDuckBot": true,
    "mistralAiUser": true,
    "mistralAiIndex": true,
    "mistralAiTraining": true,
    "yandexBot": true,
    "baiduspider": true,
    "customBots": null,
    "sitemaps": "https://www.bbc.com/afrique/sitemap.xml, https://www.bbc.com/arabic/sitemap.xml, https://www.bbc.com/bbcx/audio_archive_sitemap.xml, http…",
    "sitemapCount": 38,
    "crawlDelay": null,
    "robotsUserAgents": "amazonbot, magpie-crawler, ccbot, omgili, omgilibot, claude-web, claudebot, anthropic-ai, cohere-ai, bytespider, petalbot, scrapy, appleb…",
    "llmsTxtPresent": false,
    "llmsTxtStatus": 404,
    "llmsTxtBytes": null,
    "llmsFullTxtPresent": false,
    "llmsFullTxtStatus": 404,
    "llmsFullTxtBytes": null,
    "homeUrl": "https://www.bbc.com/",
    "homeStatusCode": 200,
    "homeMetaRobots": "NOODP, NOYDIR",
    "homeXRobotsTag": "bingbot: noarchive",
    "notes": null,
    "policyHash": "3df2ac7ae1fffeb5",
    "checkedAt": "2026-10-15T20:37:09.965Z"
  },
  {
    "id": "nodejs.org",
    "domain": "nodejs.org",
    "inputUrl": "nodejs.org",
    "robotsUrl": "https://nodejs.org/robots.txt",
    "robotsStatus": 200,
    "robotsOutcome": "found",
    "robotsBytes": 199,
    "robotsTruncated": false,
    "checkedPaths": "/",
    "botsChecked": 27,
    "allowedCount": 27,
    "blockedCount": 0,
    "allowedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended…",
    "blockedBots": null,
    "partlyBlockedBots": null,
    "explicitlyListedBots": null,
    "aiTrainingPolicy": "allows all",
    "aiSearchPolicy": "allows all",
    "searchVisibility": "all search crawlers allowed",
    "pathAccess": [
      {
        "path": "/",
        "allowedBots": "GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended…",
        "blockedBots": null
      }
    ],
    "gptBot": true,
    "chatGptUser": true,
    "oaiSearchBot": true,
    "claudeBot": true,
    "claudeUser": true,
    "claudeSearchBot": true,
    "perplexityBot": true,
    "perplexityUser": true,
    "googlebot": true,
    "googleExtended": true,
    "googleOther": true,
    "applebot": true,
    "applebotExtended": true,
    "bingbot": true,
    "ccBot": true,
    "amazonbot": true,
    "amznSearchBot": true,
    "amznUser": true,
    "metaExternalAgent": true,
    "metaWebIndexer": true,
    "duckAssistBot": true,
    "duckDuckBot": true,
    "mistralAiUser": true,
    "mistralAiIndex": true,
    "mistralAiTraining": true,
    "yandexBot": true,
    "baiduspider": true,
    "customBots": null,
    "sitemaps": "https://nodejs.org/sitemap.xml, https://nodejs.org/learn/sitemap.xml",
    "sitemapCount": 2,
    "crawlDelay": null,
    "robotsUserAgents": null,
    "llmsTxtPresent": true,
    "llmsTxtStatus": 200,
    "llmsTxtBytes": 15010,
    "llmsFullTxtPresent": false,
    "llmsFullTxtStatus": 404,
    "llmsFullTxtBytes": null,
    "homeUrl": "https://nodejs.org/en",
    "homeStatusCode": 200,
    "homeMetaRobots": "index, follow",
    "homeXRobotsTag": null,
    "notes": null,
    "policyHash": "81d3b3106cbb6572",
    "checkedAt": "2026-10-15T20:47:04.363Z"
  }
]

Results can be downloaded as JSON, CSV, Excel, XML or HTML from the Dataset tab, or read through the Apify API.

Pricing

Pay per result. You pay for each website read and delivered to your dataset, nothing else: no subscription, no minimum, no charge for websites that could not be read (anti-bot page, a server that refuses robots.txt, not found), for invalid or repeated entries, or for websites skipped by Only new rows because their policy did not change.

Your Apify plan	Price per 1,000 rows	Price per row
Free plan	$2.00	$0.002
Starter plan	$1.80	$0.0018
Scale plan	$1.70	$0.0017
Business plan and above	$1.50	$0.0015

Example: 1,000 websites cost $2.00 on the Free plan and $1.50 on the Business plan, however many crawlers and paths each row covers. The Apify platform usage of a run (compute and storage) is billed by Apify separately, as for every Actor; the actor runs in 256 MB and spends most of its time waiting between requests. A start event of $0.00005 is charged once per run (Apify default). Cap your spend: set Maximum total charge in the run options. The actor stops cleanly when the cap is reached and you keep everything delivered until then. The run summary (RUN_SUMMARY in the key-value store) says why a run stopped.

Automation: only new rows, notifications, schedules

Only new rows since the last run. Turn on onlyNew and the actor remembers, for every website it delivered to you, a fingerprint of its policy (in a key-value store named rowcraft-robots-ai-crawler-checker-state in your Apify account). The fingerprint covers the answer for every crawler on every path, the declared sitemaps, the Crawl-delay, the presence of llms.txt and llms-full.txt, and the meta robots tag and X-Robots-Tag header of the home page. It does not cover the size of the file, comments or the order of the lines. The next run with the same stateName delivers a website again only when something in that list changed; websites with the same policy are skipped and not charged. The first run delivers every website it can read. Use a different stateName for each list so they keep separate memories.

Notification when the run ends. Put a webhook address in notifyUrl and the actor sends one JSON POST with the number of delivered rows, links to the dataset (JSON and CSV) and the run. It works with Slack and Discord incoming webhooks (the message is in text and content), with Make, n8n and Zapier catch hooks, and with your own endpoint. By default it is sent only when at least one row was delivered; notifyWhenEmpty sends it every time.

Schedules. Create a schedule in Apify Console (daily, weekly, monthly or cron) with onlyNew on, and you get a feed of the websites in your list that changed their crawler policy, for the price of those rows only.

Integrations. Send the results to Make, n8n, Zapier, Google Sheets, Slack, Google Drive, Airtable or any webhook from the Integrations tab of the Actor, with no code.

API, SDKs and AI agents. Every run is available through the Apify API and the official JavaScript and Python clients; the Actor can also be called by AI agents through the Apify MCP server. The example input in the API tab is a working call.

Use cases

• AI visibility audit for your own sites: check that the search crawlers you want, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, are let in, and see in one row whether llms.txt exists and a sitemap is declared.
• Publisher and brand policy: list your websites or those of a client, see which AI training bots each one blocks, and find the sites that block none by mistake, for example after a CMS or CDN change.
• Agency reporting: a table per client with the crawlers allowed and blocked on the sections that matter, such as /blog/ or /products/.
• Competitor and market research: which publishers, retailers or software companies block AI training bots, and who lets AI search crawlers in. The aiTrainingPolicy column answers it for thousands of domains.
• Change alerts: a weekly schedule with Only new rows tells you when a website you follow starts blocking GPTBot or ClaudeBot, adds a sitemap or publishes an llms.txt.
• Before you crawl: check what a site allows for the crawler you operate, with its own name, before you send it there.

Ready-made examples, each with the input filled in (open one and press Start, or save it as your own task):

• Which news sites block AI crawlers like GPTBot and ClaudeBot?
• AI crawler access of big online stores: who is blocked

What this actor does not do

• Enforce or test anything: robots.txt is a request to crawlers, not a lock. The actor reports what the file says, not whether a crawler obeys it.
• Tell you what a crawler did: there is no access-log analysis, no crawl statistics and no history; each row is the policy seen at the time of the run.
• Collect contacts: no e-mail addresses, phone numbers, names of people or social profiles, and no page text.
• Crawl a website: only robots.txt, llms.txt, llms-full.txt and the home page are read, never other pages, the sitemap files, the shop or the checkout.
• Judge the content of llms.txt: it reports whether a file exists and how large it is, not whether it follows the proposed format.
• Read rules written in meta tags or headers for individual crawlers: the home page tags are reported as the site sends them, and your own reading decides what they mean for a given crawler.
• Run JavaScript, log in, accept cookie banners, use proxies or solve CAPTCHAs: websites that answer with an anti-bot page are reported as blocked and not charged.

Limits and fair use

• One request per second by default (minimum 250 ms). A website takes four requests: robots.txt, the home page, llms.txt and llms-full.txt, more when redirects are involved. In our test runs that came to about five seconds per website, roughly 700 websites an hour; slow servers take longer, and a request that gets no answer is given up after 15 seconds (robots.txt) or 25 seconds (other pages).
• Up to 100,000 websites and 20 paths per run. The run stops 90 seconds before its timeout with the summary, the coverage and the Only new rows memory saved; for long lists raise the timeout or split the list.
• robots.txt is obeyed for every address the actor visits, including redirect targets. If it closes the home page, llms.txt or llms-full.txt to our crawler (Rowcraft), that file is not read and the row says so in notes. The policy for the other crawlers is still reported, because it needs only robots.txt.
• Crawl-delay is not part of RFC 9309. The row reports the value of the * group, and the actor itself keeps the Crawl-delay that applies to its own crawler (up to 15 seconds) between its requests to that site.
• Anti-bot pages. Websites that answer robots.txt with a bot challenge (Cloudflare, Akamai, DataDome and similar) or refuse the request with an error such as 403, 418 or 429 are reported as blocked and not charged, because the file was not read. After 10 blocked websites in a row the run stops and says why in the run summary, because the rest of the list would most likely be blocked too. A home page that is blocked while robots.txt was read still gives a row, with the reason in notes.
• File sizes: the first 500 KiB of robots.txt, the first 1 MB of the home page and the first 256 KiB of each llms file are read.
• Lists in a row are cut at 25 sitemaps and 25 crawler names; sitemapCount always holds the full number.

Data source and licence

The data is what each website sends to every visitor: its robots.txt file, its llms.txt and llms-full.txt files if it has them, and the response headers and meta tags of its home page. The actor reads each of these once, after checking robots.txt, and delivers the conclusions (allowed or blocked per crawler), the declared sitemap addresses, short header and tag values and file sizes. The text of robots.txt, the files and the page is not stored or delivered.

How robots.txt is read: the Robots Exclusion Protocol, RFC 9309 (https://www.rfc-editor.org/rfc/rfc9309). The llms.txt file is a proposal described at https://llmstxt.org/ and is not part of that standard.

Where each crawler name comes from. A name is on the list only when the page of the operator named below documents it. The pages were read on 15 October 2026. The operators change their lists, so a crawler that appears in a later version of a page is not covered until the actor is updated.

Operator	Crawler names	Source
OpenAI	GPTBot, ChatGPT-User, OAI-SearchBot	https://platform.openai.com/docs/bots
Anthropic	ClaudeBot, Claude-User, Claude-SearchBot	https://support.anthropic.com/en/articles/8896518
Perplexity	PerplexityBot, Perplexity-User	https://docs.perplexity.ai/guides/bots
Google	Googlebot, Google-Extended, GoogleOther	https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Apple	Applebot, Applebot-Extended	Apple support article 119829 ("About Applebot"), on support.apple.com/en-us/119829
Microsoft	Bingbot, with the msnbot group as its documented fallback	Bing Webmaster Blog, the posts on the bingbot crawler and on crawl directives, on blogs.bing.com/webmaster
Common Crawl	CCBot	The CCBot page on commoncrawl.org/ccbot
Amazon	Amazonbot, Amzn-SearchBot, Amzn-User	The Amazonbot page on developer.amazon.com/amazonbot
Meta	Meta-ExternalAgent, Meta-WebIndexer	Meta's web crawlers page for webmasters, on developers.facebook.com/docs/sharing/webmasters/web-crawlers
DuckDuckGo	DuckAssistBot, DuckDuckBot	The DuckAssistBot and DuckDuckBot help pages, on duckduckgo.com/duckduckgo-help-pages/results
Mistral AI	MistralAI-User, MistralAI-Index, MistralAI-Training	The robots page of the Mistral documentation, on docs.mistral.ai/robots
Yandex	YandexBot, with the Yandex group as its documented fallback	The Yandex Webmaster help on the User-agent directive, on yandex.com/support/webmaster
Baidu	Baiduspider	The English robots.txt page on baidu.com/search/robots_english.html

What the operators say about robots.txt, as far as it matters here: OpenAI notes that rules may not apply to ChatGPT-User, Perplexity says that Perplexity-User generally ignores robots.txt, and Amazon says that Amzn-User may not follow all directives, because these three fetch a page at the request of a person. The columns for them show what robots.txt says, not what the crawler will do. Google describes Google-Extended as a robots.txt control token with no effect on Google Search, and Apple describes Applebot-Extended as an opt-out from training its models; neither is a separate crawler that visits pages.

Rowcraft is independent and not affiliated with or endorsed by OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Common Crawl, Amazon, Meta, DuckDuckGo, Mistral AI, Yandex or Baidu. Product names belong to their owners and are used only to say which crawler a rule applies to.

Privacy: no personal data

Rowcraft actors are built not to collect personal data:

• Only robots.txt, llms.txt, llms-full.txt and the home page of a website are read, and the row holds crawler names, true/false answers, sitemap addresses, file sizes and two short robots tag values.
• A web address that would contain an e-mail address or a long token is reduced before it is delivered.
• The actor sends no cookies, fills no forms and never logs in.
• No name, e-mail address, phone number or profile is ever requested or stored.

You are responsible for using the data in line with the law that applies to you.

Other Rowcraft actors

All Rowcraft actors read official or public sources, deliver flat rows without personal data and support only-new runs and notifications:

• Ashby Jobs Scraper – Job Postings with Salary Ranges — Job postings from Ashby job boards (jobs.ashbyhq.com) by company: title, department, team, locations with country, employment and workplace type, published salary range, full text.
• Broken Link Checker – Find 404s and Dead Links API — Pay per crawled page, not per link: crawl a site within robots.txt and check every link on each page for 404s, 5xx errors, DNS and TLS failures and redirects.
• Career Site Jobs Scraper: Greenhouse, Lever, Ashby, Workday — Job listings from the career pages of the companies you choose on Greenhouse, Lever, Ashby, Workday, Workable, Personio, Recruitee, Teamtailor and Gem, in one run and one format: title, locations, department, salary when published, full text.
• Chrome Web Store Scraper: Users, Ratings & Versions — Reads public Chrome Web Store listing pages with no login, no proxy and no personal data: users, rating, category, version, update date, permissions and company developers.
• Domain Email and DNS Checker API: MX, SPF, DMARC, Providers — Find which email provider, sending services and DNS host each domain uses, from public DNS records only.
• SIRENE France Company Scraper: SIREN, NAF, Revenue, Status — French companies from the official Recherche d'entreprises API (INSEE SIRENE + INPI): SIREN and SIRET, legal form, NAF activity, head-office address, employee band, revenue and net income, labels, creation date and VAT number.
• Gem Jobs Scraper – Job Board Listings from jobs.gem.com — Job listings from Gem job boards (jobs.gem.com) by company: title, department, offices with country, employment type, remote, hybrid or on-site, requisition ID, publication date and full ad text, from the public Gem Job Board API.
• Google Play Apps Scraper: App Details, Installs, Ratings — Public Google Play app details as flat rows: installs, rating, number of ratings, price, version, last update, minimum Android, ads and in-app purchases.
• Greenhouse Jobs Scraper – Job Listings by Company Board — Job listings from Greenhouse job boards by company: title, departments, offices, publication date, full ad text, published pay ranges.
• Himalayas Jobs Scraper: Remote Jobs API with Salaries — Remote jobs from Himalayas through its public jobs API: title, company, salary range, seniority, categories, the countries and time zones open to candidates, full description and job link.
• KRS Poland Companies Scraper: Register Data, PKD, Status — Polish companies from the official KRS register API of the Ministry of Justice: name, KRS, NIP and REGON numbers, legal form, status, address, PKD activity codes, share capital, branches and registration dates.
• LEI Records Scraper: Legal Entity Identifier Lookup API — Legal Entity Identifier (LEI) records from the Global LEI Index, read through the public GLEIF API: legal name, legal form, register number, addresses, status, renewal date and parent companies.
• Lever Jobs Scraper – Job Postings from jobs.lever.co — Job postings from Lever career pages (jobs.lever.co) by company: title, team, department, location, commitment, workplace type, salary range when published, description, requirements and benefits.
• Norway Company Register Scraper: Brønnøysund Business Data — Norwegian companies from the Central Coordinating Register for Legal Entities (Brønnøysund, NLOD open data): organisation number, legal form, industry, employees, share capital, addresses, website, VAT and bankruptcy status.
• Personio Jobs Scraper: Job Listings, Salaries, Departments — Job ads from any company's Personio careers feed: title, department, office, employment type, schedule, seniority, salary when published, keywords and the full description in the language you choose.
• Platsbanken Scraper: Sweden Jobs, Employers, Org Numbers — Every job ad on Platsbanken (Arbetsförmedlingen) from the official JobTech API: headline, employer and organisation number, occupation codes, municipality, coordinates, salary type, deadline and full text.
• Recruitee Jobs Scraper – Job Listings by Company Career Site — Job listings from Recruitee career sites by company: title, department, locations with country, employment type, remote flag, published salary and full text, one request per company.
• RSS Feed Finder and Reader – Blog and News Article API — Find the RSS, Atom and JSON feeds of any website and read their newest articles, from a domain name alone.
• SEO Audit Tool: Titles, Meta Tags, Headings and Links API — Audit the on-page SEO of any list of pages or websites: titles, meta descriptions, headings, canonicals, indexing, links, images and structured data, with a score and the issues found per page.
• Shopify Products Scraper: Store Catalog Export to JSON, CSV — Products from any Shopify store's public catalog: title, vendor, type, tags, description, variants with price, compare-at price, SKU and stock, images, options, store currency.
• Sitemap URL Extractor: Every Page with Last-Modified Dates — Get every page a site lists in its sitemaps, with last-modified dates, from robots.txt through sitemap indexes and .gz files.
• SSL Certificate and Security Headers Checker API — Check the TLS certificate and the HTTP security headers of every domain in one run, with an A+ to F grade and the exact issues to fix.
• Store Products Scraper – Shopify, WooCommerce, VTEX — Export the product catalogue of any Shopify, WooCommerce or VTEX store from its public catalogue endpoint: title, brand, prices, discounts, stock, categories, images and links, one row per product, the same columns on every platform.
• Structured Data Extractor: JSON-LD, Schema.org, Open Graph — Pull the schema.org JSON-LD, Microdata, RDFa and Open Graph of any page list into flat rows: types, product price, availability, ratings, organization, breadcrumbs, with validation issues.
• Teamtailor Jobs Scraper – Career Site Jobs by Company — Job listings from Teamtailor career sites by company: title, department, role, locations with city and country, remote or hybrid status, publication date and full ad text, from the official job feed.
• TED Tenders Scraper: EU Public Procurement Notices API — EU public procurement notices from TED (Tenders Electronic Daily), the official EU tenders portal, through its public Search API: title, buyer, country, CPV codes, values, deadlines, lots and document links.
• UK Companies House Scraper: Company Search & Profiles API — UK companies from the official Companies House API with your free key: search by name, status, type, SIC code, date or location, or look up company numbers.
• UK Tenders Scraper: Find a Tender + Contracts Finder API — UK public procurement notices from Find a Tender and Contracts Finder, the official UK government portals, through their open OCDS APIs: title, buyer, region, CPV codes, values, deadlines, notice type and links.
• Bulk URL Status and Redirect Chain Checker API — Check the status code, redirect chain, canonical and indexing signals of every URL in a list, with the final destination and a loop flag.
• VTEX Products Scraper – Public Catalog Export — Export the catalogue of any VTEX store (Auchan, Chedraui and thousands of retailers in Europe and Latin America) from its public search endpoint: title, brand, EAN, prices, list prices, stock, categories, specifications and images, one row per product.
• Website Tech Stack Detector – Technology Lookup API — Find the technologies behind a list of websites: CMS, e-commerce platform, analytics, tag managers, ad pixels, consent tools, chat, CDN and payments, from one polite home-page request per domain.
• WooCommerce Products Scraper – Store API Export — Export every product of a WooCommerce store through its public Store API: title, SKU, prices and sale prices, stock, categories, tags, attributes, ratings, images and links, one row per product.
• Workable Jobs Scraper: Job Listings from Career Pages — Job listings from any company's Workable career page: title, department, location, remote or on-site, employment type, description, requirements, benefits and salary when published.
• Workday Jobs Scraper – Jobs from myworkdayjobs.com Sites — Job listings from Workday career sites (myworkdayjobs.com) by company: title, locations, country, posting date, time type, requisition ID and the full description.

FAQ

Which crawlers does it check? 27, from 13 operators: OpenAI (GPTBot, ChatGPT-User, OAI-SearchBot), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot), Perplexity (PerplexityBot, Perplexity-User), Google (Googlebot, Google-Extended, GoogleOther), Apple (Applebot, Applebot-Extended), Microsoft (Bingbot), Common Crawl (CCBot), Amazon (Amazonbot, Amzn-SearchBot, Amzn-User), Meta (Meta-ExternalAgent, Meta-WebIndexer), DuckDuckGo (DuckAssistBot, DuckDuckBot), Mistral AI (MistralAI-User, MistralAI-Index, MistralAI-Training), Yandex (YandexBot) and Baidu (Baiduspider).

Why are Bytespider, cohere-ai, anthropic-ai and AI2Bot not on the list? On 15 October 2026 the operators' own documentation did not name them as robots.txt user agents in a way we could confirm, and the list holds only names we could confirm. You can still check any of them: type the name under Crawlers to check and it is evaluated like the others, and robotsUserAgents shows every name a site's robots.txt mentions, so a site that blocks Bytespider is visible either way.

How do I check a crawler of my own? Add its user-agent product token, for example MyCompanyBot, to Crawlers to check. The token is the part before any slash in the user-agent string: letters, digits, underscores and hyphens. The answer appears in customBots, in allowedBots or blockedBots, and in the lists of pathAccess.

What does "allowed" mean when I give several paths? A crawler is allowed only when every path is allowed. If it can fetch / but not /search, it is shown as blocked in blockedBots and also in partlyBlockedBots, and pathAccess shows the paths one by one.

A crawler is not named in robots.txt. Why is it allowed or blocked? Under RFC 9309 a crawler obeys the group that names it; when no group names it, it obeys the * group; when there is none of those either, everything is allowed. explicitlyListedBots shows which crawlers the file names in a group of their own.

Does blocking a crawler in robots.txt really stop it? Robots.txt is a convention, and the crawlers of the large operators say they follow it, with the exceptions of the user-requested fetchers described in the Data source section. The actor reports what the file says. It cannot tell you whether a crawler that is not run by a well-known operator obeys it.

Is it legal? The actor reads a few public files and one page per website, the same ones any visitor or crawler gets, obeys robots.txt and delivers the policy that a website itself publishes, not personal data or page content. Whether and how you may use the results depends on the law that applies to you.

Does it collect personal data? No personal data: no e-mail addresses, phone numbers or people's names are read or delivered. See the Privacy section.

Why is the result empty? Every website in the list could not be read or was skipped. The COVERAGE record gives the reason for each: an anti-bot page or a refusal on robots.txt (blocked), no website at that address (not_found), an invalid entry or a connection problem (error). With Only new rows on, websites whose policy did not change are skipped too. Nothing is charged for those, apart from the start event.

Why is a website with a robots.txt answer such as 403 or 418 not reported as "everything allowed"? RFC 9309 says a crawler may treat any 4xx answer as "no rules". The actor treats only 404 and 410 that way. A 403, 418 or 429 means the site refused our request, so we do not know its rules, and saying "everything allowed" would be false. Those websites go to COVERAGE as blocked and are not charged. A server error (5xx) is different: the standard says to treat the site as closed, and the row shows every crawler as blocked with an explanation in notes.

The home page shows no meta robots tag, but the row has null. Is something wrong? No. homeMetaRobots and homeXRobotsTag are empty when the page has no such tag or header, and also when the home page was not read; notes tells which of the two it is.

Can I see the robots.txt lines behind a verdict? The row holds the verdict for each crawler, not the lines of the file. The file is one click away: robotsUrl is the exact address that was read, and explicitlyListedBots tells you which crawlers have a group of their own in it.

Do I pay for websites that block the actor? No. Blocked, not found and invalid entries are never charged, and neither are duplicates in your list.

How does Only new rows work here? The actor remembers the policy fingerprint of each website delivered to you. The next run with the same state name delivers a website only when its policy changed, so a weekly schedule costs only the websites that changed.

Support and reviews

Problems, missing fields, questions: open an issue on the Issues tab of this Actor. We answer within one working day and fix confirmed bugs in the next release; the changelog lists every change.

If this Actor saved you time, a short review on its page helps other users find it and helps us keep improving it.