Robots.txt Checker: test URLs against robots.txt for a chosen crawler and see the rule that decided

Provide a list of URLs and the product token of a crawler (Googlebot, bingbot, a token of your own, or * for the rules every robot gets) and the Actor returns one record per URL: may this URL be crawled, which group and which rule decided, and what sitemaps and crawl-delay the host declares. Evaluation follows RFC 9309: a group for the exact token wins over the * group, the longest matching pattern wins, and a tie resolves to allow. robots.txt is fetched once per host (scheme, host and port) per run with up to 5 redirects; the pages themselves are never requested, so no browser, no proxy and no page content are involved.

What it is good for

• SEO checks after a migration or a redesign: confirm that the new sections are not disallowed by accident and that the pages you want indexed are open to Googlebot and bingbot.
• Crawler operators: audit your own crawler's compliance before you run it. Pass your product token as userAgent and see, per URL, the rule your crawler must obey and the Crawl-delay the host asks for.
• Link directories and content teams: check hundreds of listed URLs in one run and find the ones a search engine is not allowed to fetch.
• Feeding other tools: one flat JSON record per URL with the decision, the matched rule and the host's sitemaps, easy to join with your own spreadsheets or to schedule as a recurring run.

How to use it

1. Open the Actor and paste your list of URLs into the urls field, one per line (or pass a JSON array through the API).
2. Enter the robot's product token in userAgent (Googlebot, bingbot, MyCrawler) or leave * to check the rules that apply to every robot. The other options can stay at their defaults.
3. Click Start. When the run finishes, the status message reads Processed N of M URLs (X with errors, Y not charged).
4. Download the results from the Dataset tab as JSON, CSV or Excel, or read them through the API. Switch the dataset view to "All fields" to see every column.

Input

Field	Type	Default	Description
urls	list of strings	small self-test list	Absolute, publicly reachable http(s) URLs, one per line; exact duplicates skipped; max 1,000 per run. Private, loopback and localhost addresses, embedded credentials and blocked ports are reported as INVALID_URL and not charged
userAgent	string	*	Product token of the robot to check, as it appears in User-agent: lines of robots.txt. Matched as a whole token, case-insensitively. * checks the default group. Allowed: * or 1 to 100 characters from A-Z, a-z, 0-9, _ and -; any other value is rejected by the input validation (API calls included), and a value that still reaches the Actor unusable falls back to *
timeoutSecs	integer 1 to 60	15	Maximum time for fetching robots.txt of one URL, redirects included; on timeout the record gets error = TIMEOUT
maxConcurrency	integer 1 to 50	10	URLs processed in parallel; lower it if many URLs point to one host

Output

One dataset record per input URL. Example for https://www.google.com/search?q=anvil checked with userAgent = *:

{
  "inputValue": "https://www.google.com/search?q=anvil",
  "error": null,
  "errorMessage": null,
  "checkedAt": "2026-09-09T10:12:31.482Z",
  "billable": true,
  "userAgent": "*",
  "robotsUrl": "https://www.google.com/robots.txt",
  "robotsStatus": 200,
  "robotsState": "ok",
  "allowed": false,
  "matchedGroup": "*",
  "matchedRule": { "type": "disallow", "pattern": "/search" },
  "crawlDelay": null,
  "sitemaps": ["https://www.google.com/sitemap.xml"],
  "robotsTruncated": false,
  "redirectCount": 0,
  "robotsFromCache": false
}

Field	Type	Meaning
inputValue	string	URL as provided (credentials masked)
error	string or null	INVALID_URL (not charged), TIMEOUT, NETWORK_ERROR, TOO_MANY_REDIRECTS or null
errorMessage	string or null	Human-readable reason
checkedAt	string	ISO 8601 timestamp
billable	boolean	true when the record was charged; false for records written free of charge (INVALID_URL)
userAgent	string	The product token that was checked, after normalization (* when the input value was invalid)
robotsUrl	string or null	<scheme>://<host>[:port]/robots.txt derived from the URL; null only for INVALID_URL
robotsStatus	integer or null	HTTP status code of the final response for robots.txt; null when there was no final response (timeout, network error, too many redirects)
robotsState	string or null	ok (2xx, parsed), missing (4xx, including 401, 403, 404 and 410), unreachable (5xx, timeout, network error, too many redirects, redirect without a usable target), invalid (2xx but the body is not text), null for INVALID_URL
allowed	boolean or null	Decision for userAgent: from the rules when robotsState is ok; true when the file is missing or invalid (RFC 9309 section 2.3.1.3); false when it is unreachable (section 2.3.1.4); null for INVALID_URL
matchedGroup	string or null	Token of the group that decided, spelled as in the file, or *; for a group declared with several User-agent: lines, the first token of that group; null when the file has no matching group or was not used
matchedRule	object or null	{ "type": "allow" or "disallow", "pattern": "..." } of the longest matching rule; null means no rule in the group matched, so the URL is allowed
crawlDelay	number or null	Crawl-delay of the matched group in seconds, when declared
sitemaps	list of strings	All Sitemap: lines of the file with an absolute http(s) URL, in order, without duplicates; empty when there are none or the file was not used
robotsTruncated	boolean	true when the file exceeded 512,000 bytes and only the complete lines before that limit were processed
redirectCount	integer	Redirects followed while fetching robots.txt (0 to 5)
robotsFromCache	boolean	true when robots.txt of this host had already been fetched earlier in the same run

A SUMMARY record in the default key-value store holds the run counters (inputEntries, inputItems, skippedDuplicates, droppedEmptyOrNonString, truncatedOverLimit, resumedSkipped, processed, checked = charged, failed, notBilled, notSavedAfterLimit, unexpectedErrors, stoppedByChargeLimit).

Pricing

This Actor uses Apify pay per event pricing. Each run costs a small fixed start fee plus a price per URL checked; the exact amounts are shown in the Pricing tab. A URL is charged when it received a decision, which includes unreachable hosts (the work was done and "unreachable" is the answer RFC 9309 gives). Invalid entries (INVALID_URL) are written to the dataset but not charged. There are no additional platform usage fees.

A URL is charged only when its record is written, and every charged record is written. When the run's "Maximum cost per run" is reached, the Actor stops and reports how many URLs were left unprocessed.

Limits and behavior

• Up to 1,000 URLs per run; exact duplicates are skipped.
• Private, loopback, link-local and localhost addresses, URLs with embedded credentials and blocked ports are reported as INVALID_URL without any request and without charge. Credentials are masked in inputValue (user:***@host). A user name without a password is masked as well (https://***@host/); a password containing a raw /, \, ? or # is not recognized as credentials, so percent-encode such characters.
• A run interrupted by a platform migration resumes without processing or charging the same URL twice.
• One robots.txt request per host per run: all URLs of the same scheme, host and port share the fetched file (and share its error, if any); robotsFromCache marks the records that reused it.
• Up to 5 redirects per robots.txt; the sixth gives TOO_MANY_REDIRECTS and robotsState = unreachable. A redirect to another host is followed (the file at the target counts as the file of the original host); a redirect to a private address or without a Location header makes the file unreachable.
• The file is read up to 512,000 bytes (RFC 9309 section 2.5); a longer file is cut at the last complete line before that limit and robotsTruncated is true, so rules that appear later in the file are not applied.
• No retries: 403, 429 and 5xx answers for robots.txt are reported through robotsStatus and robotsState, not retried.
• Matching follows RFC 9309: the group for the exact token (case-insensitive) beats *, and no matching group means everything is allowed; several groups with the same token are merged; * in a pattern matches any string, $ anchors the end of the path; the path compared is path plus query string, percent-encoding normalized on both sides; the rule with the longest pattern wins and a tie goes to allow; an empty Disallow: is no rule; line keys are case-insensitive, comments start with #, unknown keys and a leading BOM are ignored; Sitemap: lines are read regardless of the group; Crawl-delay: is taken from the matched group.
• The Actor sends a descriptive User-Agent (GridlineTools-RobotstxtChecker) so site owners can identify it.

FAQ

Why is a URL allowed when the site has no robots.txt? A 404, like any other 4xx answer (401, 403, 410), counts as "unavailable" under RFC 9309, and a robot may then fetch everything. The record shows robotsState = missing and allowed = true.

Why is every URL of a host disallowed after a server error? When robots.txt answers with 5xx, times out or cannot be reached, RFC 9309 tells robots to assume they may fetch nothing. The record shows robotsState = unreachable, allowed = false, and it is charged because the answer is a real result.

Does Googlebot match Googlebot-Image? No. Tokens are matched as a whole, so Googlebot-Image only matches its own group. When the file has no group for your token, the * group decides, and without a * group everything is allowed.

What does matchedRule = null with allowed = true mean? The matching group has no rule that covers this path, so the URL may be fetched. matchedGroup still tells you which group was consulted.

Is Crawl-delay part of the standard? No, but many hosts declare it and many crawlers honor it, so the Actor reports the value from the matched group. null means the group does not declare one or declares a value that is not a number.

Can I check servers on my own network or on localhost? No. The Actor runs in Apify's cloud and only checks publicly reachable hosts; private, loopback and link-local addresses are rejected as INVALID_URL without a request.

Other Actors by Gridline Tools

Other checks from the same publisher, each with one row per address and the same pricing rule (start fee plus price per checked item):

• HTTP Status and Headers Checker: Audit a list of URLs in one run: status code, every hop of the redirect chain and the final URL, response time, server and content type, and six key security headers.
• RDAP Domain Lookup and Expiry Checker: Query RDAP registries for a list of domains and get registrar, creation and expiry dates, EPP statuses, name servers and DNSSEC in one table.
• AI Crawler Access Audit: See which AI and search crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and 150+ more) each site allows or blocks in robots.txt, and whether llms.txt is published.

Disclaimer

The Actor sends plain HTTP requests only for the robots.txt file of the hosts you list and reads that file; it never requests the pages themselves and never stores page content. You are responsible for having the right to query those URLs and for complying with the terms of service of the sites you check. Gridline Tools does not store the checked URLs or results beyond your own Apify account.