AI Crawler Checker
Pricing
from $2.00 / 1,000 site checkeds
AI Crawler Checker
Shows what your site tells AI crawlers such as GPTBot and ClaudeBot: robots.txt rules, the homepage robots tag, ai.txt and llms.txt. One row per site with a short summary and a robots.txt snippet you can copy. $0.002 per site. General information only.
Pricing
from $2.00 / 1,000 site checkeds
Rating
0.0
(0)
Developer
Vi Zellie
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Share
See what a website's public files say to known AI crawlers. Each site gets one row: which listed crawlers are allowed, blocked, or blocked only on some paths, plus a short summary and a robots.txt snippet for the policy you choose.
General information only.
This reads files the site publishes. It does not show whether a crawler actually visits.
What you get
For each public site:
- Whether
robots.txtwas found. - For each listed crawler: allowed, blocked, or blocked only on some paths. If
robots.txtcannot be reached or the server fails, every listed crawler is blocked. If the file is missing or refused, the crawlers are allowed. - Whether the homepage asks robots not to index or not to follow, from the page and from the response.
- Whether
ai.txtandllms.txtare present. Their text is not saved. - A plain-language summary.
- A robots.txt snippet you can paste, matching the policy you chose.
Unreachable sites are stored with no crawler list and are not charged. Private addresses, internal names, and URLs on a port other than 80 or 443 are skipped and are not charged. The summary counts every URL you supplied, how many were accepted, and how many of those unsafe URLs were skipped. A run whose addresses are all private still finishes, with nothing checked.
How to use it
- Paste your own site URLs under Website URLs. Use a full URL such as
https://example.com/. Up to 100 sites. Extra sites beyond that cap are counted and not checked. - Choose a policy:
- Allow all: the snippet allows every listed crawler.
- Block training: the snippet blocks crawlers commonly used to train models, and allows crawlers that fetch a page because a person asked, or that power a search answer.
- Block all: the snippet blocks every listed crawler.
- Start the run. Open the dataset for one row per site. The
OUTPUTrecord has the run counts.
The snippet is a suggestion. You still edit robots.txt on your own site.
Pricing
You pay a small fee when the run starts, plus a fee for each public site that could be checked. A site that cannot be reached, or a private address, is free.
| Event | Price |
|---|---|
| Run start | $0.00005 |
| Site checked | $0.002 |
What is not collected
robots.txt, ai.txt, llms.txt and the homepage text are not saved. Email addresses in those files are not saved. Query strings and fragments are not saved. Logs name the domain and how many crawlers were blocked or allowed.
Limits
Up to 100 sites in one run. Each request times out, redirects are capped, and response bodies are capped. Only http and https on ports 80 and 443 are accepted.
