# Changelog of Robots.txt and AI Crawler Access Checker API (`rowcraft/robots-ai-crawler-checker`) Actor

- **URL**: https://apify.com/rowcraft/robots-ai-crawler-checker/changelog.md
- **Full Actor documentation**: https://apify.com/rowcraft/robots-ai-crawler-checker.md

## Changelog

### 1.0.3 (2026-10-17)

- Fix: a robots.txt served as an HTML page (catch-all page or a redirect to the home page, for example thumb.wikimedia.org) counted as an anti-bot block, so every page on that host was refused; it is now treated as a missing robots.txt, as the standard says. Real anti-bot pages are still recognised.

### 1.0.2 (2026-10-17)

- Shared library: URL-like fields named link or guid (feed items) are no longer flagged as phone numbers; retry log line now reads "retry n/m".
- Shared library: an HTML page where JSON was expected (maintenance or block page, seen on 17.10.2026 at the Polish KRS API and at Workday) is retried and then reported with the page text instead of crashing the run on a parse error.

### 1.0.1 (2026-10-16)

- README: real screenshots of the input form and of the results (Console).
- Icon uploaded in Console; two example tasks run on the platform.

### 1.0.0 (2026-10-15)

- First release: which AI and search crawlers a website lets in, read from its robots.txt exactly as RFC 9309 describes (groups of user-agent lines, the longest matching rule wins, Allow wins a tie, `*` and `$` in patterns, percent-encoding, the 500 KiB limit, a missing file allows everything, a server error disallows everything, redirects followed).
- 27 crawlers in 13 operators (OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Common Crawl, Amazon, Meta, DuckDuckGo, Mistral, Yandex, Baidu). A crawler is on the list only when its operator's own documentation names the token; the source of each one is in the README.
- One row per website: one true/false column per crawler, the lists of allowed and blocked crawlers, an AI training policy, an AI search policy and a search visibility summary, the access of every crawler on every path you give, the sitemaps and the Crawl-delay declared in robots.txt, the names of all crawlers the file mentions, llms.txt and llms-full.txt (present, status, size), and the meta robots tag and X-Robots-Tag header of the home page.
- Your own paths (default: the home page) and your own crawler names are checked in the same run.
- Documented fallbacks are applied: Applebot follows the Googlebot group, Bingbot the msnbot group, YandexBot the Yandex group, when robots.txt has no group for the crawler's own name.
- `COVERAGE` per website (ok, blocked, not_found, error, not_checked, with the reason); anti-bot pages, servers that refuse robots.txt, missing websites, invalid entries and duplicates are never charged; the run stops after 10 anti-bot pages in a row.
- Standard Rowcraft automation: only websites whose policy changed since the last run (`onlyNew`, `stateName`), end-of-run notification (`notifyUrl`), charge cap respected, stop with the summary saved 90 seconds before the run timeout, one request per second by default (minimum 250 ms), the Crawl-delay of robots.txt for our own crawler kept up to 15 seconds.