# Changelog of AI Crawler Access Checker: robots.txt & llms.txt (`offerastudio/ai-crawler-access-audit`) Actor

- **URL**: https://apify.com/offerastudio/ai-crawler-access-audit/changelog.md
- **Full Actor documentation**: https://apify.com/offerastudio/ai-crawler-access-audit.md

## Changelog

### 1.0 (2026-09-30)

- First public version.
- robots.txt parsed as RFC 9309 specifies: groups, case-insensitive user-agent matching, combined groups, `*` fallback, longest match wins, Allow wins a tie, `*` and `$` wildcards, 500 KiB limit.
- 26 AI crawler tokens checked against vendors' own documentation (OpenAI, Anthropic, Google, Apple, Meta, Amazon, Perplexity, Mistral AI, DuckDuckGo, Common Crawl, Diffbot) plus common undocumented tokens (Bytespider, anthropic-ai, cohere-ai). Each gets allowed / partial / blocked with the rule, line and a plain-English reason.
- AI visibility verdict (open to all AI, blocks training only, blocks AI search, blocks all AI …) and Googlebot/Bingbot access for context.
- `/llms.txt` and `/llms-full.txt`: presence, H1 title, summary, link sections, size and format issues.
- Sitemap URLs from robots.txt, or `/sitemap.xml` when none are listed.
- Pay per event: `domain-checked` at $0.002. Domains that can't be reached are free.