# Changelog of Crawlability & AI Crawler Access Auditor (`lwsdjfls/crawlability-ai-access-auditor`) Actor

- **URL**: https://apify.com/lwsdjfls/crawlability-ai-access-auditor/changelog.md
- **Full Actor documentation**: https://apify.com/lwsdjfls/crawlability-ai-access-auditor.md

## Changelog

### 0.1

- Initial release: robots.txt parsed per crawler for 29 AI, search, dataset and archive tokens plus your own; Cloudflare Content-Signal and IETF Content-Usage; sitemap tree with URL counts and lastmod range; llms.txt, llms-full.txt and ai.txt; meta robots, X-Robots-Tag and TDM headers on a page sample; TDMRep (tdmrep.json, headers, meta) and terms-page scanning in four languages; bot-management detection for fourteen services; audit and monitor modes.
- Hardened for hostile sites: sitemaps (declared, listed in an index or discovered) and sample pages are only fetched on the domain, its www twin or the host its homepage redirects to, compared by parsed host; other sitemaps stay in `sitemaps.declared` and are named in a warning, apart from locations that are not absolute http(s) URLs. A sitemap is read up to 32 MB per file (downloaded and unpacked; a refused file counts as 32 MB) and 512 MB of sitemap text per domain, every other response up to 4 MB (a larger robots.txt, homepage or llms.txt is named as too large to read, a larger llms-full.txt or ai.txt that answers 200 counts as present, and a larger error page, such as a 404 of megabytes, counts as a missing file and is not downloaded again), robots.txt up to 500 KiB (like Google); all HTML, sitemap, llms.txt and robots.txt readers run in linear time on unclosed tags and long lines, sitemaps and feeds are counted without collecting their entries, and robots.txt wildcards are matched the way Google's parser does, in linear time (a rule of thousands of `*` no longer stalls or fails an audit). What a record keeps is copied off the page, file or sitemap it was read from, so memory does not grow with the size of what a run read. The record keeps at most 500 robots.txt rules per group and 2,000 in total, 200 Sitemap lines and 20 Content-Usage rules per group (`robots.truncated: true` when a file was cut); an audit too large for a dataset item fails on its own, free, instead of failing the run. An out-of-range character reference no longer fails a domain. A robots.txt that cannot be read (over 4 MB, answering 5xx or 429, or a failed request) is no longer reported as a site without robots.txt: every crawler row is `unknown`, no crawler is listed as allowed, the verdict is at least `restricted`, the summaries say crawler access is unknown, and a monitor reports neither blocks as lifted when it becomes unreadable nor blocks as new when it can be read again. A tdmrep.json that cannot be read is no longer named among the places searched for a TDM reservation. A sitemap found at a usual location that answers 404 or 410 when it is read, or answers 200 with something that is not a sitemap (a plain-text or JSON error page), is dropped, and the next location is tried; another error on that read keeps it, with the error. Sitemap files over 4 MB are downloaded at most two at a time across the whole run, and gzip sitemaps are unpacked one at a time, so sitemap memory does not grow with the number of domains in parallel; smaller files, gzip ones included, are read in parallel as before and do not wait for another domain's large download. The scraper summary counts sitemap files that could not be read (an error status, a failed request, too large, not a sitemap) with their reasons, and says the URL count is unknown when none could be read, instead of "0 URLs"; past the cap it counts the files requested, not read, when one of them could not be read. Sitemaps declared in robots.txt or found at a usual location but not fetched (on another host, not an absolute http(s) URL, or **Maximum sitemap files** 0) are summarised as such, not as "No sitemap declared or found at the usual locations". A monitor reports a robots.txt that could not be read and now answers without a file as "robots.txt could be read again: none (HTTP 404)", or as "robots.txt gone" when the earlier file was too large to read, and a domain that is unreachable in one of the two runs by its verdict only, not as blocks lifted, llms.txt gone or robots.txt gone.
- The charge for a domain is reserved before its audit starts: once the spending limit is reached, the remaining domains are not audited and are named in a warning. A run whose spending limit does not cover a single domain now ends with that warning (exit code 0) instead of failing. The webhook URL is a secret input, stored encrypted.