# Changelog of SiteVerdict — robots.txt & Sitemap Monitor (`siteverdict/siteverdict-robots-sitemap-monitor`) Actor

- **URL**: https://apify.com/siteverdict/siteverdict-robots-sitemap-monitor/changelog.md
- **Full Actor documentation**: https://apify.com/siteverdict/siteverdict-robots-sitemap-monitor.md

## Changelog

### Unreleased (NEXT5 prep, 17 Oct 2026)

- AI fix step: the Anthropic and OpenAI calls never follow redirects, so your key and findings go only to the provider you chose.
- Webhook and Slack URLs are checked when the run starts; delivery errors are fixed texts that never repeat the webhook host.
- Each site is charged right before its row is saved (after its alert and AI step), so an aborted run never charges a site with no row.
- README: run time, aborts and partial results; stored-URL cap corrected to 100,000; "Only changes" mode stated plainly (an unchanged site is still charged but gets no dataset row).

### 1.0.0 (2026-10-11)

- First release.
- robots.txt monitoring: RFC 9309 parsing, crawler access for Googlebot, Bingbot, `*` and AI crawlers, and rule diffs that ignore comments and whitespace. Flags `Disallow: /` on search crawlers, 5xx robots.txt, blocked CSS/JS and robots.txt over 500 KiB.
- Sitemap monitoring: discovery from robots.txt or /sitemap.xml, nested indexes, `.gz` files, URL counts, lastmod quality, off-host and http URLs, size and URL-count limits, URL add/remove diffs, and count-drop thresholds.
- Key URL watch: HTTP status, redirects, noindex (meta robots, googlebot meta, X-Robots-Tag), canonical, and robots.txt blocking per crawler.
- Sitemap spot checks of indexability.
- Named-store state with change history, "only changes" output mode, Slack and webhook alerts (https only).
- HTML report, CSV report and an agent-ready SUMMARY.
- Optional AI fix suggestions using your own Anthropic or OpenAI key.
- Free sample mode.