SiteVerdict — robots.txt & Sitemap Monitor

Find out within the hour when a deploy blocks Googlebot, noindexes your best page or empties your sitemap.

A single Disallow: / or a stray noindex left over from staging can remove a site from Google within days. This Actor watches robots.txt, your XML sitemaps and your key pages. It compares every run with the last one and alerts you in Slack or by webhook when something risky changes.

Who it is for

• SEO agencies. Monitor every client site from one schedule and learn about a broken deploy before the client does.
• In-house SEO and dev teams. Use it as a post-deploy guard-rail for robots.txt, sitemaps, canonicals and noindex.
• Site owners and e-commerce managers. Get an alert when the sitemap loses half its products or a category page starts redirecting.

What it checks

Area	Checks
robots.txt	HTTP status (5xx = Google stops crawling), RFC 9309 rules, access for *, Googlebot, Bingbot and AI crawlers (GPTBot, Google-Extended, ClaudeBot, PerplexityBot, CCBot), blocked CSS/JS, size over 500 KiB, missing Sitemap: line, Crawl-delay, rule-level diff
Sitemaps	Discovery from robots.txt or /sitemap.xml, nested indexes, .gz, HTTP errors, HTML soft-404s, XML problems, namespace, 50,000-URL and 50 MB limits, URL count and drops, URLs added or removed, invalid or future lastmod, off-host and http:// URLs, duplicates
Key URLs	Status, redirects and final URL, noindex (meta robots, googlebot meta, X-Robots-Tag), canonical (tag or Link header), blocked by robots.txt for each search crawler
Sitemap spot checks	N URLs spread across the sitemap: 200, indexable, not redirected, self-canonical

Each issue has a severity (critical, warning or info) and a plain-English fix. Each change shows its before and after values.

Example: the alert you get

{
  "site": "https://shop.example.com",
  "riskLevel": "critical",
  "summary": "5 change(s) since 2026-10-10 08:00: robots.txt now blocks the entire site for Googlebot. noindex appeared on https://shop.example.com/products/blue-shoe (meta robots). Sitemap URL count fell 50% (6 → 3).",
  "changes": [
    { "code": "robots-newly-blocks-all", "severity": "critical", "message": "robots.txt now blocks the entire site for Googlebot." },
    { "code": "key-noindex-added", "severity": "critical", "message": "noindex appeared on https://shop.example.com/products/blue-shoe (meta robots)." },
    { "code": "sitemap-count-drop", "severity": "critical", "message": "Sitemap URL count fell 50% (6 → 3)." }
  ],
  "robotsDiff": "+ User-agent: Googlebot\n+ Disallow: /"
}

Run with Mode = Sample to see this full output for free, without any network calls.

How to use

1. Add your sites and your money pages under Key URLs.
2. Run it once to save the baseline.
3. Add an Apify Schedule (daily, or hourly with Output = "Only changes").
4. Optionally add a Slack or webhook URL and choose the alert threshold.

Each run writes:

• Dataset: one row per site. Views: Overview, Changes and alerts, Open issues and fixes, Crawler access.
• Key-value store: REPORT (HTML report), REPORT_CSV (one line per issue or change), SUMMARY (compact JSON for agents and integrations).
• State: the named store (default siteverdict-robots-sitemap-monitor) keeps each site's last snapshot and change history. Use one store name per client.

Pricing

Pay per event: $0.01 per site check, $0.001 per URL checked, $0.01 per delivered alert, and $0.02 per AI answer if you turn that on. Sample mode is free. Every checked site is charged, including unchanged sites in "Only changes" mode.

• 1 site checked daily with 5 key URLs and 10 spot checks costs about $0.75 a month.
• 50 client sites checked daily cost about $20 a month.

See PRICING.md for details.

Optional: AI fix suggestions (bring your own key)

Turn on AI fix suggestions and paste your own Anthropic or OpenAI key. For sites with warnings or critical issues, the findings are sent to your provider, never any personal data. You get a prioritised JSON list of fixes, each with an exact snippet (robots.txt lines, sitemap XML or HTML tags). Your key is stored encrypted and never logged or output. It is sent only to the provider you choose (Anthropic or OpenAI), and redirects are never followed, so it cannot be forwarded to another host.

FAQ

Does it crawl my whole site? No. It reads robots.txt, the sitemap files and the key URLs, plus up to 100 spot-check URLs per site. Requests to one host are sequential, at most one per second, and slower if your robots.txt sets a Crawl-delay.

What if robots.txt blocks the Actor itself? The Actor obeys robots.txt for user-agent SiteVerdict (or *), so pages it may not fetch are reported as "Not fetched" and not charged. robots.txt access for Googlebot and others is evaluated from the rules alone; we never pretend to be Googlebot.

Will a cosmetic robots.txt edit trigger an alert? No. Comments, blank lines, whitespace and field-name case are ignored. Only real rule changes count.

Can it check Search Console? No. It uses public URLs only and needs no Google login.

Run time, aborts and partial results

• Run time is mostly polite waiting: requests to one host are at most one per second (slower with a Crawl-delay), so one site with 5 key URLs and 10 spot checks usually takes well under a minute; a site with 50 large sitemap files takes several minutes. Up to 3 sites run in parallel by default. Memory does not make it faster.
• Each site is saved as soon as it is done. Its dataset row is pushed and its events are charged together (except in "Only changes" mode, see below), right after its checks, alert and AI step finish. If you abort the run or it times out, finished sites keep their rows and were charged; the site in progress has no row and is not charged (except a change alert that was already delivered), and sites not started yet are not charged.
• "Only changes" mode: a site that has not changed since the last run is still checked and charged (the site check and its URL checks), but it gets no dataset row. It still appears in REPORT, REPORT_CSV and SUMMARY. The first (baseline) run and every run with a change write a row as usual.
• Saved only at the end: the REPORT, REPORT_CSV and SUMMARY records are written when the run finishes, so an aborted run has none of them. The dataset rows are complete on their own.
• Monitoring state after an abort: a site's new snapshot is stored as soon as its checks finish, before its alert is sent. If the run is aborted in between, that change will not be alerted again next time; look at the aborted run's dataset (changes).
• Partial success is normal: a site that cannot be fetched, a blocked URL or one bad sitemap gives findings or an error row for that site; the run still succeeds for the others.

Limits

• At most 50 sitemap files per site by default (configurable up to 500). The URL list stored for diffs is capped at 100,000 URLs per site (each sitemap file over the 50,000-URL protocol limit is reported as an error).
• Sites behind bot protection, a login or a CAPTCHA are reported as fetch errors. We never try to bypass them.
• JavaScript-injected meta robots or canonical tags are not seen, because pages are fetched as HTML without a browser.
• Allowed ports are 80, 443, 8080, 8443, 8000 and 8888. Private and internal addresses are refused.

Sources and compliance

• Data sources: public robots.txt, sitemap and HTML responses of the sites you list. No third-party API and no personal data.
• Standards: robots.txt follows RFC 9309 (https://www.rfc-editor.org/rfc/rfc9309, checked 2026-10-11). Sitemaps follow the sitemaps.org protocol (https://www.sitemaps.org/protocol.html, checked 2026-10-11). Google limits (500 KiB robots.txt, 5xx handling) follow https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt (checked 2026-10-11).
• Polite crawling: honest User-Agent SiteVerdict/robots-sitemap-monitor (+https://apify.com/siteverdict/siteverdict-robots-sitemap-monitor). robots.txt is respected on every redirect hop, and a 5xx robots.txt is treated as "disallow all". No proxy rotation and no fingerprinting.
• Security: SSRF guard on every connection and every redirect (private, loopback, link-local and metadata addresses are refused; the DNS answer is checked inside the connection itself), size caps and timeouts. Webhook and Slack URLs must be public https:// addresses; unsafe ones are refused when the run starts, and delivery errors never repeat the URL or its host. Secrets (webhook URLs, AI key) are stored encrypted and redacted from logs and output.
• UK GDPR / PECR: the Actor processes no personal data. Only monitor sites you own or are authorised to monitor.