Robots.txt and Sitemap.xml Change Monitor Actor

Status: published on Apify.

Open the Actor on Apify

This Apify-style actor checks public robots.txt and sitemap.xml metadata for SEO deploy regressions. It reports changed crawler directives, missing sitemap references, sitemap URL additions/removals, sitemap indexes, XML parse problems, and high-risk sitewide disallow rules.

Safe Use Boundary

• Public robots.txt and sitemap.xml metadata only.
• No account login, CAPTCHA handling, MFA, private workspace access, or evasion.
• No high-frequency polling. Buyers should use reasonable scheduled checks.
• Buyers are responsible for using it only on sites they own, maintain, or are allowed to monitor.

Input

{
  "sites": [
    {
      "url": "https://example.com",
      "fetch_live": false,
      "robots_text": "User-agent: *\nDisallow: /private\nSitemap: https://example.com/sitemap.xml\n",
      "sitemap_text": "<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\"><url><loc>https://example.com/</loc></url></urlset>",
      "previous_snapshot": {
        "robots_sha256": "previous hash",
        "sitemap_urls": ["https://example.com/old"]
      }
    }
  ]
}

For live runs, set fetch_live to true; the actor fetches the public root robots.txt and the first discovered sitemap URL.

Output

• summary.site_count, warning_count, and change_count.
• sites[].robots: hash, sitemap declarations, allow/disallow rules, and warnings.
• sites[].sitemap: parsed sitemap URLs, lastmod values, sitemap index URLs, and warnings.
• sites[].changes: records with record_type of robots or sitemap_url and change_type of changed, new, removed, or unchanged.

Local Commands

python -m unittest discover -s tests -v
python .\robots_sitemap_monitor.py --input .\samples\input.json --out .\samples\output.json
python .\main.py

Differentiator

Generic SEO suites bury this behind broad site audits. This actor is a small, scheduled metadata sentinel: it produces change records and stable hashes that can feed alerts, regression dashboards, and release checks without crawling whole websites.