Sitemap Robots Delta Monitor

Track sitemap and robots.txt URL inventory changes for SEO, website migration, and competitor monitoring workflows.

Why use this Actor?

Most sitemap tools give you a one-time URL dump. Most website change monitors compare page content. This Actor sits between those use cases: it checks public robots.txt and sitemap files on a schedule and returns only URL inventory changes.

Use it to monitor:

• new competitor landing pages
• removed or changed sitemap URLs during migrations
• fresh content URLs from publisher or documentation sites
• sitemap metadata changes before a larger crawl
• URL lists for downstream crawlers, RAG ingestion, or webhook automations

Features

• Discovers sitemap URLs from robots.txt.
• Also tries /sitemap.xml.
• Accepts direct sitemap index or sitemap URLs.
• Traverses sitemap indexes within safe limits.
• Supports regex include and exclude filters.
• Can optionally check URL HTTP status with lightweight HEAD requests.
• Keeps state across runs by stateKey.
• Emits new, changed, and removed URL events.
• Produces dataset rows plus OUTPUT, RUN_SUMMARY, and WEBHOOK_PAYLOAD.

Input

{
  "mode": "delta",
  "changeMode": "new_and_changed",
  "stateKey": "seo-watch",
  "sites": ["https://apify.com"],
  "includeRobotsSitemaps": true,
  "includeDefaultSitemap": true,
  "sameHostOnly": true,
  "maxSitemapsPerSite": 10,
  "maxUrlsPerSite": 1000,
  "maxItems": 100,
  "includePatterns": ["/store", "/actors"],
  "excludePatterns": ["/login", "/users/"],
  "checkStatus": false,
  "includeRemoved": true,
  "webhookMode": "summary"
}

Output

Each dataset item represents one URL inventory event:

{
  "change_type": "new",
  "site": "https://apify.com",
  "url": "https://apify.com/store",
  "sitemap_url": "https://apify.com/sitemap.xml",
  "lastmod": "2026-05-05T00:00:00.000Z",
  "changefreq": "",
  "priority": "",
  "http_status": null,
  "final_url": "",
  "stable_id": "sitemap:...",
  "fingerprint": "...",
  "state_key": "seo-watch"
}

Pricing signal

The primary value event is a URL change alert. For public Store monetization, use pay per result with the dataset item event as the primary event.

Limitations

• Only public robots and sitemap data is used.
• The Actor does not fetch page bodies by default.
• lastmod quality depends on the website's sitemap.
• Optional HTTP status checks increase runtime.

Suggested schedule

For most SEO monitoring, run daily or every few hours. Use separate stateKey values for unrelated site groups.