Sitemap, robots.txt & RSS Change Monitor

Monitor public website source files that often reveal SEO, content, and publishing changes: robots.txt, sitemap XML, and RSS/Atom feeds. The Actor stores a normalized snapshot hash per target and outputs changed targets by default.

What it does

• Checks public HTTPS websites without browser automation.
• Reads robots.txt, sitemap XML URLs, and RSS/Atom feed links.
• Normalizes volatile fields such as fetch duration and check time out of the comparison hash.
• Compares the current source snapshot to the previously stored baseline.
• Outputs changed targets with changeTypes such as robots, sitemaps, feeds, and status.

Good use cases

• SEO teams monitoring sitemap or robots.txt regressions.
• Content teams watching public RSS/Atom source changes.
• Agencies checking client websites after deploys.
• RAG/data-ingestion teams tracking source-file drift.

Input

• targets: domains or HTTPS URLs to check.
• maxTargets: safety cap, maximum 25 targets in v1.
• maxSitemaps: maximum sitemap URLs fetched per target, max 5.
• maxFeeds: maximum RSS/Atom feeds fetched per target, max 5.
• includeUnchanged: include targets that have not changed since the previous baseline.
• baselineKey: namespace for separate schedules, clients, or environments.

Example input:

{
  "targets": ["example.com"],
  "maxTargets": 10,
  "maxSitemaps": 3,
  "maxFeeds": 2,
  "includeUnchanged": false,
  "baselineKey": "production"
}

Output

Each dataset item represents one checked target that changed, or one included unchanged target when includeUnchanged is enabled.

Important fields:

• normalizedTarget: normalized HTTPS target URL.
• status: ok, partial, or error.
• changed: whether this target differs from the stored baseline.
• changeTypes: source groups that changed.
• robots, sitemaps, feeds: source-specific fetch metadata and extracted links.
• warnings: non-fatal warning codes or messages.
• snapshotHash: stable hash used for future comparisons.

Pricing

This Actor uses Apify Pay Per Event with one custom event:

Event	When it is charged	Current price
checked-target	Before each public target is fetched and checked, including unchanged checks that may be omitted from output	$0.0006

That equals $0.60 per 1,000 checked targets. Pricing is target-based rather than dataset-item based because unchanged checks still consume runtime while includeUnchanged=false omits unchanged dataset rows.

Platform usage is included in the Actor price on the public Store pricing page.

Limitations

• Public HTTPS targets only.
• No browser automation, screenshots, login pages, or cookie-based crawling.
• No private-network fetching and no PII enrichment.
• Sitemap and feed fetch counts are intentionally capped for predictable low-cost runs.
• A partial status can still be useful: it means at least one source returned a warning while other source checks completed.

Recommended workflow

1. Run once with a stable baselineKey to create the first baseline.
2. Schedule repeat runs with the same baselineKey.
3. Keep includeUnchanged=false for compact change-only datasets.
4. Enable includeUnchanged=true only when you need a full audit trail.