Sitemap Monitor: New Pages & Content Updates

Watch up to 10 website sitemaps on a schedule and receive only what changed since the previous run: the URLs that did not exist before, and the pages whose <lastmod> moved. Every page a company publishes lands in its sitemap first, so a competitor's publishing calendar is already public. What is missing is the memory and the diff, and that is what this monitor adds.

SEO teams pay serious money for "competitor content monitoring". The raw signal is free and sitting in sitemap.xml: what they published, what they refreshed, what they deleted.

What the sitemap monitor detects

1. Auto-discovers each site's sitemap: direct .xml/.xml.gz URL, robots.txt Sitemap: declaration, or the conventional /sitemap.xml — with sitemap-index recursion and gzip support.
2. Reads every page URL and its <lastmod>.
3. Compares against the previous run and outputs:
   • changeType: "new" — pages that didn't exist before (their new content),
   • changeType: "updated" — pages whose lastmod changed (their content refreshes),
   • removed-page counts in the run summary.

Competitor content monitoring, in practice

• Competitor content watch: weekly schedule on your competitors → every new blog post, landing page or product page they ship, in your inbox before it ranks.
• SEO agencies: one monitor per client vertical; new competitor pages = fresh brief opportunities for your client.
• Your own site: catch pages accidentally dropped from your sitemap (removedPages in the summary) — a classic silent SEO regression.

Input example

{
  "siteUrls": [
    "https://competitor-a.com",
    "https://competitor-b.com/sitemap.xml"
  ]
}

The defaults work as-is.

Output example

{
  "siteUrl": "https://competitor-a.com",
  "pageUrl": "https://competitor-a.com/blog/new-feature-announcement",
  "lastmod": "2026-07-06T08:00:00.000Z",
  "changeType": "new",
  "isBaseline": false,
  "detectedAt": "2026-07-07T06:00:00.000Z"
}

Data returned for each changed page

Field	What it holds
siteUrl	The watched site the page belongs to
pageUrl	The URL that is new or was updated
lastmod	The <lastmod> value read from the sitemap
changeType	new, updated or baseline
isBaseline	true on the first run, when the memory is created
detectedAt	When this run saw the change

The run SUMMARY adds, per site: pages in sitemap, new, updated and removed counts, which sitemap was used, and any truncation flag.

Fair use & data

• Reads only robots.txt and sitemap files — the two files websites publish specifically for automated readers.
• No page content is fetched, no personal data collected.
• Hard caps (10 sites, 50 sitemap files, 20k pages per site) keep runs bounded; truncation is always flagged, never silent.

FAQ

A site has no sitemap? The run summary reports the error for that site and the others still work.

Huge site (100k+ pages)? Raise maxUrlsPerSite up to 20k, or point siteUrls at the specific sub-sitemap you care about (e.g. sitemap-blog.xml).

First run floods me. That's the baseline (flagged isBaseline: true); disable includeBaselineInOutput to start silent.

How do I reset the memory? Set a new monitorId.