Sitemap Change Monitor: New Pages with Titles

What does Sitemap Change Monitor do?

Sitemap Change Monitor watches the sitemaps of the sites you choose and tells you what changed since the last run: pages that were added, removed or updated. Every new page arrives with its title and meta description, so an alert reads "New page on competitor.com (/blog/): Pricing for teams, explained" instead of a bare URL. Each site also gets one summary row with per-section counts (/blog/ +4, /products/ +2) and a ready-to-post message for Slack, email or a spreadsheet.

• 🆕 New pages with titles: title, meta description, H1, canonical, noindex flag, publish date and language for every added page, read from the page itself (robots.txt respected).
• 🗂️ Per-section digest: one row per site with added, removed and updated counts by section, the newest pages, and a one-line message.
• 🔕 Only changes: the first run saves a baseline; later runs return only what changed. In "only when changed" mode a quiet site produces no row, so Make, Zapier or Slack fire only when there is news.
• 🧭 Finds the sitemaps for you: robots.txt, sitemap indexes (nested), gzipped .xml.gz files, plain-text and news sitemaps, and the site's RSS/Atom feed when there is no sitemap.
• 👥 Many sites per run: up to 100 competitors or clients in one scheduled run.

Generic sitemap scrapers give you the full URL list every time, and most sitemap diff tools give you bare URLs. This Actor gives you only the changes, explains each new page, and groups them by section.

Who is it for?

• SEO teams: see a competitor's new landing pages and content clusters the day they appear in the sitemap, and catch pages dropped from your own sitemap after a release.
• Content marketers: a daily or weekly feed of what competitors published, with titles and descriptions, ready for a Slack channel or a content brief.
• Competitor and market watchers: new product, pricing, careers or docs pages across a whole list of companies, one summary per company.

What data do you get?

Two kinds of rows, told apart by type.

Change rows (type: "change"), one per added, removed or updated page:

Field	Example
changeType	added
domain	simonwillison.net
url	https://simonwillison.net/2026/Oct/15/latex/
section	/2026/
title	Tool: LaTeX renderer
description	LaTeX renderer turns LaTeX source into a rendered view, either a single display equation typeset by MathJax or a full document...
h1, canonical, noindex, publishedAt, language, httpStatus	as the page declares them
lastmod / previousLastmod	the sitemap's <lastmod> now and at the previous check
message	New page on simonwillison.net (/2026/): "Tool: LaTeX renderer" https://simonwillison.net/2026/Oct/15/latex/

Site rows (type: "site"), one per site and run: status (baseline, compared, partial, failed), urlCount, added, removed, updated, sections (up to 15, most active first), newPages (the 5 newest with titles), changesPending, source (sitemap or feed), discoveredVia, previousCheckAt and a message such as:

simonwillison.net: 2 new since 2026-10-17 17:15 UTC. New by section: /2026/ +2. Latest: "A quote from Addy Osmani, Morgan Westlee Lunt, and Qing Wang" https://simonwillison.net/2026/Oct/16/code-review/

Download as JSON, CSV, Excel or XML, or read it through the Apify API. The dataset has two views: Changes and site summaries, and Site summaries.

How to use it

1. Click Try for free and list your sites under Sites to watch (a domain, a home page, or a sitemap or feed URL).
2. Press Start. The first run saves each site's URL list as the baseline and returns one summary row per site.
3. Save the input as a task and add a schedule (daily is typical). Every later run returns only the changes since the run before.
4. For alerts, set Site summary rows to Only for sites that changed and connect the task to Slack, email, Make or Zapier: post each row's message.

Input example

This is the prefilled input:

{
    "sites": ["https://crawlee.dev", "https://astro.build"],
    "changeTypes": ["added", "removed", "updated"],
    "fetchPageDetails": true,
    "maxDetailsPerSite": 20,
    "maxChangesPerSite": 200,
    "summaryRows": "always"
}

Optional: includeUrlPatterns (watch only /blog/), excludeUrlPatterns (skip /tag/), sectionDepth (2 groups /docs/guides/ separately), maxUrlsPerSite, stateStoreName, resetState and maxConcurrency.

Output example

From a run on 2026-10-17, shortened. To show change rows without waiting a day, two existing posts were taken out of the saved snapshot, so the run reports them as new; the titles and descriptions were read live from the pages.

[
    {
        "type": "change",
        "domain": "simonwillison.net",
        "changeType": "added",
        "url": "https://simonwillison.net/2026/Oct/15/latex/",
        "section": "/2026/",
        "title": "Tool: LaTeX renderer",
        "description": "LaTeX renderer turns LaTeX source into a rendered view, either a single display equation typeset by MathJax or a full document or prose fragment typeset by faster-latex, with a live …",
        "canonical": "https://simonwillison.net/2026/Oct/15/latex/",
        "noindex": false,
        "language": "en-gb",
        "httpStatus": 200,
        "detailsStatus": "ok",
        "message": "New page on simonwillison.net (/2026/): \"Tool: LaTeX renderer\" https://simonwillison.net/2026/Oct/15/latex/",
        "previousCheckAt": "2026-10-17T17:15:07.481Z",
        "checkedAt": "2026-10-17T17:15:23.685Z"
    },
    {
        "type": "site",
        "domain": "simonwillison.net",
        "status": "compared",
        "message": "simonwillison.net: 2 new since 2026-10-17 17:15 UTC. New by section: /2026/ +2. Latest: \"A quote from Addy Osmani, Morgan Westlee Lunt, and Qing Wang\" https://simonwillison.net/2026/Oct/16/code-review/",
        "urlCount": 17079,
        "added": 2,
        "removed": 0,
        "updated": 0,
        "changesPending": 0,
        "removalsChecked": true,
        "sections": [{ "section": "/2026/", "urls": 1073, "added": 2, "removed": 0, "updated": 0 }],
        "checkedAt": "2026-10-17T17:15:23.685Z"
    }
]

How much does it cost?

Pay per event, two events. Platform usage is included.

Event	When	Price (Free and Bronze)	Silver	Gold
Change or site summary row	One row saved: an added, removed or updated page, or one site's summary	$0.0016 ($1.60 per 1,000)	$0.00144	$0.00128
New page details	One added page opened to read its title and description	$0.001 ($1.00 per 1,000)	$0.0009	$0.0008

Plus Apify's standard start fee of $0.00005 per run. Failed checks (no sitemap or feed found, site blocked) are free.

Why two events: the summary row pays for downloading a site's sitemap, which is the real work on a quiet day; page details are separate work you can switch off or cap. Examples:

• 10 competitors, daily, 3 new pages on a typical day: 13 rows and 3 page reads, about $0.024 a day, about $0.72 a month.
• The same with "Only for sites that changed": quiet sites cost nothing; if the 3 new pages are on 2 sites, that is 5 rows and 3 page reads, about $0.011 a day.
• A site migration with 5,000 changed URLs: capped at 200 rows per site by default ($0.32); the rest come in the next runs.
• An online store whose platform reshuffles product dates (some Shopify stores serve a different <lastmod> on every request): those date changes are recognised and ignored, so they cost nothing. New and removed products are still reported.

Set a maximum cost per run and the Actor stops cleanly when it is reached. Changes it could not save are kept and reported by the next run, never charged twice. (Exception: if you abort a run while a site is being saved, that site keeps its previous snapshot and its changes are reported again.)

Snapshots and how to reset them

Each site's last URL list is kept in a named key-value store in your account, huffworks-sitemap-monitor-state by default (one gzipped record per site, URLs and lastmod dates only). All runs and tasks that use the same store name share it.

• Start one site over: in Console open Storage > Key-value stores > huffworks-sitemap-monitor-state (or the name in the run log, if that name was taken) and delete that site's record (named after the domain), or run once with that site and Start over turned on.
• Start everything over: delete the store, or set a new Snapshot store name.
• Separate watchlists (for example one per client): give each task its own store name.

Limits and good to know

• Only what the sitemap says. A page that is live but not in the sitemap is invisible; a page taken out of the sitemap is reported as removed even if it still loads.
• Updated depends on <lastmod>. Many sites publish no lastmod at all (several of our test sites did not), and some stamp the build time on every URL. A page counts as updated only when its lastmod moves forward past the latest date seen before. Lastmod changes are ignored, and counted in updatedSuppressed, when 90% of a site's URLs share one lastmod, when a sitemap file's dates move backwards (that file stays ignored until you start over), or when half or more of one file (at least 20 URLs) jumps forward at once.
• Big sites and the URL limit. When a site has more URLs than Maximum URLs per site, the newest sitemap files are read first (by the dates in the sitemap index). A page counts as added only if it comes from a sitemap file that was also read in full at the previous check, or from a file that newly appeared in the index. URLs from files read for the first time (for example after you raise the limit) are saved to the snapshot without rows or charges, and counted in urlsAbsorbed. The trade-off: on a site bigger than the limit, a new page can occasionally be missed, but an old page is never reported (or charged) as new. To watch every page of a big site, raise the limit above its size or watch only part of it with Only URLs matching.
• Removals are reported only after a complete read. If any sitemap file fails or a limit cuts the read short, added pages are still reported (by the rule above) but removed ones are not. If more than half of a site (at least 50 URLs) vanishes at once, those removals are held back until the next run confirms them.
• Feeds instead of sitemaps only show recent items, so for feed-based sites only new pages are reported, never removals.
• Polite by design: one request at a time per site, at least 0.5 s apart (or the site's robots.txt Crawl-delay, up to 10 s). Pages are opened only where robots.txt allows. No proxies, no logins, no CAPTCHA solving: a site behind a bot wall is reported as failed, free of charge.
• Up to 100 sites per run, 500 sitemap files and 50,000 URLs per site by default (up to 200,000), 50 MB per sitemap file.

FAQ

Is it legal to monitor sitemaps?

Sitemaps and feeds are published by site owners for machines to read. The Actor reads only public, logged-out pages, follows robots.txt for every page it opens, and keeps no personal data. How you use the results is up to you and the terms of the sites you watch.

Why did my first run return no changes?

The first run for a site saves the baseline. Changes are measured between two runs, so schedule the task and the next run reports what changed.

Can I watch only part of a site?

Yes: add /blog/ (or any regular expression) under Only URLs matching, or give a specific sitemap URL such as https://example.com/post-sitemap.xml.

Can I get the full URL list too?

No. This Actor reports changes and per-section counts, not the full list. For a one-off full URL list, use a sitemap extractor.

Support

Found a site it cannot read, or need another field? Open an issue on the Issues tab with the site and the run link, and we will look at it.