Sitemap Change Monitor: New & Removed URLs

What does Sitemap Change Monitor do?

Sitemap Change Monitor keeps a URL inventory of any website from its sitemaps and tells you what changed since the last run: new pages, removed pages and pages whose lastmod changed, plus publishing-speed analytics. Other sitemap Actors only list URLs; this one remembers them, so every scheduled run turns into a diff.

For every site you get:

• 🗂️ URL count by section (the first path segment: /blog/, /products/, /en/ ...)
• 🆕 new URLs with their lastmod and the time they were first seen, ❌ removed URLs and 🔁 URLs whose lastmod changed (before and after)
• 📈 publishing statistics: new URLs per day and per week, top sections by new URLs, the median lag between a page's lastmod and the day it appeared in the sitemap, and the stale share (URLs whose lastmod is older than 12 months)
• 🔗 a link to the full list of changes of the run as a JSON record, when the lists in the row are cut

It finds sitemaps through robots.txt and /sitemap.xml, follows sitemap indexes, reads gzip (.xml.gz) files and plain text sitemaps. It fetches only sitemap files and robots.txt, never the pages themselves, respects the site's robots.txt (including Crawl-delay) and reads one file at a time. No proxy, no login, no API key.

Who is it for?

• Competitor content tracking: see which pages a competitor publishes, how fast, and in which section.
• SEO and content teams: spot removed pages (broken links, lost rankings), unexpected new sections, and stale content.
• E-commerce and catalogue monitoring: watch product and category URLs appear and disappear.
• News and publishing analysts: measure publishing velocity and how quickly the sitemap follows the article.
• Agencies and site owners who want a daily "what changed in the sitemap" report.

How to use it

1. Click Start with the default input: three sites with public sitemaps. The first run saves a baseline for each (nothing is reported as new yet).
2. Replace Sites or sitemap URLs with your own list (up to 200). Use a domain or site URL (example.com) to find its sitemaps automatically, or the URL of one sitemap file (https://example.com/sitemap-news.xml).
3. Optional: limit the inventory with Include only these paths (/blog/*) and Exclude these paths (/tag/*, *.pdf), and set Maximum URLs per site (default 50,000).
4. Schedule the Actor (for example daily). From the second run each site's row has new, removed and changed URLs and the publishing statistics. Connect the dataset to Slack, e-mail, a spreadsheet or a webhook.

Baselines live in a named key-value store of your account (compressed JSON), so runs of the same input find them. Give the monitor a name to keep history when you edit the site list. A run with Include unchanged sites off adds a row only for sites that changed; it is switched on in the example input only so a first try always shows rows.

Input example

{
    "sites": ["https://blog.apify.com", "example-shop.com", "https://news.example.org/sitemap-news.xml"],
    "maxUrlsPerSite": 50000,
    "includePaths": ["/blog/*", "/products/*"],
    "excludePaths": ["*/tag/*"],
    "includeUnchanged": false
}

Field	Meaning
sites	Up to 200 domains, site URLs or sitemap URLs.
maxUrlsPerSite	Default 50,000, at most 250,000 (the baseline of a site must fit in 9 MB compressed).
includePaths, excludePaths	Path patterns with * (/blog/*, */tag/*, *.pdf, /shop/$). Matched against the URL path from its start; exclude is applied after include.
maxListedUrls	Per site and category, how many new, removed and changed URLs are listed in the row (default 1,000). Counts are always complete; diffUrl has every URL.
maxSitemapsPerSite	Sitemap files read per site (default 100).
includeUnchanged	Add a row for sites without changes (default off).
monitorName	Names the baseline store.

Output example

Real output of a local run on 2 October 2026 with the default input (first run, baselines):

Site	Status	URLs	Sections	Sitemap files	Stale share (lastmod older than 12 months)
www.sitemaps.org	baseline	84	21	1	100 %
blog.apify.com	baseline	971	821	5 (an index and 4 files)	30 %
nodejs.org	baseline	1,740	4	2	0 %

A later run. To show a change without waiting for one, the saved baseline of blog.apify.com was edited locally (four posts removed from it, two made-up posts added, two lastmod values altered, tracking start moved back three days); the sitemap itself is the real one of 2 October 2026. The row (shortened to two sitemap files, three sections and two URLs per list):

{
    "site": "https://blog.apify.com",
    "status": "changed",
    "summary": "4 new, 2 removed and 2 URLs with a new lastmod; 971 URLs in total.",
    "checkedAt": "2026-10-02T05:59:34.760Z",
    "previousCheckedAt": "2026-09-29T05:59:15.739Z",
    "baselineAt": "2026-09-29T05:59:15.739Z",
    "robotsTxt": "read",
    "sitemaps": [
        { "url": "https://blog.apify.com/sitemap.xml", "status": "ok", "kind": "index", "urls": 4, "error": null },
        { "url": "https://blog.apify.com/sitemap-pages.xml", "status": "ok", "kind": "urlset", "urls": 1, "error": null }
    ],
    "sitemapsRead": 5,
    "sitemapsFailed": 0,
    "sitemapsSkipped": 0,
    "urlCount": 971,
    "truncated": false,
    "filters": { "includePaths": [], "excludePaths": [], "maxUrlsPerSite": 50000 },
    "sectionCount": 823,
    "sections": [
        { "section": "author", "sectionUrls": 116, "sectionNewUrls": 0, "sectionRemovedUrls": 0 },
        { "section": "tag", "sectionUrls": 36, "sectionNewUrls": 0, "sectionRemovedUrls": 0 },
        { "section": "ai-implementation-challenges", "sectionUrls": 1, "sectionNewUrls": 1, "sectionRemovedUrls": 0 }
    ],
    "newUrlsCount": 4,
    "removedUrlsCount": 2,
    "changedUrlsCount": 2,
    "newUrls": [
        { "url": "https://blog.apify.com/google-trends-compare-keywords/", "section": "google-trends-compare-keywords", "lastmod": "2026-10-01T11:53:55.000Z", "firstSeenAt": "2026-10-02T05:59:34.760Z" },
        { "url": "https://blog.apify.com/martina-gelnerova-data-career-change/", "section": "martina-gelnerova-data-career-change", "lastmod": "2026-10-01T08:46:24.000Z", "firstSeenAt": "2026-10-02T05:59:34.760Z" }
    ],
    "removedUrls": [
        { "url": "https://blog.apify.com/example-removed-post-one/", "section": "example-removed-post-one", "lastmod": "2025-05-04T10:00:00.000Z", "firstSeenAt": "2026-09-29T05:59:15.739Z" }
    ],
    "changedUrls": [
        { "url": "https://blog.apify.com/firecrawl-vs-apify/", "section": "firecrawl-vs-apify", "previousLastmod": "2026-01-01T00:00:00.000Z", "lastmod": "2026-09-30T17:54:31.000Z", "firstSeenAt": "2026-09-29T05:59:15.739Z" }
    ],
    "listsTruncated": false,
    "diffUrl": "https://api.apify.com/v2/key-value-stores/<store id>/records/diff-59249a68d59243762e936176-20261002055934",
    "publishing": {
        "trackingSince": "2026-09-29T05:59:15.739Z",
        "trackingDays": 3,
        "newUrlsSinceBaseline": 4,
        "newUrlsPerDay": 1.33,
        "newUrlsPerWeek": 9.33,
        "newUrlsByDay": [{ "date": "2026-09-29", "newUrls": 0 }, { "date": "2026-09-30", "newUrls": 0 }, { "date": "2026-10-01", "newUrls": 0 }, { "date": "2026-10-02", "newUrls": 4 }],
        "topSectionsByNewUrls": [{ "section": "ai-implementation-challenges", "newUrls": 1 }, { "section": "google-trends-compare-keywords", "newUrls": 1 }],
        "medianLagDays": 0.92,
        "lagSamples": 4,
        "urlsWithLastmod": 971,
        "urlsWithLastmodShare": 1,
        "staleShare": 0.2997,
        "staleUrls": 291,
        "modifiedLast7Days": 33,
        "modifiedLast30Days": 96,
        "newestLastmod": "2026-10-01T12:15:28.863Z",
        "oldestLastmod": "2021-10-27T19:00:27.000Z"
    },
    "notes": [],
    "error": null,
    "errorMessage": null
}

Reading the statistics

• New URLs are URLs listed now but not in the saved baseline. Their firstSeenAt is the time of the run that saw them first, so the resolution is your schedule: run daily and you know the day a page appeared.
• Median lag is the median number of days between a new URL's lastmod and the run that first saw it, over the new URLs since the baseline that have a lastmod. It says how fast the sitemap follows publishing (or, when the lag is large, that lastmod is not the publication time).
• Stale share is the share of URLs with a lastmod whose lastmod is more than 12 months old.
• Rates (newUrlsPerDay, newUrlsPerWeek) are new URLs since the baseline divided by the days tracked, and are null until a day has passed.
• Sections are the first path segment. Sites that publish posts at the root (/post-title/, like a blog on Ghost) get one section per post, so the section table is only useful for sites with a path structure.

How much does it cost?

This Actor uses pay per event:

Event	Price
Site checked (its sitemaps were read and compared with the baseline)	$0.004
Sites that fail (no sitemap found, unreachable, blocked, invalid entry)	free

• 100 sites cost $0.40; the default run (3 sites) costs $0.012.
• With Include unchanged sites off, a site that did not change is still checked and charged, but adds no row.
• Apify also charges a tiny standard start fee per run. Compute is small: a site with 50,000 URLs in a few files takes a minute or two. Large sites with many sitemap files are read one file at a time on purpose.
• Set Maximum cost per run in the run options and the Actor stops when it is reached.

Limitations

• Only what the sitemap lists. Pages that are not in a sitemap are not seen. A sitemap can be incomplete, stale or generated lazily; lastmod is whatever the site's software says (some sites put the crawl time there for every URL).
• Large sites are capped: 50,000 URLs by default (250,000 at most) and 100 sitemap files per site. A cut-short site is marked truncated, and removed URLs are not reported for it, because the missing URLs may simply not have been read.
• A sitemap file that fails to load (timeout, 5xx) does not make its URLs "removed": they are kept as they were and the row says how many files failed.
• robots.txt is respected: a sitemap under a disallowed path is not read (blocked-by-robots), and a Crawl-delay is honoured up to 10 seconds. Sites that block automated requests (403) come back as free error rows.
• First seen is the run time, not the real publication time. Use lastmod for that.
• A change of the path filters or the URL cap starts a new baseline for the site (reported as baseline), so settings changes never look like thousands of removed pages.
• URLs are compared as listed (query strings included); sitemap entries on other hosts are kept.

FAQ

Does it crawl the website?

No. It reads robots.txt and the sitemap files, nothing else. It never requests a page of the site.

How does it find the sitemap?

From the Sitemap: lines of robots.txt and from /sitemap.xml (then /sitemap_index.xml and /sitemap.xml.gz). If a site keeps its sitemap somewhere else, enter the sitemap URL itself. Sitemap indexes are followed up to four levels deep, and gzip files are unpacked.

What does the first run show?

A baseline: the URL count, sections and lastmod-based statistics. New, removed and changed URLs appear from the second run.

Can I watch only part of a site?

Yes: use the path filters (for example only /blog/*). Changing the filters starts a new baseline.

How do I get alerts?

Schedule the Actor, leave Include unchanged sites off, and connect the dataset or a webhook to Slack, e-mail or Zapier: a row appears only when something changed.

Does it store anything about me?

Only the public URL lists of the sites you monitor, in a named key-value store of your account. No personal data is processed.

More tools from the same developer

All pay-per-result, no proxy or login needed, built and maintained by the same developer:

Website audits

• Website Accessibility Checker: WCAG 2.2 & EAA: accessibility issues with fixes, SEO basics and security headers.
• Cookie & Tracker Audit: GDPR Consent Checker: cookies and tracking tags that load before consent.
• AI Crawler Access Checker: robots.txt & llms.txt: which AI crawlers a site allows, plus llms.txt.
• Structured Data Validator: Schema & Rich Results: JSON-LD and Microdata checked against Google's rich result rules.
• Website Change Monitor: Diffs, Prices & Alerts: get a row only when a page changes, with a clean diff.
• Policy Change Monitor: Privacy & Terms Tracker: privacy policy and terms changes, classified clause by clause.
• SaaS Pricing Monitor: Track Competitor Plans: competitor pricing plans as structured data, with price-change diffs.
• Site SEO Crawler: Broken Links & On-Page Audit: site-wide crawl for broken links, redirects and on-page SEO issues.
• Website Tech Stack Detector: Bulk Lead Filters: technologies used by websites, with filters for lead lists.
• Domain Availability & Expiry Checker (RDAP): domain availability, registrar and expiry dates from RDAP.
• Status Page SLA Monitor: Uptime & Incidents: SaaS vendor uptime, incidents and a monthly SLA table from public status pages.

Company data and compliance

• Company Contact Finder: Emails, Phones & Socials: contact details published on company websites.
• UK New Companies Feed: Companies House Daily: newly incorporated UK companies with sector filters.
• EU VAT Number Validator: Bulk VIES Checker: bulk VAT checks with name, address and consultation number.
• LEI Corporate Tree: GLEIF Parents & Subsidiaries: LEI lookup with parents, subsidiaries and a KYC summary.
• UK Company Distress Watch: Insolvency Alerts: administrations, liquidations and winding-up petitions from The Gazette.
• IBAN Validator: Bulk Check & Bank Lookup: IBAN format and checksum, plus bank name and BIC where open data exists.
• Sanctions Screening: OFAC, EU, UN & UK Lists: names screened against official sanctions lists, with match scores.
• OSHA & EPA Violation Monitor: Fines by Company: US OSHA and EPA enforcement cases and penalties by company.
• Email List Precheck: Syntax, MX & Disposable Check: cheap email list hygiene before a real verification.
• UK Company Officer Changes: Director & PSC Alerts: director, owner, address and status changes for a watchlist of UK companies.
• FDA Warning Letters Monitor: By Company & Subject: FDA warning letters by company and subject, with cited regulations.
• Supplier Risk Report: Sanctions, VAT & Insolvency: one risk row per supplier from eight official registers, with changes since last run.
• Brand Impersonation Domain Monitor: Look-alikes: look-alike domains and certificates that imitate your brand, with a risk score.

Media and AI

• Audio & Video Transcriber: Podcasts, SRT, Speakers: audio, video, podcasts and social links to text, SRT and VTT.
• Podcast Mention Alerts: Brand Mentions by Second: where your brand is said in new podcast episodes, with the exact second.
• News Brand Mentions Monitor: New Articles Only: new news articles about your brand, people or topics.
• YouTube Channel Monitor: New Videos & Growth: new videos and view growth of the channels you watch.

Market signals

• US WARN Layoff Notices: 12 States Daily Feed: layoff and plant closure notices from official state sources.
• US Product Recalls Monitor: FDA & CPSC Feed: FDA and CPSC recalls in one feed, with severity.
• EU Tenders Feed: TED Public Procurement Alerts: EU public tenders from TED with CPV, country and value filters.
• Liquor License Openings: New Bars & Stores (US): new US liquor licences from official state open data.
• Clinical Trials Change Monitor: Status & Updates: status, enrollment and results changes on ClinicalTrials.gov studies.
• Career Site Jobs Monitor: ATS Jobs & New Roles: open roles from company career sites, with new and closed roles since the last run.
• App Reviews Monitor: App Store & Google Play: app reviews from both stores, only new ones in monitor mode.
• US Federal Contracts Feed: SAM.gov & USAspending: SAM.gov contract opportunities with the buyer's award history.
• Grants.gov Opportunities: US Federal Grants Feed: US federal grant opportunities by keyword, eligibility and agency.
• Federal Register Monitor: US Rules & Notices Feed: new US rules, proposed rules and notices by agency or topic.
• B2B Sales Trigger Feed: Hiring + Tech Leads: companies hiring for a role and using (or not using) a technology, with contacts.
• Local Business Leads: OpenStreetMap, No Website: local businesses by city and category, including those with no website.
• Solana Token Pair Monitor: New & Trending Pairs: new and trending Solana pairs with liquidity, volume and safety flags.

Feedback

Missing a sitemap format or a statistic you need? Open an issue on the Issues tab.