Sitemap Change Monitor: New & Removed URLs

What does Sitemap Change Monitor do?

Sitemap Change Monitor keeps a URL inventory of any website from its sitemaps and tells you what changed since the last run: new pages, removed pages and pages whose lastmod changed, plus publishing-speed analytics. Other sitemap Actors only list URLs; this one remembers them, so every scheduled run turns into a diff.

For every site you get:

• 🗂️ URL count by section (the first path segment: /blog/, /products/, /en/ ...)
• 🆕 new URLs with their lastmod and the time they were first seen, ❌ removed URLs and 🔁 URLs whose lastmod changed (before and after)
• 📈 publishing statistics: new URLs per day and per week, top sections by new URLs, the median lag between a page's lastmod and the day it appeared in the sitemap, and the stale share (URLs whose lastmod is older than 12 months)
• 🔗 a link to the full list of changes of the run as a JSON record, when the lists in the row are cut

It finds sitemaps through robots.txt and /sitemap.xml, follows sitemap indexes, reads gzip (.xml.gz) files and plain text sitemaps. It fetches only sitemap files and robots.txt, never the pages themselves, respects the site's robots.txt (including Crawl-delay) and reads one file at a time. No proxy, no login, no API key.

Who is it for?

• Competitor content tracking: see which pages a competitor publishes, how fast, and in which section.
• SEO and content teams: spot removed pages (broken links, lost rankings), unexpected new sections, and stale content.
• E-commerce and catalogue monitoring: watch product and category URLs appear and disappear.
• News and publishing analysts: measure publishing velocity and how quickly the sitemap follows the article.
• Agencies and site owners who want a daily "what changed in the sitemap" report.

How to use it

1. Click Start with the default input: three sites with public sitemaps. The first run saves a baseline for each (nothing is reported as new yet).
2. Replace Sites or sitemap URLs with your own list (up to 200). Use a domain or site URL (example.com) to find its sitemaps automatically, or the URL of one sitemap file (https://example.com/sitemap-news.xml).
3. Optional: limit the inventory with Include only these paths (/blog/*) and Exclude these paths (/tag/*, *.pdf), and set Maximum URLs per site (default 50,000).
4. Schedule the Actor (for example daily). From the second run each site's row has new, removed and changed URLs and the publishing statistics. Connect the dataset to Slack, e-mail, a spreadsheet or a webhook.

Baselines live in a named key-value store of your account (compressed JSON), so runs of the same input find them. Give the monitor a name to keep history when you edit the site list. A run with Include unchanged sites off adds a row only for sites that changed; it is switched on in the example input only so a first try always shows rows.

Input example

{
    "sites": ["https://blog.apify.com", "example-shop.com", "https://news.example.org/sitemap-news.xml"],
    "maxUrlsPerSite": 50000,
    "includePaths": ["/blog/*", "/products/*"],
    "excludePaths": ["*/tag/*"],
    "includeUnchanged": false
}

Field	Meaning
sites	Up to 200 domains, site URLs or sitemap URLs.
maxUrlsPerSite	Default 50,000, at most 250,000 (the baseline of a site must fit in 9 MB compressed).
includePaths, excludePaths	Path patterns with * (/blog/*, */tag/*, *.pdf, /shop/$). Matched against the URL path from its start; exclude is applied after include.
maxListedUrls	Per site and category, how many new, removed and changed URLs are listed in the row (default 1,000). Counts are always complete; diffUrl has every URL.
maxSitemapsPerSite	Sitemap files read per site (default 100).
includeUnchanged	Add a row for sites without changes (default off).
monitorName	Names the baseline store.

Output example

Real output of a local run on 2 October 2026 with the default input (first run, baselines):

Site	Status	URLs	Sections	Sitemap files	Stale share (lastmod older than 12 months)
www.sitemaps.org	baseline	84	21	1	100 %
blog.apify.com	baseline	971	821	5 (an index and 4 files)	30 %
nodejs.org	baseline	1,740	4	2	0 %

A later run. To show a change without waiting for one, the saved baseline of blog.apify.com was edited locally (four posts removed from it, two made-up posts added, two lastmod values altered, tracking start moved back three days); the sitemap itself is the real one of 2 October 2026. The row (shortened to two sitemap files, three sections and two URLs per list):

{
    "site": "https://blog.apify.com",
    "status": "changed",
    "summary": "4 new, 2 removed and 2 URLs with a new lastmod; 971 URLs in total.",
    "checkedAt": "2026-10-02T05:59:34.760Z",
    "previousCheckedAt": "2026-09-29T05:59:15.739Z",
    "baselineAt": "2026-09-29T05:59:15.739Z",
    "robotsTxt": "read",
    "sitemaps": [
        { "url": "https://blog.apify.com/sitemap.xml", "status": "ok", "kind": "index", "urls": 4, "error": null },
        { "url": "https://blog.apify.com/sitemap-pages.xml", "status": "ok", "kind": "urlset", "urls": 1, "error": null }
    ],
    "sitemapsRead": 5,
    "sitemapsFailed": 0,
    "sitemapsSkipped": 0,
    "urlCount": 971,
    "truncated": false,
    "filters": { "includePaths": [], "excludePaths": [], "maxUrlsPerSite": 50000 },
    "sectionCount": 823,
    "sections": [
        { "section": "author", "sectionUrls": 116, "sectionNewUrls": 0, "sectionRemovedUrls": 0 },
        { "section": "tag", "sectionUrls": 36, "sectionNewUrls": 0, "sectionRemovedUrls": 0 },
        { "section": "ai-implementation-challenges", "sectionUrls": 1, "sectionNewUrls": 1, "sectionRemovedUrls": 0 }
    ],
    "newUrlsCount": 4,
    "removedUrlsCount": 2,
    "changedUrlsCount": 2,
    "newUrls": [
        { "url": "https://blog.apify.com/google-trends-compare-keywords/", "section": "google-trends-compare-keywords", "lastmod": "2026-10-01T11:53:55.000Z", "firstSeenAt": "2026-10-02T05:59:34.760Z" },
        { "url": "https://blog.apify.com/martina-gelnerova-data-career-change/", "section": "martina-gelnerova-data-career-change", "lastmod": "2026-10-01T08:46:24.000Z", "firstSeenAt": "2026-10-02T05:59:34.760Z" }
    ],
    "removedUrls": [
        { "url": "https://blog.apify.com/example-removed-post-one/", "section": "example-removed-post-one", "lastmod": "2025-05-04T10:00:00.000Z", "firstSeenAt": "2026-09-29T05:59:15.739Z" }
    ],
    "changedUrls": [
        { "url": "https://blog.apify.com/firecrawl-vs-apify/", "section": "firecrawl-vs-apify", "previousLastmod": "2026-01-01T00:00:00.000Z", "lastmod": "2026-09-30T17:54:31.000Z", "firstSeenAt": "2026-09-29T05:59:15.739Z" }
    ],
    "listsTruncated": false,
    "diffUrl": "https://api.apify.com/v2/key-value-stores/<store id>/records/diff-59249a68d59243762e936176-20261002055934",
    "publishing": {
        "trackingSince": "2026-09-29T05:59:15.739Z",
        "trackingDays": 3,
        "newUrlsSinceBaseline": 4,
        "newUrlsPerDay": 1.33,
        "newUrlsPerWeek": 9.33,
        "newUrlsByDay": [{ "date": "2026-09-29", "newUrls": 0 }, { "date": "2026-09-30", "newUrls": 0 }, { "date": "2026-10-01", "newUrls": 0 }, { "date": "2026-10-02", "newUrls": 4 }],
        "topSectionsByNewUrls": [{ "section": "ai-implementation-challenges", "newUrls": 1 }, { "section": "google-trends-compare-keywords", "newUrls": 1 }],
        "medianLagDays": 0.92,
        "lagSamples": 4,
        "urlsWithLastmod": 971,
        "urlsWithLastmodShare": 1,
        "staleShare": 0.2997,
        "staleUrls": 291,
        "modifiedLast7Days": 33,
        "modifiedLast30Days": 96,
        "newestLastmod": "2026-10-01T12:15:28.863Z",
        "oldestLastmod": "2021-10-27T19:00:27.000Z"
    },
    "notes": [],
    "error": null,
    "errorMessage": null
}

Reading the statistics

• New URLs are URLs listed now but not in the saved baseline. Their firstSeenAt is the time of the run that saw them first, so the resolution is your schedule: run daily and you know the day a page appeared.
• Median lag is the median number of days between a new URL's lastmod and the run that first saw it, over the new URLs since the baseline that have a lastmod. It says how fast the sitemap follows publishing (or, when the lag is large, that lastmod is not the publication time).
• Stale share is the share of URLs with a lastmod whose lastmod is more than 12 months old.
• Rates (newUrlsPerDay, newUrlsPerWeek) are new URLs since the baseline divided by the days tracked, and are null until a day has passed.
• Sections are the first path segment. Sites that publish posts at the root (/post-title/, like a blog on Ghost) get one section per post, so the section table is only useful for sites with a path structure.

How much does it cost?

This Actor uses pay per event:

Event	Price
Site checked (its sitemaps were read and compared with the baseline)	$0.004
Sites that fail (no sitemap found, unreachable, blocked, invalid entry)	free

• 100 sites cost $0.40; the default run (3 sites) costs $0.012.
• With Include unchanged sites off, a site that did not change is still checked and charged, but adds no row.
• Apify also charges a tiny standard start fee per run. Compute is small: a site with 50,000 URLs in a few files takes a minute or two. Large sites with many sitemap files are read one file at a time on purpose.
• Set Maximum cost per run in the run options and the Actor stops when it is reached.

Limitations

• Only what the sitemap lists. Pages that are not in a sitemap are not seen. A sitemap can be incomplete, stale or generated lazily; lastmod is whatever the site's software says (some sites put the crawl time there for every URL).
• Large sites are capped: 50,000 URLs by default (250,000 at most) and 100 sitemap files per site. A cut-short site is marked truncated, and removed URLs are not reported for it, because the missing URLs may simply not have been read.
• A sitemap file that fails to load (timeout, 5xx) does not make its URLs "removed": they are kept as they were and the row says how many files failed.
• robots.txt is respected: a sitemap under a disallowed path is not read (blocked-by-robots), and a Crawl-delay is honoured up to 10 seconds. Sites that block automated requests (403) come back as free error rows.
• First seen is the run time, not the real publication time. Use lastmod for that.
• A change of the path filters or the URL cap starts a new baseline for the site (reported as baseline), so settings changes never look like thousands of removed pages.
• URLs are compared as listed (query strings included); sitemap entries on other hosts are kept.

FAQ

Does it crawl the website?

No. It reads robots.txt and the sitemap files, nothing else. It never requests a page of the site.

How does it find the sitemap?

From the Sitemap: lines of robots.txt and from /sitemap.xml (then /sitemap_index.xml and /sitemap.xml.gz). If a site keeps its sitemap somewhere else, enter the sitemap URL itself. Sitemap indexes are followed up to four levels deep, and gzip files are unpacked.

What does the first run show?

A baseline: the URL count, sections and lastmod-based statistics. New, removed and changed URLs appear from the second run.

Can I watch only part of a site?

Yes: use the path filters (for example only /blog/*). Changing the filters starts a new baseline.

How do I get alerts?

Schedule the Actor, leave Include unchanged sites off, and connect the dataset or a webhook to Slack, e-mail or Zapier: a row appears only when something changed.

Does it store anything about me?

Only the public URL lists of the sites you monitor, in a named key-value store of your account. No personal data is processed.

More tools from the same developer

All pay-per-result, no proxy or login needed, built and maintained by the same developer:

Website audits

• Website Accessibility Checker: WCAG 2.2 & EAA: accessibility issues with fixes, SEO basics and security headers.
• Cookie & Tracker Audit: GDPR Consent Checker: cookies and tracking tags that load before consent.
• AI Crawler Access Checker: robots.txt & llms.txt: which AI crawlers a site allows, plus llms.txt.
• Structured Data Validator: Schema & Rich Results: JSON-LD and Microdata checked against Google's rich result rules.
• Website Change Monitor: Diffs, Prices & Alerts: get a row only when a page changes, with a clean diff.
• Policy Change Monitor: Privacy & Terms Tracker: privacy policy and terms changes, classified clause by clause.
• SaaS Pricing Monitor: Track Competitor Plans: competitor pricing plans as structured data, with price-change diffs.
• Site SEO Crawler: Broken Links & On-Page Audit: site-wide crawl for broken links, redirects and on-page SEO issues.
• Website Tech Stack Detector: Bulk Lead Filters: technologies used by websites, with filters for lead lists.
• Domain Availability & Expiry Checker (RDAP): domain availability, registrar and expiry dates from RDAP.
• Status Page SLA Monitor: Uptime & Incidents: SaaS vendor uptime, incidents and a monthly SLA table from public status pages.

Company data and compliance

• Company Contact Finder: Emails, Phones & Socials: contact details published on company websites.
• UK New Companies Feed: Companies House Daily: newly incorporated UK companies with sector filters.
• EU VAT Number Validator: Bulk VIES Checker: bulk VAT checks with name, address and consultation number.
• LEI Corporate Tree: GLEIF Parents & Subsidiaries: LEI lookup with parents, subsidiaries and a KYC summary.
• UK Company Distress Watch: Insolvency Alerts: administrations, liquidations and winding-up petitions from The Gazette.
• IBAN Validator: Bulk Check & Bank Lookup: IBAN format and checksum, plus bank name and BIC where open data exists.
• Sanctions Screening: OFAC, EU, UN & UK Lists: names screened against official sanctions lists, with match scores.
• OSHA & EPA Violation Monitor: Fines by Company: US OSHA and EPA enforcement cases and penalties by company.
• Email List Precheck: Syntax, MX & Disposable Check: cheap email list hygiene before a real verification.
• UK Company Officer Changes: Director & PSC Alerts: director, owner, address and status changes for a watchlist of UK companies.
• FDA Warning Letters Monitor: By Company & Subject: FDA warning letters by company and subject, with cited regulations.
• Supplier Risk Report: Sanctions, VAT & Insolvency: one risk row per supplier from eight official registers, with changes since last run.
• Brand Impersonation Domain Monitor: Look-alikes: look-alike domains and certificates that imitate your brand, with a risk score.

Media and AI

• Audio & Video Transcriber: Podcasts, SRT, Speakers: audio, video, podcasts and social links to text, SRT and VTT.
• Document to Audiobook: EPUB, PDF, DOCX to MP3: EPUB, PDF, DOCX, MOBI and 150+ formats to MP3 or M4B audiobooks with chapters.
• Podcast Mention Alerts: Brand Mentions by Second: where your brand is said in new podcast episodes, with the exact second.
• News Brand Mentions Monitor: New Articles Only: new news articles about your brand, people or topics.
• YouTube Channel Monitor: New Videos & Growth: new videos and view growth of the channels you watch.

Market signals

• US WARN Layoff Notices: 12 States Daily Feed: layoff and plant closure notices from official state sources.
• US Product Recalls Monitor: FDA & CPSC Feed: FDA and CPSC recalls in one feed, with severity.
• EU Tenders Feed: TED Public Procurement Alerts: EU public tenders from TED with CPV, country and value filters.
• Liquor License Openings: New Bars & Stores (US): new US liquor licences from official state open data.
• Clinical Trials Change Monitor: Status & Updates: status, enrollment and results changes on ClinicalTrials.gov studies.
• Career Site Jobs Monitor: ATS Jobs & New Roles: open roles from company career sites, with new and closed roles since the last run.
• App Reviews Monitor: App Store & Google Play: app reviews from both stores, only new ones in monitor mode.
• US Federal Contracts Feed: SAM.gov & USAspending: SAM.gov contract opportunities with the buyer's award history.
• Grants.gov Opportunities: US Federal Grants Feed: US federal grant opportunities by keyword, eligibility and agency.
• Federal Register Monitor: US Rules & Notices Feed: new US rules, proposed rules and notices by agency or topic.
• B2B Sales Trigger Feed: Hiring + Tech Leads: companies hiring for a role and using (or not using) a technology, with contacts.
• Local Business Leads: OpenStreetMap, No Website: local businesses by city and category, including those with no website.
• Solana Token Pair Monitor: New & Trending Pairs: new and trending Solana pairs with liquidity, volume and safety flags.

Social media

• Instagram Search Scraper: Find Accounts by Keyword: find Instagram accounts for any keyword or niche.
• Instagram Profile Scraper: Followers, Bio & Latest Posts: scrape public Instagram profiles.
• Instagram Hashtag Scraper: Top Posts, Views & Related Tags: get the top posts Instagram shows for any hashtag (shortcode, URL, type, full caption, play count, owner...
• LinkedIn Public Profile Scraper: Headline & Current Role: get the public headline data of LinkedIn profiles.
• TikTok Profile Scraper: Followers, Likes & Bio: get public TikTok profile data in bulk.
• Facebook Pages Search: Find Business Pages by Keyword: find Facebook business Pages for any keyword or local niche.
• TikTok Comments Scraper: Comments & Replies: scrape comments (and optionally replies) from TikTok videos.
• YouTube Transcript Scraper: Captions to Text: get the transcript of YouTube videos and Shorts.
• YouTube Comments Scraper: Text, Likes & Authors: scrape comments from YouTube videos and Shorts.
• YouTube Channel Search: Find Channels by Keyword: find YouTube channels for any keyword with YouTube's own Channels filter.
• TikTok Video Scraper: Views, Likes, Shares & Saves: get full data for TikTok videos by URL.
• YouTube Shorts Scraper: Channel Shorts & Views: get the Shorts of YouTube channels.
• Threads Search Scraper: Posts by Keyword or Hashtag: search Meta Threads for keywords or hashtags and get the posts.
• Instagram Posts & Reels Scraper: Latest Posts: get the 12 newest posts and reels of public Instagram accounts.
• Threads Replies Scraper: Comments on Threads Posts: get the replies under Meta Threads posts.
• YouTube Channel Videos Scraper: All Videos & Views: get every video of YouTube channels.
• Google Ads Transparency Scraper: Competitor Ads: get every Google ad of an advertiser or website from the Google Ads Transparency Center.
• Threads Profile Scraper: Followers, Bio & Latest Posts: scrape public Meta Threads profiles.
• Instagram Bio Email & Phone Scraper: get the emails and phone numbers Instagram accounts write in their public bio, plus their website, bio...
• Facebook Page Contact Scraper: Emails, Phones & Websites: get the contact details Facebook Pages publish.
• TikTok Sound Scraper: Song Usage & Top Videos: track TikTok sounds and songs: title, artist, total number of videos made with the sound, cover, and the...
• Telegram Channel Scraper: Posts, Views & History: scrape public Telegram channels.
• YouTube Channel Info & Email Scraper: About Page: get YouTube channel About data in bulk.
• Telegram Channel Keyword Search: Posts by Keyword: search public Telegram channels for keywords and get every matching post.
• Facebook Page Followers Checker: Bulk Follower Counts: check exact follower counts of Facebook Pages in bulk.

Jobs and talent

• LinkedIn Jobs Scraper: Descriptions, Criteria & Applicants: scrape LinkedIn job listings by keywords or a LinkedIn search URL.
• Wellfound Jobs Scraper: Startup Jobs & Salaries: scrape startup jobs from Wellfound (formerly AngelList Talent) by role, remote or by city.
• SEEK Jobs Scraper: Australia & New Zealand: scrape job ads from SEEK Australia and New Zealand by keywords and location.
• Naukrigulf Jobs Scraper: Gulf Jobs (UAE, Saudi, Qatar): scrape Naukrigulf job listings for the Gulf.
• Dice Jobs Scraper: US Tech Jobs & Salaries: scrape tech job listings from Dice.com by keyword and location.
• XING Jobs Scraper: Germany, Austria & Switzerland: scrape XING job listings (DACH region) by keyword and location.
• AI & ML Jobs Scraper: Data Science Roles: the newest ~100 AI, machine learning and data jobs (aijobs.net / Foorilla), optionally filtered by keyword.
• foundit Jobs Scraper: India (formerly Monster India): scrape job listings from foundit.in (formerly Monster India) by keyword and city.
• Malt Freelancers Scraper: Profiles & Day Rates: search Malt (France, Spain, Germany, Belgium, Netherlands...) for freelancers.
• Cybersecurity Jobs Scraper: InfoSec & Privacy Roles: the newest ~100 cybersecurity and privacy jobs (infosec-jobs.com / Foorilla), optionally filtered by keyword.

E-commerce and marketplaces

• Facebook Marketplace Scraper: Listings & Prices: search Facebook Marketplace in any city.
• AliExpress Search Scraper: Products, Prices & Sales: scrape AliExpress search results.
• TikTok Shop Scraper: Best-Selling Products by Category: scrape TikTok Shop category best-sellers.
• Whatnot Search Scraper: Listings, Bids & Sellers: search Whatnot (live shopping marketplace) listings.
• App Store Search Scraper: Apps, Ratings & Publishers: search the Apple App Store for apps or publishers.
• USA Scientific Lab Products Scraper: Prices & Specs: search USA Scientific's lab supply catalog.

Travel, sports and places

• Airbnb Search Scraper: Prices, Ratings & Listings: scrape Airbnb search results for any place and dates.
• Flashscore Scraper: Live Scores, Results & Fixtures: get Flashscore matches for 18 sports (football, tennis, basketball, hockey, cricket, esports...) for any...
• Google Hotels Prices Scraper: Lowest Rates by City: get Google Hotels "prices starting from" for any city and dates.
• Google Flights Scraper: Cheapest Fares by Route & Date: get Google Flights one-way fares for many routes and dates.
• PropertyPal Scraper: Northern Ireland Property Listings: scrape PropertyPal (Northern Ireland) listings for sale or rent.
• Live Flight Status by Area: Position & Ground Flag: live flight status inside a latitude/longitude box.
• Airbnb Listing Ratings Scraper: Scores & Review Themes: get the rating profile of Airbnb listings.
• Live Aircraft Tracker: ADS-B Positions in an Area: live aircraft positions inside a latitude/longitude box from OpenSky Network ADS-B receivers.
• Flights in an Area: Live Callsigns & Altitudes: flights currently in the air inside a latitude/longitude box (OpenSky ADS-B).

Finance and crypto

• Pump.fun New Tokens: Launches & Migrations: newest pump.fun Solana token launches.
• Bybit Funding Rates & Open Interest: Bybit perpetual futures data per symbol.
• Ethereum ETF Flows: Daily Net Inflows by Fund: daily net flows of US spot Ether ETFs (ETHA, FETH, ETHE...).
• Bitcoin ETF Flows: Daily Net Inflows by Fund: daily net flows of US spot Bitcoin ETFs (IBIT, FBTC, GBTC, ARKB...).
• California Cap-and-Trade Auction Results: California-Quebec cap-and-trade auction results from CARB.

Research and public data

• LinkedIn Company Scraper: Size, Industry & HQ: get public LinkedIn company page data in bulk.
• Google News Scraper: Headlines, Sources & Real URLs: search Google News for any query, language and country.
• Indie Hackers Posts Scraper: Trending & Newest: get Indie Hackers posts (trending, featured, newest or throwbacks).
• UNGM Tenders Scraper: UN Procurement Notices: search United Nations Global Marketplace (UNGM) procurement notices.
• Global Patent Search: US, EP, WO & More: search patents worldwide by keywords, assignee, inventor, country and dates.
• AlienVault OTX Threat Lookup: IPs, Domains & Hashes: look up IPs, domains, URLs or file hashes in AlienVault OTX threat intelligence.
• China Patents Search (CNIPA): Titles, Assignees & Dates: search Chinese patents (CNIPA) by keywords, assignee or inventor.
• California License Lookup: Doctors, Nurses & 33 Boards (DCA): look up California professional licenses from the Department of Consumer Affairs' official monthly licensee...

Feedback

Missing a sitemap format or a statistic you need? Open an issue on the Issues tab.