Sitemap Extractor & New Page Monitor

Get every page URL a website publishes in its sitemaps, and find out which pages are new, updated or removed since your last run. Give it a domain or a sitemap URL. It finds the sitemaps itself (robots.txt, sitemap.xml, sitemap_index.xml, WordPress, gzip and more), follows nested sitemap indexes and returns clean rows with lastmod, section, images, hreflang and Google News data. You can add an HTTP status check per URL and get a sitemap health report with SEO issues.

No login, no API key, no proxies. It only reads the files the website publishes for crawlers.

What you can use it for

• Competitor monitoring: schedule it daily on competitors' sites and get only the new product pages, blog posts, landing pages or pricing pages they added.
• SEO audits and migrations: export the full URL inventory, check that every sitemap URL answers 200 and isn't redirected or noindexed, and compare the URL list before and after a migration.
• Content tracking: follow news sitemaps and see new articles with title and publication date within minutes of publishing.
• Crawl seeding: feed the URL list into a scraper (for example Website Content Crawler) so you scrape only pages that changed since the last run.
• Lead and market research: count how many products, locations or job pages a company has, and how fast that changes.

Input

Field	What it does
websites	Domains, sections or sitemap URLs, one per line. allbirds.com finds all sitemaps on the site. wordpress.org/news keeps only URLs under /news, uses the section's own sitemap when it has one, and keeps its own change history. https://techcrunch.com/news-sitemap.xml reads only that sitemap and its children.
outputMode	urls (default): one row per URL, marked new, updated or unchanged since the last run. changes: only new, updated and removed pages. summary: one health report per website.
includePatterns / excludePatterns	Keep or drop URLs containing a text such as /blog/, or matching a * pattern against the whole URL. Case-insensitive.
lastmodWithinDays	Keep only URLs modified (or published, for news) in the last N days. 0 keeps everything.
trackChanges / stateKey	Remember URLs between runs (on by default). Use a tracking key to keep separate histories, for example one per client.
checkStatus / maxStatusChecksPerSite	Add-on: request each output URL and add the status code, redirect chain, final URL, X-Robots-Tag and response time.
maxUrlsPerSite / maxSitemapsPerSite	Safety limits. Default 20,000 URLs and 500 sitemap files per website.

Example: monitor new product and blog pages on two shops.

{
    "websites": ["allbirds.com", "gymshark.com"],
    "outputMode": "changes",
    "includePatterns": ["/products/", "/blogs/"]
}

The first changes run saves a baseline for free. Every later run returns only what changed.

Output

Every URL row (type: "url"):

{
    "type": "url",
    "website": "allbirds.com",
    "url": "https://www.allbirds.com/products/mens-tree-runners",
    "changeType": "new",
    "lastmod": "2026-10-30T03:38:19.000Z",
    "previousLastmod": null,
    "changefreq": "daily",
    "priority": null,
    "section": "/products",
    "imageCount": 1,
    "videoCount": 0,
    "hreflangCount": 0,
    "alternates": [],
    "newsTitle": null,
    "newsPublishedAt": null,
    "sitemap": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
    "scannedAt": "2026-10-30T08:39:57.961Z"
}

With the status check on, rows also get statusCode, finalUrl, redirectCount, redirectChain, xRobotsTag, responseTimeMs and checkError.

changeType is null on the first run of a website, then new, updated (lastmod changed), unchanged or removed (changes mode only).

Each website also gets one row (type: "site") with its status (ok, partial, not_found, error), how the sitemaps were found, the number of sitemaps and URLs, and the change counts. In summary mode that row is the full report:

• duplicate and invalid URLs, share of URLs with a lastmod, newest and oldest lastmod;
• pages updated in the last 7, 30 and 90 days;
• hreflang, image and news URL counts;
• the top 20 sections (first path segment) by URL count;
• every sitemap file read, with its status and size;
• SEO issues with a code, a count, an explanation and an example.

Issue codes include SITEMAP_NOT_IN_ROBOTS_TXT, SITEMAP_FETCH_FAILED, INVALID_SITEMAP, SITEMAP_TOO_LARGE, EMPTY_SITEMAP, DUPLICATE_URLS, INVALID_LASTMOD, FUTURE_LASTMOD, UNIFORM_LASTMOD, CROSS_HOST_URLS, HTTP_URLS_ON_HTTPS, BROKEN_URLS, REDIRECTING_URLS, NOINDEX_URLS and UNREACHABLE_URLS.

The dataset has views for URLs, Changes, Status checks and Websites. A RUN_SUMMARY record in the key-value store has totals for the run.

Supported formats

• XML sitemaps and sitemap indexes, nested up to 5 levels;
• gzip (.xml.gz), plain-text sitemaps and RSS/Atom feeds;
• image, video, hreflang (xhtml:link) and Google News extensions;
• WordPress, Yoast, Rank Math, Shopify, Squarespace, Wix, Webflow and custom generators.

Some generators stamp the same build time on every URL in a file. The Actor detects this and ignores those dates, so a regenerated sitemap doesn't mark every page as updated. It reports it as UNIFORM_LASTMOD instead.

Pricing

Pay per event, plus a tiny start fee:

Event	Price	When
Sitemap URL	$0.0002 ($0.20 per 1,000)	each URL row in urls mode, or each new, updated or removed page in changes mode
Status-checked URL	$0.0008 ($0.80 per 1,000)	add-on, each URL checked for its HTTP status
Website sitemap report	$0.005	each website in summary mode

You don't pay for:

• the baseline on the first changes run, or unchanged pages in later changes runs;
• website rows in urls and changes mode;
• websites without a sitemap, unreachable websites and invalid input;
• URLs skipped by your filters, and status checks blocked by robots.txt.

Monitoring a site daily in changes mode usually costs a few cents a month. Set a maximum charge on the run if you want a hard ceiling. The Actor then stops cleanly and doesn't save a partial history.

How it works and what it respects

The Actor reads robots.txt and the sitemap files a website publishes for search engines. It doesn't crawl pages or follow links. Requests to the same host are paced, and it identifies itself as SitemapPageMonitor in the user agent. If robots.txt disallows that agent from a sitemap, the sitemap is skipped. The optional status check sends one HEAD request per URL (GET when HEAD isn't supported) and also respects robots.txt.

Limits

• Only URLs listed in sitemaps are found. Pages a site leaves out of its sitemap won't appear.
• updated depends on the site's lastmod values. Sites without lastmod only report new and removed pages.
• Removed pages are only reported when the whole sitemap was read. If maxUrlsPerSite or maxSitemapsPerSite cut the read short, removals are skipped and the site row says so.
• Some large sites block requests from cloud servers. Their sitemaps then come back as not_found with SITEMAP_FETCH_FAILED and the HTTP status, at no charge.
• Change history is kept for up to 300,000 URLs per website and tracking key.

Support

Found a sitemap the Actor can't read, or need another field? Open an issue on the Actor page with the URL, and it will usually be fixed in the next version.