Sitemap URL Extractor & New Page Monitor

Get every URL a website publishes in its XML sitemaps, as clean rows with lastmod, changefreq, priority, hreflang alternates, images and Google News fields. Give it a domain and it finds the sitemaps for you. Turn on monitor mode to get only the pages that are new since the last run, or status check to find 404s and redirects listed in sitemaps.

• Finds sitemaps automatically: reads Sitemap: lines in robots.txt, then tries /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and /sitemaps.xml.
• Reads every format: sitemap indexes (nested to any depth), gzipped .xml.gz, plain-text sitemaps, RSS and Atom feeds, image, video and Google News sitemaps.
• Light and fast: plain HTTP requests, no browser. A test run read 1,010 URLs from 3 sites in 3 seconds at 256 MB.
• Doesn't fail the whole run on one bad file: each sitemap is retried with backoff. Broken or blocked files are listed in the per-site summary while the rest of the site is still extracted.
• Pay only for URLs you get. No start fee, and the monitor's baseline run is free.

Use cases

• SEO audits: a full URL inventory with lastmod dates. Find stale pages, missing hreflang, and 404s or redirects that shouldn't be in the sitemap.
• Competitor monitoring: schedule a daily run in monitor mode and get each new product, landing page, blog post or job page a competitor publishes, the day it appears.
• Crawl seeding: feed the URL list into Website Content Crawler, a RAG pipeline or your own scraper, so it crawls only real, listed pages.
• Content migration: export every URL before a redesign and check the redirects afterwards.
• News tracking: read Google News sitemaps (news:title, news:publication_date) and RSS feeds as one table.

Input

Field	What it does	Default
Websites or sitemap URLs	example.com, a homepage, or a direct sitemap/feed URL. One per line, as many sites as you like.	required
Maximum URLs per site	Stop reading a site after this many unique URLs.	10,000
Only URLs matching (regex)	Keep URLs that match any pattern, e.g. /blog/, /products/	none
Skip URLs matching (regex)	Drop URLs that match any pattern, e.g. /tag/, /page/\d+	none
Modified on or after	Keep URLs whose lastmod is on or after a date (2026-09-01).	none
Monitor: return only new URLs	First run saves a baseline (no URLs charged). Later runs return only URLs that weren't there before.	off
Check HTTP status of each URL	HEAD request (GET fallback) per URL: status code, redirect, final URL.	off

Example: full URL list of a site

{
  "startUrls": ["apify.com", "https://blog.apify.com/sitemap.xml"],
  "maxUrlsPerSite": 50000
}

Example: daily "new pages" monitor for competitors

{
  "startUrls": ["competitor-one.com", "competitor-two.com"],
  "onlyNewUrls": true,
  "excludePatterns": ["/tag/", "/author/"]
}

Schedule this daily in Apify (Schedules → Create) and connect the dataset to Slack, email, Google Sheets or a webhook. Each run returns only URLs that appeared since the previous run.

Example: find broken URLs in your own sitemap

{
  "startUrls": ["https://www.example.com/sitemap_index.xml"],
  "checkStatus": true,
  "statusCheckConcurrency": 5
}

Output

One row per URL. Export it as JSON, CSV, Excel or HTML, or read it through the API.

{
  "url": "https://www.nytimes.com/live/2026/10/08/us/midterms-elections-debates",
  "site": "https://www.nytimes.com",
  "sitemapUrl": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
  "lastmod": "2026-10-08T22:03:22Z",
  "changefreq": null,
  "priority": null,
  "alternates": [],
  "imageCount": 1,
  "images": ["https://static01.nyt.com/images/2026/10/08/multimedia/08midterms-promo6p-lphw/08midterms-promo6p-lphw-articleLarge.jpg"],
  "newsTitle": "Midterm Elections Updates: Senate Debates to Be Held in Michigan, Maine and Georgia",
  "newsPublicationDate": "2026-10-08T12:46:34Z",
  "videoTitles": []
}

With status check on, each row also gets statusCode, finalUrl, redirected and statusError. In monitor mode, each row has isNew: true.

A per-site summary (sitemaps found and read, URLs found, files that failed and why) is saved in the run's key-value store as SUMMARY.

Pricing

Pay per event, with no start fee:

Event	Price
URL returned	$0.0002 (20¢ per 1,000 URLs)
HTTP status check (only if you turn it on)	$0.0002 per URL

A 5,000-page site costs $1.00, or $2.00 with status checks. In monitor mode you pay only for the new URLs each run returns, so a daily check of a site that adds 10 pages a day costs about $0.06 a month. Set Maximum URLs per site or a maximum charge per run to cap your spend.

Tips and limits

• A domain input finds sitemaps for the whole host. To read just one section, give the section's sitemap URL directly.
• If a site has no sitemap at all, the summary says so. This actor doesn't crawl links. Use Website Content Crawler for sites without sitemaps.
• Some sites block automated requests to their sitemaps (HTTP 403). The actor reports this instead of trying to get around it. For your own site, you can set a custom User-Agent that your firewall allows.
• Monitor memory is stored per site in a named key-value store in your account (sitemap-extractor-state). Use a different Monitor state store name for each separate monitor, and delete the store to reset the baseline.
• If Maximum URLs per site is lower than the site's real size, monitor mode only compares that slice. Set the cap above the site's size for reliable new-page detection.

FAQ

Is it legal to read sitemaps? Sitemaps and robots.txt are public files that sites publish so automated tools can find their pages. The actor reads only those files and, if you ask, sends one HEAD request per URL. It doesn't log in, solve CAPTCHAs or collect personal data. You're responsible for how you use the URLs.

Why did a site return 0 URLs? Check SUMMARY in the key-value store. The usual causes are no sitemap, a 403 block, or filters that removed every URL.

Can I use it from an AI agent or via API? Yes. Call it through the Apify API or MCP server with startUrls. It returns structured rows and needs no browser.

More from open-data-tools

• USAspending Federal Awards Search: US federal contracts, grants and loans from the official USAspending.gov API.

Questions or a sitemap format that doesn't parse? Open an issue on this actor's Issues tab and include the sitemap URL.