Sitemap URL Extractor: Get All URLs of a Website & Monitor New Pages

Extract every page URL of a website from its XML sitemaps. Enter a homepage and the Actor finds the sitemaps through robots.txt and common paths, follows nested sitemap indexes, reads gzip (.xml.gz), text and RSS sitemaps, and returns each URL with lastmod, changefreq, priority, hreflang alternates, image URLs and Google News title and date. Filter by URL pattern or modification date, or switch on Only new URLs to get pages added since the last run. It's fast, with no browser and no crawling.

• ✅ Sitemap auto-discovery (robots.txt Sitemap: lines, /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and more), or give a sitemap URL directly
• ✅ Nested sitemap indexes, .xml.gz, text and RSS/Atom sitemaps, and a broken-XML tolerant parser
• ✅ Include/exclude regex patterns (such as /blog/ or /product/) and a modified after date filter
• ✅ Follow only matching child sitemaps (such as post or product) to save time on huge sites
• ✅ hreflang alternates, images, Google News and video flags
• ✅ Only new URLs monitoring: track competitors' new pages, products and posts. You pay only for new URLs.

Input examples

First 100 URLs of a website (default):

{
  "startUrls": [
    "https://apify.com"
  ],
  "maxUrlsPerSite": 100
}

All blog posts changed since 1 October 2026:

{
  "startUrls": [
    "https://www.theverge.com"
  ],
  "includePatterns": [
    "/tech/",
    "/news/"
  ],
  "lastmodAfter": "2026-10-01",
  "maxUrlsPerSite": 5000
}

Monitor new product pages of competitors (schedule it daily):

{
  "startUrls": [
    "https://shop-one.example",
    "https://shop-two.example/sitemap_products_1.xml"
  ],
  "includePatterns": [
    "/products?/"
  ],
  "maxUrlsPerSite": 100000,
  "onlyNewUrls": true,
  "stateStoreName": "competitor-products"
}

Output example (real run)

{
  "site": "theverge.com",
  "url": "https://www.theverge.com/tech/1017066/meta-muse-sharing-back-on-artifiacts",
  "lastmod": "2026-10-22T20:24:04Z",
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://www.theverge.com/sitemaps/entries/2026/10",
  "path": "/tech/1017066/meta-muse-sharing-back-on-artifiacts",
  "alternates": [],
  "images": [
    "https://platform.theverge.com/wp-content/uploads/sites/2/2026/09/STKB394_MUSE_AI_CVIRGINIA_C.png"
  ],
  "news_title": null,
  "news_publication_date": null,
  "has_video": false,
  "scraped_at": "2026-10-25T19:06:40Z"
}

Use the URLs or URLs with hreflang, images & news views, or export to JSON, CSV or Excel. The OUTPUT record has per-site counts and errors (for example, when a site has no sitemap).

Pricing

Pay per event, no subscription:

Event	Price
URL	$0.0002 ($0.20 per 1,000)
Actor start (per run)	$0.002

The default run (100 URLs) costs about $0.022. 100,000 URLs cost $20. Filtered and already-seen URLs are never charged.

Use cases

• SEO audits: the full list of indexable URLs, lastmod dates and hreflang coverage.
• Competitor monitoring: new products, landing pages, blog posts and job pages as soon as they reach the sitemap.
• Crawl seeding: feed the URL list to a web scraper, a link checker or an AI crawler instead of crawling blindly.
• Content inventory and migrations: export every URL of a site to a spreadsheet.
• News tracking: Google News sitemaps with titles and publication dates.

FAQ

What if a site has no sitemap? The site is reported in OUTPUT.errors with "No sitemap found", and the other sites still run. You can also enter a sitemap URL directly when it's in a non-standard place.

How does Only new URLs work? Seen URLs are stored per site in a named key-value store (stateStoreName). In this mode the Actor reads all sitemap files, up to Max sitemap files per site, so it remembers every URL even when it outputs fewer, and the next run returns only URLs that are actually new.

Does it visit the pages? No. It only reads sitemap files, so it's very fast and light on the website. To scrape page content, pass the url field to a web scraper.

Why is lastmod null? Many sitemaps don't include it. The Modified after filter drops URLs without a lastmod.

Related actors

More Actors by the same author:

• RSS Feed Reader & Monitor (sysco32/rss-feed-reader): RSS, Atom and JSON Feed items as clean JSON, with only-new-item monitoring.
• Bulk Domain WHOIS/RDAP, DNS & Email Security Checker (sysco32/domain-rdap-dns-lookup): registrar and expiry dates, DNS records, mail provider and SPF/DMARC checks for many domains.
• Substack Scraper (sysco32/substack-newsletter-scraper): posts, likes, comments and full text of free posts from any public Substack newsletter.

Need a custom scraper or data pipeline?

If you need page content or metadata from the URLs you found, extra fields, or the results delivered to Google Sheets, a database or an n8n workflow on a schedule, I take custom projects on Fiverr: Python scraping scripts and n8n automation workflows. Public data only; message me with the details first and I'll confirm whether it's doable.