Pricing

Pay per event

Try for free

Go to Apify Store

Sitemap URL Extractor & New Page Monitor

Try for free

Get every URL from any website's XML sitemaps (robots.txt discovery, sitemap indexes, .gz, text, RSS/Atom) with lastmod, hreflang and images. Optional HTTP status check for 404s and redirects, and a monitor mode that returns only new pages. No browser, no start fee.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Open Data Tools

Actor stats

Bookmarked

Total users

Monthly active users

13 days ago

Last modified

Categories

SEO tools

Developer tools

Automation

Websites or sitemap URLs

startUrls

Required

One per line. A domain (example.com) or homepage finds the sitemaps through robots.txt and common locations. A direct sitemap URL (https://example.com/sitemap_index.xml, .xml.gz, .txt, RSS or Atom feed) is read as is.

Type:array

Maximum URLs per site

maxUrlsPerSite

Optional

Stop reading a site's sitemaps after this many unique URLs. Filters below apply after this cap.

Type:integer

Minimum:1

Maximum:2000000

Default:10000

Only URLs matching (regex)

includePatterns

Optional

Keep only URLs matching at least one of these regular expressions (case-insensitive). Example: /blog/ or .pdf$

Type:array

Skip URLs matching (regex)

excludePatterns

Optional

Drop URLs matching any of these regular expressions (case-insensitive). Example: /tag/ or ?

Type:array

Modified on or after

lastmodFrom

Optional

Keep only URLs whose sitemap is on or after this date (YYYY-MM-DD). URLs without a lastmod are dropped when this is set.

Type:string

Monitor: return only new URLs

onlyNewUrls

Optional

Remembers every URL seen per site (in a named key-value store in your account). The first run saves a baseline and returns nothing, so it costs nothing per URL. Later runs return only URLs that were not in the sitemaps before. Schedule it daily to watch competitors' new pages, products or posts.

Type:boolean

Default:false

Monitor state store name

stateStoreName

Optional

Named key-value store that keeps the monitor's memory. Use different names for separate monitors.

Type:string

Default:sitemap-extractor-state

Check HTTP status of each URL

checkStatus

Optional

Sends a HEAD request (GET if HEAD is refused) to every output URL and records the status code, redirect and final URL. Finds 404s and redirects listed in sitemaps. Charged as a separate event per URL.

Type:boolean

Default:false

Status check concurrency

statusCheckConcurrency

Optional

Parallel status requests. Keep it low for small sites.

Type:integer

Minimum:1

Maximum:50

Default:10

Maximum sitemap files per site

maxSitemapsPerSite

Optional

Safety cap for huge sitemap indexes.

Type:integer

Minimum:1

Maximum:50000

Default:500

Request timeout (seconds)

requestTimeoutSecs

Optional

How long to wait for one sitemap or status request before retrying.

Type:integer

Minimum:5

Maximum:300

Default:30

User-Agent

userAgent

Optional

Optional. By default the actor identifies itself as SitemapExtractor. Set your own if you are checking your own site behind a firewall rule.

Type:string

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

403

3.2

(8)

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

Sitemap Change Monitor: New & Removed Page Alerts

scrapemint/sitemap-change-monitor

Watch any website's sitemap and get one row per change: new pages, removed pages, optional lastmod updates. See competitors' new products, content clusters and landing pages the day they ship. Pay only for changes; quiet sites cost nothing. Keyless, no proxy.

Ken M

5.0

(1)

Sitemap Extractor

cerebral_aluminum/sitemap-extractor

Extract all URLs from website sitemaps. Pages, images, PDFs. Handles sitemap indexes and WordPress.

Benny

Website Sitemap URL Extractor

myagizm/website-sitemap-extractor

Extract and deduplicate page URLs with metadata from a public XML sitemap or bounded sitemap index.

MYM

Sitemap URL Extractor & 404 Checker — XML Sitemap Scraper

factpipe/sitemap-url-extractor

Extract every URL from any website's XML sitemaps: robots.txt discovery, nested sitemap indexes, lastmod and priority. Optional HTTP status check finds 404s and redirects. $0.30 per 1,000 URLs.

Neo B

Sitemap Change Detector

tri_angle/sitemap-change-detector

Identify and monitor sitemaps for specified websites. Retrieve only the new, updated, or removed URLs since the last crawl.

Tri⟁angle

Sitemap Extractor

johnlenflure/sitemap-extractor

Extract and parse XML sitemaps from any website. Discovers sitemaps via robots.txt and common paths. Outputs structured URL data with metadata.

Sinan Donmez

Sitemap Extractor – All URLs from sitemap.xml + Status Check

gazidev/sitemap-url-extractor

Extract every URL from any website's sitemap.xml: auto-discovery via robots.txt, nested sitemap indexes, .xml.gz, image/news/video/hreflang tags and lastmod. Filter by date or glob, get only new URLs, and optionally check HTTP status, redirects and canonical per URL.