Pricing

from $0.21 / 1,000 urls

Try for free

Go to Apify Store

Sitemap URL Extractor with Change Tracking

Try for free

Extracts every URL from a site's sitemaps (indexes, gzip, text and feed formats included) and reports which URLs were added, removed or re-dated since the previous run.

Pricing

from $0.21 / 1,000 urls

Rating

0.0

(0)

Developer

Shyheim Ulrickson

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

SEO tools

Developer tools

Automation

Domains or sitemap URLs

targets

Optional

What to extract. Give a bare domain (example.com) and the Actor finds the sitemaps through robots.txt, falling back to common paths such as /sitemap.xml and /wp-sitemap.xml. Give a direct sitemap URL and it is used as-is. Sitemap indexes, .xml.gz files, plain-text URL lists and RSS/Atom feeds are all handled. Leave empty to run a small demo on crawlee.dev (100 URLs).

Type:array

Min. items:1

Max. items:1000

Example:

[
  "example.com",
  "https://www.example.org/sitemap_index.xml"
]

Run mode

mode

Optional

Snapshot returns every URL found. Change tracking compares this run against the previous run's baseline for the same target and labels each URL added, changed, removed or unchanged. The first run in change mode has no baseline, so everything is reported as added and the baseline is written for next time.

Type:string

Default:snapshot

Options:

snapshotchanges

Change types to output

outputChangeTypes

Optional

Only used in change-tracking mode. Leave the default to get just the differences, which is what a scheduled monitor usually wants. Add 'unchanged' to get a full snapshot with change labels attached.

Type:string[]

Default:

[
  "added",
  "changed",
  "removed"
]

Baseline key-value store name

stateStoreName

Optional

Named key-value store that holds the previous run's URL set. A named store survives between runs, unlike the default store, which is what makes change tracking work. Use one store name per monitoring job; separate targets are kept under separate keys inside it.

Type:string

Default:sitemap-change-tracker-state

Max sitemap failure rate for updating the baseline

maxSitemapFailurePercentForBaselineUpdate

Optional

Safety valve for change tracking. If more than this percentage of a target's sitemaps fail to load, the baseline is left untouched, so a bad day at the origin server does not make the next run report the whole site as 'added'. A run that extracts zero URLs never overwrites the baseline, regardless of this setting.

Type:integer

Minimum:0

Maximum:100

Default:25

Check each URL's HTTP status

checkUrlStatus

Optional

Sends a cheap HEAD request per URL (falling back to GET when a server rejects HEAD) and adds statusCode, statusOk and redirectedTo to every row. Useful for finding sitemap entries that 404. Off by default because it multiplies the number of requests.

Type:boolean

Default:false

Max URLs to status-check per target

statusCheckLimit

Optional

Caps the HEAD checks so a 300,000-URL sitemap cannot turn into a 300,000-request run by accident.

Type:integer

Minimum:1

Maximum:50000

Default:500

Only URLs modified on or after

onlyUrlsModifiedSince

Optional

YYYY-MM-DD. Drops URLs whose lastmod is older than this date. URLs with no lastmod at all are kept, because a missing date is not evidence that a page is old.

Type:string

Max URLs per target

maxUrlsPerTarget

Optional

Hard ceiling on URLs collected per domain. Sitemaps are walked breadth-first, so hitting this limit still returns a spread across the whole site rather than everything from one section. The default of 1,000 caps a run at about $0.25 per target; raise it for a full inventory.

Type:integer

Minimum:1

Maximum:5000000

Default:1000

Max sitemap files per target

maxSitemapsPerTarget

Optional

Hard ceiling on sitemap documents fetched per domain. Large e-commerce sites routinely publish hundreds of child sitemaps behind one index.

Type:integer

Minimum:1

Maximum:50000

Default:500

Max sitemap index depth

maxSitemapDepth

Optional

How many levels of nested sitemap index to follow. 0 means only the sitemaps given or discovered directly.

Type:integer

Minimum:0

Maximum:20

Default:5

Include hreflang alternates in change mode

includeAlternates

Optional

Adds the xhtml:link hreflang alternates to each row. Snapshot mode always includes them when present; this switch controls the (usually much larger) change-mode rows.

Type:boolean

Default:false

Respect robots.txt

respectRobotsTxt

Optional

Checks the sitemap path against robots.txt before fetching it. Leave this on. Sitemaps are published for crawlers, but a few sites still disallow specific sitemap paths.

Type:boolean

Default:true

Probe common sitemap paths

discoverViaCommonPaths

Optional

When robots.txt declares no Sitemap: directive, try /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and nine other well-known locations. Turn this off to rely on robots.txt only.

Type:boolean

Default:true

Requests per second

requestsPerSecond

Optional

Global rate ceiling across all targets. Lower this for small or slow origin servers.

Type:integer

Minimum:1

Maximum:50

Default:5

Concurrent requests

concurrency

Optional

How many sitemap or status requests are in flight at once. The rate limit above still applies.

Type:integer

Minimum:1

Maximum:50

Default:10

Attempts per request

maxRetries

Optional

Total attempts, including the first. Retries use exponential backoff with jitter and honour Retry-After. Only transient failures (timeouts, 5xx, 429) are retried; a 404 fails immediately instead of burning four timeouts.

Type:integer

Minimum:1

Maximum:10

Default:4

Request timeout (seconds)

requestTimeoutSecs

Optional

How long to wait for a single sitemap or status request before giving up on that attempt. Large gzipped sitemap indexes on slow origins can legitimately take 30 seconds or more, so lowering this below the default will start costing you URLs.

Type:integer

Minimum:5

Maximum:300

Default:60

Contact string for the User-Agent

userAgentContact

Optional

Appended to the User-Agent header, for example a URL or an email address, so site owners can see who is crawling them. Optional here; the Actor sends a descriptive User-Agent either way.

Type:string

Example:

+https://your-company.example/bot

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

403

3.2

(8)

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Sitemap Change Monitor: New & Removed Page Alerts

scrapemint/sitemap-change-monitor

Watch any website's sitemap and get one row per change: new pages, removed pages, optional lastmod updates. See competitors' new products, content clusters and landing pages the day they ship. Pay only for changes; quiet sites cost nothing. Keyless, no proxy.

Ken M

5.0

(1)

Sitemap URL Extractor & 404 Checker — XML Sitemap Scraper

factpipe/sitemap-url-extractor

Extract every URL from any website's XML sitemaps: robots.txt discovery, nested sitemap indexes, lastmod and priority. Optional HTTP status check finds 404s and redirects. $0.30 per 1,000 URLs.

Neo B

Sitemap URL Extractor

gifted_wagon/sitemap-url-inventory

Traverse public sitemap indexes into a deduplicated URL inventory with source sitemap, lastmod, changefreq, priority, media counts, and partial-failure evidence.

Michael Olmos

Sitemap URL Extractor

sootesting/sitemap-url-extractor

Extract URLs from XML sitemaps in bulk. Supports sitemap indexes, auto-discovers nested sitemaps, and outputs structured JSON. Pay-per-event: per 1,000 URLs extracted.

soot

SEO Audit API — On-Page Checks, Meta Tags, Headings & Schema

highbrow_fame/seo-audit-api

On-page SEO audit of any URL over plain HTTP: title, meta, canonical, robots, hreflang, headings, images alt, links, JSON-LD, redirects, a 0-100 score with issues. SEO Audit API.

yestrue

Website Sitemap URL Extractor

myagizm/website-sitemap-extractor

Extract and deduplicate page URLs with metadata from a public XML sitemap or bounded sitemap index.

MYM

Bulk PageSpeed and Core Web Vitals Checker

opendatalabs/bulk-pagespeed-cwv-checker

Audits a list of URLs with Google's official PageSpeed Insights API v5 and returns performance, accessibility, SEO and best-practices scores alongside lab and CrUX field Core Web Vitals.