Sitemap URL Scraper: List URLs & Check Changes

What does Sitemap URL Scraper do?

Give it a site or a sitemap URL and you get every address the site lists there, up to 1,000 a run by default. It costs you $0.40 per 1,000 URLs listed. By default you get one row per URL the sitemap lists, like this one:

{
  "url": "https://nces.ed.gov/programs/coe/ataglance",
  "status": 200,
  "changefreq": "monthly",
  "parent_sitemap": "https://nces.ed.gov/programs/coe/sitemap.xml"
}

Bare domains and index files are resolved for you. A domain's robots.txt is checked for Sitemap lines, and an index is followed into its children. One site is enough to start:

{
  "startUrls": ["https://cmu.edu/sitemap.xml"]
}

A longer list works the same way. This input also adds a health row for each sitemap file, from two sites:

{
  "startUrls": ["https://apify.com", "https://www.sitemaps.org/sitemap.xml"],
  "output": "urls_and_audit",
  "maxUrls": 500
}

Features

• A health check of every sitemap a site declares: HTTP status, URL count, newest lastmod and a dead-link sample. An alert flags an error answer, a shrink or a stale file.
• What changed since your last run: with the health check on, from your second run on you see which URLs were added or removed, and how long the list has gone unchanged.
• A row for every site you give it: each declared URL lands with its lastmod, changefreq and the file that declared it. If a site can't be read, you still get a row saying why.
• Blocked sites named, not skipped: if a site blocks the read, you get a row saying so, and the run moves on.

Why this one

The most-used alternative on this platform lists the URLs and stops there. Both were run on cmu.edu's sitemap, which lists 307 addresses:

	The most-used alternative	This Actor
URLs listed	307 in 51 seconds	307 in 36 seconds, with the health check on
Health check of each sitemap	Not offered	Yes, with an alert
What changed since your last run	Not offered	Yes
Price per 1,000 URLs	$0.50	$0.40, plus $0.003 a sitemap when the health check is on

Every row names the input it came from, so you can trace an unreadable file back to what you pasted.

How much does it cost?

You pay $0.003 for each sitemap audited (the health rows) and $0.0004 for each URL listed. Apify also charges $0.00005 for each Actor start. The prefilled run lists addresses only and costs you about 16 cents. The pricing page carries every price on the ladder.

How do I use it?

1. Paste one or more starting points: a bare domain, a site URL, a sitemap URL, or the address of a text file listing sitemap URLs.
2. Under What to get, keep URL list only, or add the health check of each sitemap, or pick the health check alone.
3. Run it, then download your dataset as JSON, CSV or Excel. You can also wire it into your own workflow through the platform's integrations (the API, webhooks, Zapier, Make). A run lists up to 1,000 addresses by default; raise Cost cap: URLs listed for more.

Input

Setting	Type	What it does
Start URLs (startUrls)	array	Sitemap URLs, site URLs or domains to read. A site root is resolved through its robots.txt, a sitemap index is followed into its children.
Sitemap list file URL (sitemapListUrl)	string	Address of a text or CSV file listing sitemap URLs or site roots, one per line.
What to get (output)	string	What to get: urls_only (the default), urls_and_audit (adds a health row per file), or audit_only.
Include URL patterns (urlFilterInclude)	array	Glob patterns a listed address must match to be listed.
Exclude URL patterns (urlFilterExclude)	array	Glob patterns a listed address must not match to be listed.
Dead-link check (linkCheck)	string	Dead-link check: sample (the default), all listed pages, or off.
Cost cap: URLs listed (maxUrls)	integer	Cap on listed-address rows. Default 1,000.
Cost cap: URLs per site (maxUrlsPerSite)	integer	The most URL rows one site may return, so one large site cannot use the whole URL cap.
Cost cap: sitemap health rows (maxItems)	integer	The most sitemaps checked per run when you add the health check. Default 100.
Proxy (proxyConfiguration)	object	Optional proxy routing for the fetches and probes. Off by default; a refusal is reported on its row.

These stay off the form. Set them over the API or in the JSON editor:

Field	Default	What it does
deadLinkSample	50	Pages a sample check reads per run, at most 500.
staleAfterDays	30	Days without a change before a stale-sitemap alert.
maxItemsPerStartUrl	none	Sitemap files one start URL may audit, at most 200.
maxCrawlingDepth	2	Sitemap levels to follow; 1 reads only the files you name.
maxRequestRetries	2	Retries after a network error. A short stated pause (a 429) is waited for once.
proxyRotation	PER_REQUEST	PER_REQUEST: a new proxy address per request. UNTIL_FAILURE: one address until it fails.
includeUrls, urlsOnly	none	Older switches, kept for saved tasks; What to get replaces them.
checkPageStatus, checkAllUrls	none	Older switches, kept for saved tasks; Dead-link check replaces them.

Output

Each listed-address row describes one address; with the health check on, each health row describes one sitemap file. The fields most buyers use are below. Every field and what it holds is on the Output tab of this page.

On every row

Field	What it holds
url	The address the row is about: the sitemap file on a health row, the declared page on an address row.
status	The page's HTTP status, or a word saying why there is none, such as not_checked or blocked. Every word is on the Output tab.
row_type	Tells you which rows you pay for: only ROW rows are billed.
entry_kind	sitemap on a file row, url on a declared-address row.
parent_sitemap	The sitemap file that declared this address (the index on a child file).
source_url	The input entry the row came from, exactly as you submitted it.
lastmod	Newest lastmod a file declares on a health row; the address's own lastmod on a URL row.
changefreq	The declared <changefreq> on a URL row. Empty when absent or not one of the protocol's seven words; the health row counts those.
priority	The declared <priority> on a URL row, as a number from 0 to 1. Empty when absent or malformed; the health row counts the malformed ones.

On a health row

Field	What it holds
sitemap_kind	What the file answered with: urlset, sitemapindex, none, or not_xml.
declared_url_count	How many URLs the file lists.
drift_kind	How the list moved since your last run: unchanged, added, removed and the like, or first_run.
added_since_last_run	URLs new since your last run (first 500).
removed_since_last_run	URLs gone since your last run (first 500).
sampled_dead_urls	Checked URLs that came back dead.
alert	True when the sitemap itself has a problem.
alert_reasons	Which problems set alert.

A site the run couldn't read gets a free row, and reason says why: refused, not found, or a cap reached.

A few fields stay empty until a site has the matching problem or content: challenge_page_detected, soft_404_signal, robots_advertised_but_missing, sampled_dead_urls, not_found_kind (filled only when a sitemap can't be found), video_count, video_urls, video_titles, image_titles (when the sitemap titles its images), and invalid_priority_examples and invalid_changefreq_examples (when a file carries a malformed value; the counts beside them read 0 otherwise).

Limits and fair use

Bound	Value
Sitemap files audited per run	maxItems, default 100
Listed addresses	maxUrls, default 1,000
Sitemap files read, hard ceiling	200 per run; one entry's files bounded by maxItemsPerStartUrl
Dead-link probes	A sample: deadLinkSample, default 50. Every listed page: up to the URL cap.
Sitemap body	Read up to 50 MB
One fetch	Gives up after 20 seconds

By default the dead-link check covers up to 50 pages a run, so most listed URLs stay not_checked: 998 of the 1,109 rows in a large test run (90%). Set Dead-link check to every listed page to check them all.

FAQ

Does it fetch the pages a sitemap lists?

Only for the dead-link check. By default it checks up to 50 listed pages a run for their HTTP status and reads the first 64 KB of each. Set Dead-link check to off and no page is fetched.

Do I need a proxy?

You don't. A sitemap is a file the site publishes for crawlers, and a normal read needs none. A host that refuses the read, or answers with an empty or challenge page, gets a blocked row saying what it answered.

Why is lastmod missing on some records?

A sitemap declares it per address and many files declare none at all. lastmod stays empty where the sitemap publishes no date; lastmod_url_count on the file's row says how many it carried.

What if a site blocks the read?

You get a row naming the block, and the run moves on to the next site, leaving that site alone for the rest of the run. A site that asks the run to wait a minute or less is waited for once.

What does a status row cost you?

Nothing. A row for a site it could not read, or for an entry it skipped, is free.

Watch a sitemap URL over time

With the health check on, each sitemap's URL list is compared with the list from your last run. Your run after the first reports what was added, what was removed and how many days the set has gone unchanged. You get an alert on a problem with the sitemap itself, such as an error or empty answer (a site refusing the read, such as a 403, counts), a shrink of a fifth or more, a stale file (unchanged for 30 days, or dated more than 30 days ago), a fifth of its checked links dead, or dates stamped in bulk. alert_reasons names each one. A site that times out or drops the connection gets a row saying so rather than an alert. Schedule it with the health check on and every run tells you what changed.

Your responsibility

You are responsible for having the right to read the sitemaps you give it and for how you use the rows. A site's robots.txt and terms tell you what is off limits.

Personal data

The rows carry sitemap metadata: file addresses, listed addresses, timestamps. Read the platform's data protection terms for how a run's data is stored and retained.

Release notes

• 2026-10-06: very large sitemap indexes finish within the default memory; a dead-link sample is shared across a site's files; a robots.txt that cannot be read is reported as blocked or unreachable, never as "no sitemap"; an old newest date raises the stale alert on the first run. priority is now a number, so it sorts as one, and the health row counts malformed priority and changefreq values with five examples. The health-row cap is shared across your start URLs, a list file that is missing is named as missing, and a domain that does not exist says so.
• 2026-10-06: a site that answers "too many requests" and asks for a pause of a minute or less is now waited for once, then asked again, instead of being dropped.
• 2026-10-06: one What to get choice replaces the two URL switches, and one Dead-link check choice replaces three check fields. A run now returns the URL list by default, every URL up to Cost cap: URLs listed (1,000), and adds the health check when you pick it. Saved tasks and API calls that use the old settings keep working.
• 2026-10-06: dead-link checks no longer wait between requests: on cmu.edu's sitemap a run now takes 36 seconds where it took 97. A refusal still stops the checks, and a site that refuses through a proxy is not asked again in that run.

Not affiliated

This Actor is an unofficial, independent tool. It reads files a site publishes for crawlers and is not affiliated with, sponsored, or endorsed by any site whose sitemap it reads.

Feedback

Something not working, or a case the rows miss? Open an issue on the Issues tab and it will be looked at. If the Actor saved you time, a review on the Store helps other buyers find it.