Sitemap URL Extractor — every URL in a site's sitemaps avatar

Sitemap URL Extractor — every URL in a site's sitemaps

Pricing

from $0.20 / 1,000 url extracteds

Go to Apify Store
Sitemap URL Extractor — every URL in a site's sitemaps

Sitemap URL Extractor — every URL in a site's sitemaps

Extract every URL from a site's XML sitemaps: robots.txt discovery, nested sitemap indexes, gzip and RSS/Atom feeds, with lastmod, changefreq, priority, image count and hreflang alternates. Optional HTTP status check per URL. $0.30 per 1,000 URLs.

Pricing

from $0.20 / 1,000 url extracteds

Rating

0.0

(0)

Developer

subimpact

subimpact

Maintained by Community

Actor stats

0

Bookmarked

1

Total users

1

Monthly active users

3 hours ago

Last modified

Features

Limited permissions

Share

Get every URL on a website as JSON, from the source that websites publish for exactly this purpose: their sitemaps. Point it at a domain and it finds the sitemaps itself through robots.txt, follows sitemap indexes to their children, unpacks .xml.gz, and reads RSS, Atom and plain-text sitemap files too.

One dataset row per URL, with the sitemap it came from, its lastmod, changefreq, priority, image count and hreflang alternates. Optionally add one HTTP status code per URL so you can spot 404s and redirects in the same run.

Pay only for what you get: $0.30 per 1,000 URLs. No start fee, no monthly minimum.

  • Sitemap discovery done for you. robots.txt Sitemap: lines first (all of them, mixed case and relative paths included), then /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml. Direct sitemap URLs are also accepted and read as-is.
  • Nested indexes, gzip and feeds. Sitemap indexes are followed up to 5 levels deep (Google's own limit), .xml.gz files are gunzipped, and RSS/Atom feeds listed as sitemaps are parsed as URL sources.
  • The extra fields nobody exports. lastmod, changefreq, priority, the number of <image:image> entries per URL (plus the image URLs themselves) and the xhtml:link hreflang alternates that multilingual sites publish.
  • One bad sitemap never kills a run. A sitemap that 404s, times out or turns out to be an HTML error page becomes one free error row and the run carries on with the next sitemap.
  • Filters before you pay. Regex include/exclude patterns and a lastmodAfter cutoff decide what is delivered, so you are only charged for URLs that reach your dataset.
  • Built for big sitemaps. Sitemaps are parsed as a stream: a 50,000-URL sitemap file never has to fit in memory, and delivery starts while the file is still downloading. Measured peak memory in the runs below: about 100 MB for 1,000 URLs, 125 MB for 20,000.

Input

FieldTypeDefaultNotes
startUrlsarray of stringsprefill ["https://www.smashingmagazine.com"]Website roots (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml). At least one
includePatternsarray of strings[]Keep only URLs matching at least one JavaScript regular expression, e.g. /blog/. Empty = keep everything
excludePatternsarray of strings[]Drop URLs matching any regular expression, e.g. \.(jpg|png|pdf)$. Applied after the includes
lastmodAfterstring""ISO date (2026-01-31 or 2026-01-31T12:00:00Z). Keeps entries whose lastmod is on or after it; entries with no lastmod are dropped while this is set
maxUrlsinteger1000Hard cap on delivered URL rows, 1 to 1,000,000. The run stops at the cap
checkStatusbooleanfalseSend a HEAD request (GET fallback when the server refuses HEAD) per delivered URL and fill statusCode. Billed separately
maxSitemapsinteger200How many sitemap files one run may read, index children included, 1 to 10,000
requestTimeoutSecsinteger60How long to wait for a sitemap or robots.txt response, 5 to 300 seconds
proxyConfigurationobjectnot setOptional Apify proxy for sites that block datacenter IPs. Sitemaps are public, so the default is a direct connection

Invalid input fails the run immediately with a clear message, before any request and before any charge: a maxUrls of 0, an unparseable regex, a lastmodAfter that is not an ISO date, or a start URL like mailto:.... A start URL that is not usable is also delivered as one free error row while the other start URLs still run.

Example inputs

A whole site, the pre-filled example (1,000 URLs maximum):

{
"startUrls": ["https://www.smashingmagazine.com"]
}

Only blog posts, no images or PDFs, 2,000 URLs:

{
"startUrls": ["https://blog.apify.com"],
"includePatterns": ["^https://blog\\.apify\\.com/"],
"excludePatterns": ["\\.(jpg|jpeg|png|webp|pdf)$"],
"maxUrls": 2000
}

Sitemap content that changed this year, with a status code per URL:

{
"startUrls": ["https://example.com/sitemap.xml"],
"lastmodAfter": "2026-01-01",
"checkStatus": true,
"maxUrls": 500
}

A sitemap on a site that only serves datacenter IPs through a proxy:

{
"startUrls": ["https://example.com"],
"proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

Output

FieldTypeNotes
urlstring or nullAbsolute URL from the sitemap, fragment stripped. null on error rows
sitemapUrlstring or nullThe sitemap file this URL came from (after redirects)
lastmodstring or nullThe <lastmod> value exactly as published (W3C datetime, date-only, or RSS pubDate / Atom updated)
changefreqstring or nullalways, hourly, daily, weekly, monthly, yearly, never
prioritynumber or nullThe sitemap <priority>, 0 to 1
imagesinteger or nullHow many <image:image> entries the sitemap lists for this URL (0 when it lists none)
imageUrlsarray or nullThe <image:loc> values for this URL, in sitemap order
alternatesarray or nullxhtml:link rel="alternate" entries: { hreflang, href } per language variant
statusCodeinteger or nullWith checkStatus: the first HTTP status code, redirects not followed (so 301/302 shows a redirect). null when the check is off or failed
scrapedAtstringISO 8601 timestamp of delivery
statusstringok for a URL row, error for a sitemap or discovery step that could not be read
errorstring or nullWhy a sitemap could not be read (HTTP status, timeout, "not a sitemap", empty). null on URL rows

Real rows from the live runs on 2026-11-15 (node test/live.mjs https://blog.apify.com/sitemap-posts.xml 20):

{
"url": "https://blog.apify.com/black-friday-checklist/",
"sitemapUrl": "https://blog.apify.com/sitemap-posts.xml",
"lastmod": "2026-11-13T15:02:27.000Z",
"changefreq": null,
"priority": null,
"images": 1,
"imageUrls": ["https://storage.ghost.io/c/f2/6e/f26ec999-9a90-4aee-a0d4-9b3ca2bb668f/content/images/2026/11/black-friday-ecommerce-checklist-feature-2.webp"],
"alternates": null,
"statusCode": null,
"scrapedAt": "2026-11-15T10:02:25.764Z",
"status": "ok",
"error": null
}

A site with no sitemap at all — node test/live.mjs https://example.com 5 — delivers one free error row instead of failing the run:

{
"url": null,
"sitemapUrl": null,
"lastmod": null,
"changefreq": null,
"priority": null,
"images": null,
"imageUrls": null,
"alternates": null,
"statusCode": null,
"scrapedAt": "2026-11-15T10:00:12.004Z",
"status": "error",
"error": "No sitemap found for https://example.com. Tried: https://example.com/sitemap.xml: HTTP 404; https://example.com/sitemap_index.xml: HTTP 404; https://example.com/sitemap-index.xml: HTTP 404; https://example.com/wp-sitemap.xml: HTTP 404"
}

Pricing

EventPriceCharged when
url-extracted$0.0003 per URL ($0.30 per 1,000)a URL row was pushed to the dataset
url-status-checked$0.0005 per URL ($0.50 per 1,000)checkStatus is on and a status code came back for a delivered URL

Nothing else is charged. Error rows are free. URLs removed by your filters, duplicates, and URLs that never got delivered because the cap or the run's max charge stopped the run are free too — a row is pushed first and charged second, so you never pay for a row you did not receive.

Examples: 1,000 URLs = $0.30; 10,000 URLs = $3.00; 1,000 URLs with a status check = $0.80. Apify platform usage (a default 1,000-URL run takes a few seconds and peaks around 100 MB) is counted separately by Apify.

How it works

  1. Discovery. For each start URL the run reads robots.txt and takes every Sitemap: line (all of them, not just the first). If there is none, it probes /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml in that order and stops at the first file that really contains sitemap records. Only when all four fail is one error row delivered, so a site without sitemaps does not fill your dataset with noise.
  2. Traversal. A sitemap index (<sitemapindex>) queues its children, up to 5 levels deep and up to maxSitemaps files. Each sitemap URL is fetched once per run, so a sitemap linked from two places is not read twice.
  3. Parsing. The response body is streamed through a tolerant XML tokenizer: <urlset>, <sitemapindex>, RSS <item> and Atom <entry> documents are recognised by their root element, with the image and xhtml:link extensions. Real sitemaps often contain raw & and stray markup, so a malformed document is parsed rather than thrown away. Bytes are decoded as UTF-8 across chunk boundaries, so an accented or non-Latin URL that is split between two network chunks still comes out whole. A body that is not XML is read as a plain-text sitemap (one URL per line). An HTML page — a soft 404 — is reported as "not a sitemap", never as zero URLs.
  4. Delivery. URLs are normalised (whitespace stripped, relative <loc> resolved, fragments removed), deduplicated, filtered, then pushed to the dataset in batches of up to 500 rows as the file streams in — each batch is pushed first and charged second. A 50,000-URL sitemap therefore starts producing rows immediately instead of after the download.
  5. Stop conditions. maxUrls (counted in delivered rows), maxSitemaps, an ACTOR_MAX_PAID_DATASET_ITEMS limit, or Apify's max-charge limit for the run. Whichever comes first, the run stops cleanly, the open response is destroyed and the dataset keeps everything already delivered.

RUN_SUMMARY in the run's key-value store records the counts for the run: sitemaps read, URLs seen, delivered, duplicates, each filter reason, and error rows.

Reliability, blocks and limits

  • No proxy needed by default. Sitemaps are public files and the live runs below were done directly, without a proxy: blog.apify.com (999 URLs from 5 sitemaps, 0.2 s, 98 MB peak), www.apify.com (a 22-child sitemap index), www.nytimes.com (a gzipped sitemap index with thousands of children: 1,000 URLs in 1.5 s, 103 MB) and example.com (no sitemap: one free error row in 0.2 s). None of them blocked the run; every response was HTTP 200.
  • If a site does block datacenter IPs, set proxyConfiguration (an Apify proxy or your own). The actor never retries forever: 408/429/5xx responses are retried twice with backoff, a 404 is not retried, a stalled connection is dropped after 30 seconds without data, and any sitemap that still fails becomes a free error row.
  • Size guardrails: 100 MB is the decompressed ceiling for a single sitemap (Google's own limit is 50 MB), 5 levels of nested indexes, maxSitemaps 200 by default, and maxUrls 1,000 by default so a first run costs cents.
  • Redirects: sitemap and robots.txt requests follow up to 5 redirects; statusCode checks do not.

What it does not do

  • It does not crawl your pages: it reads sitemaps. A site with no sitemap has no URLs to return.
  • Sitemaps that only exist behind JavaScript rendering are not discovered (sitemap files are XML, not rendered pages).
  • checkStatus is a HEAD request (GET only when the server refuses HEAD), so it reports the HTTP status, not the page content, its title or whether the page is soft-404 content.
  • hreflang alternates are reported on the row of the URL that declares them; they are not expanded into extra rows, and they are not charged as URLs.
  • Sitemaps are decoded as UTF-8, which the sitemap protocol requires; a sitemap served in a legacy 8-bit encoding (ISO-8859-1) would deliver mangled accented characters in its URLs.
  • Sitemap entries are reported as published. If a sitemap lists a URL that has since been deleted, the row still appears — use checkStatus to see the 404.

Development

npm install
npm test # offline: node --test, fixtures only, no network
node test/live.mjs https://blog.apify.com 25 # live smoke test, not part of npm test
node test/live.mjs https://blog.apify.com 10 --check-status

The offline suite covers sitemap dialects (test/fixtures/), chunk boundaries, gzip, plain text, 50,000-URL documents, the discovery and traversal rules, the billing order (push before charge, failed pushes never charged, error rows free, cap counted in delivered rows), input validation and the manifest/ dataset-schema limits Apify enforces at deploy time. test/live.mjs runs the real fetch-and-parse path against one live site and prints the row count, one full row and peak memory; it exits non-zero when nothing is delivered.

Run monitoring and privacy

If a run fails or times out, this Actor tells its developer the run ID and status, so we can fix problems fast. It never sends your input, your results or your account data.