Sitemap URL Extractor — every URL in a site's sitemaps
Pricing
from $0.20 / 1,000 url extracteds
Sitemap URL Extractor — every URL in a site's sitemaps
Extract every URL from a site's XML sitemaps: robots.txt discovery, nested sitemap indexes, gzip and RSS/Atom feeds, with lastmod, changefreq, priority, image count and hreflang alternates. Optional HTTP status check per URL. $0.30 per 1,000 URLs.
Pricing
from $0.20 / 1,000 url extracteds
Rating
0.0
(0)
Developer
subimpact
Maintained by CommunityActor stats
0
Bookmarked
1
Total users
1
Monthly active users
3 hours ago
Last modified
Categories
Features
Share
Get every URL on a website as JSON, from the source that websites publish for exactly this
purpose: their sitemaps. Point it at a domain and it finds the sitemaps itself through
robots.txt, follows sitemap indexes to their children, unpacks .xml.gz, and reads RSS, Atom
and plain-text sitemap files too.
One dataset row per URL, with the sitemap it came from, its lastmod, changefreq, priority,
image count and hreflang alternates. Optionally add one HTTP status code per URL so you can spot
404s and redirects in the same run.
Pay only for what you get: $0.30 per 1,000 URLs. No start fee, no monthly minimum.
- Sitemap discovery done for you.
robots.txtSitemap:lines first (all of them, mixed case and relative paths included), then/sitemap.xml,/sitemap_index.xml,/sitemap-index.xmland/wp-sitemap.xml. Direct sitemap URLs are also accepted and read as-is. - Nested indexes, gzip and feeds. Sitemap indexes are followed up to 5 levels deep (Google's
own limit),
.xml.gzfiles are gunzipped, and RSS/Atom feeds listed as sitemaps are parsed as URL sources. - The extra fields nobody exports.
lastmod,changefreq,priority, the number of<image:image>entries per URL (plus the image URLs themselves) and thexhtml:linkhreflang alternates that multilingual sites publish. - One bad sitemap never kills a run. A sitemap that 404s, times out or turns out to be an HTML
error page becomes one free
errorrow and the run carries on with the next sitemap. - Filters before you pay. Regex include/exclude patterns and a
lastmodAftercutoff decide what is delivered, so you are only charged for URLs that reach your dataset. - Built for big sitemaps. Sitemaps are parsed as a stream: a 50,000-URL sitemap file never has to fit in memory, and delivery starts while the file is still downloading. Measured peak memory in the runs below: about 100 MB for 1,000 URLs, 125 MB for 20,000.
Input
| Field | Type | Default | Notes |
|---|---|---|---|
startUrls | array of strings | prefill ["https://www.smashingmagazine.com"] | Website roots (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml). At least one |
includePatterns | array of strings | [] | Keep only URLs matching at least one JavaScript regular expression, e.g. /blog/. Empty = keep everything |
excludePatterns | array of strings | [] | Drop URLs matching any regular expression, e.g. \.(jpg|png|pdf)$. Applied after the includes |
lastmodAfter | string | "" | ISO date (2026-01-31 or 2026-01-31T12:00:00Z). Keeps entries whose lastmod is on or after it; entries with no lastmod are dropped while this is set |
maxUrls | integer | 1000 | Hard cap on delivered URL rows, 1 to 1,000,000. The run stops at the cap |
checkStatus | boolean | false | Send a HEAD request (GET fallback when the server refuses HEAD) per delivered URL and fill statusCode. Billed separately |
maxSitemaps | integer | 200 | How many sitemap files one run may read, index children included, 1 to 10,000 |
requestTimeoutSecs | integer | 60 | How long to wait for a sitemap or robots.txt response, 5 to 300 seconds |
proxyConfiguration | object | not set | Optional Apify proxy for sites that block datacenter IPs. Sitemaps are public, so the default is a direct connection |
Invalid input fails the run immediately with a clear message, before any request and before any
charge: a maxUrls of 0, an unparseable regex, a lastmodAfter that is not an ISO date, or a
start URL like mailto:.... A start URL that is not usable is also delivered as one free error
row while the other start URLs still run.
Example inputs
A whole site, the pre-filled example (1,000 URLs maximum):
{"startUrls": ["https://www.smashingmagazine.com"]}
Only blog posts, no images or PDFs, 2,000 URLs:
{"startUrls": ["https://blog.apify.com"],"includePatterns": ["^https://blog\\.apify\\.com/"],"excludePatterns": ["\\.(jpg|jpeg|png|webp|pdf)$"],"maxUrls": 2000}
Sitemap content that changed this year, with a status code per URL:
{"startUrls": ["https://example.com/sitemap.xml"],"lastmodAfter": "2026-01-01","checkStatus": true,"maxUrls": 500}
A sitemap on a site that only serves datacenter IPs through a proxy:
{"startUrls": ["https://example.com"],"proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }}
Output
| Field | Type | Notes |
|---|---|---|
url | string or null | Absolute URL from the sitemap, fragment stripped. null on error rows |
sitemapUrl | string or null | The sitemap file this URL came from (after redirects) |
lastmod | string or null | The <lastmod> value exactly as published (W3C datetime, date-only, or RSS pubDate / Atom updated) |
changefreq | string or null | always, hourly, daily, weekly, monthly, yearly, never |
priority | number or null | The sitemap <priority>, 0 to 1 |
images | integer or null | How many <image:image> entries the sitemap lists for this URL (0 when it lists none) |
imageUrls | array or null | The <image:loc> values for this URL, in sitemap order |
alternates | array or null | xhtml:link rel="alternate" entries: { hreflang, href } per language variant |
statusCode | integer or null | With checkStatus: the first HTTP status code, redirects not followed (so 301/302 shows a redirect). null when the check is off or failed |
scrapedAt | string | ISO 8601 timestamp of delivery |
status | string | ok for a URL row, error for a sitemap or discovery step that could not be read |
error | string or null | Why a sitemap could not be read (HTTP status, timeout, "not a sitemap", empty). null on URL rows |
Real rows from the live runs on 2026-11-15 (node test/live.mjs https://blog.apify.com/sitemap-posts.xml 20):
{"url": "https://blog.apify.com/black-friday-checklist/","sitemapUrl": "https://blog.apify.com/sitemap-posts.xml","lastmod": "2026-11-13T15:02:27.000Z","changefreq": null,"priority": null,"images": 1,"imageUrls": ["https://storage.ghost.io/c/f2/6e/f26ec999-9a90-4aee-a0d4-9b3ca2bb668f/content/images/2026/11/black-friday-ecommerce-checklist-feature-2.webp"],"alternates": null,"statusCode": null,"scrapedAt": "2026-11-15T10:02:25.764Z","status": "ok","error": null}
A site with no sitemap at all — node test/live.mjs https://example.com 5 — delivers one free
error row instead of failing the run:
{"url": null,"sitemapUrl": null,"lastmod": null,"changefreq": null,"priority": null,"images": null,"imageUrls": null,"alternates": null,"statusCode": null,"scrapedAt": "2026-11-15T10:00:12.004Z","status": "error","error": "No sitemap found for https://example.com. Tried: https://example.com/sitemap.xml: HTTP 404; https://example.com/sitemap_index.xml: HTTP 404; https://example.com/sitemap-index.xml: HTTP 404; https://example.com/wp-sitemap.xml: HTTP 404"}
Pricing
| Event | Price | Charged when |
|---|---|---|
url-extracted | $0.0003 per URL ($0.30 per 1,000) | a URL row was pushed to the dataset |
url-status-checked | $0.0005 per URL ($0.50 per 1,000) | checkStatus is on and a status code came back for a delivered URL |
Nothing else is charged. Error rows are free. URLs removed by your filters, duplicates, and URLs that never got delivered because the cap or the run's max charge stopped the run are free too — a row is pushed first and charged second, so you never pay for a row you did not receive.
Examples: 1,000 URLs = $0.30; 10,000 URLs = $3.00; 1,000 URLs with a status check = $0.80. Apify platform usage (a default 1,000-URL run takes a few seconds and peaks around 100 MB) is counted separately by Apify.
How it works
- Discovery. For each start URL the run reads
robots.txtand takes everySitemap:line (all of them, not just the first). If there is none, it probes/sitemap.xml,/sitemap_index.xml,/sitemap-index.xmland/wp-sitemap.xmlin that order and stops at the first file that really contains sitemap records. Only when all four fail is one error row delivered, so a site without sitemaps does not fill your dataset with noise. - Traversal. A sitemap index (
<sitemapindex>) queues its children, up to 5 levels deep and up tomaxSitemapsfiles. Each sitemap URL is fetched once per run, so a sitemap linked from two places is not read twice. - Parsing. The response body is streamed through a tolerant XML tokenizer:
<urlset>,<sitemapindex>, RSS<item>and Atom<entry>documents are recognised by their root element, with the image andxhtml:linkextensions. Real sitemaps often contain raw&and stray markup, so a malformed document is parsed rather than thrown away. Bytes are decoded as UTF-8 across chunk boundaries, so an accented or non-Latin URL that is split between two network chunks still comes out whole. A body that is not XML is read as a plain-text sitemap (one URL per line). An HTML page — a soft 404 — is reported as "not a sitemap", never as zero URLs. - Delivery. URLs are normalised (whitespace stripped, relative
<loc>resolved, fragments removed), deduplicated, filtered, then pushed to the dataset in batches of up to 500 rows as the file streams in — each batch is pushed first and charged second. A 50,000-URL sitemap therefore starts producing rows immediately instead of after the download. - Stop conditions.
maxUrls(counted in delivered rows),maxSitemaps, anACTOR_MAX_PAID_DATASET_ITEMSlimit, or Apify's max-charge limit for the run. Whichever comes first, the run stops cleanly, the open response is destroyed and the dataset keeps everything already delivered.
RUN_SUMMARY in the run's key-value store records the counts for the run: sitemaps read, URLs
seen, delivered, duplicates, each filter reason, and error rows.
Reliability, blocks and limits
- No proxy needed by default. Sitemaps are public files and the live runs below were done
directly, without a proxy:
blog.apify.com(999 URLs from 5 sitemaps, 0.2 s, 98 MB peak),www.apify.com(a 22-child sitemap index),www.nytimes.com(a gzipped sitemap index with thousands of children: 1,000 URLs in 1.5 s, 103 MB) andexample.com(no sitemap: one free error row in 0.2 s). None of them blocked the run; every response was HTTP 200. - If a site does block datacenter IPs, set
proxyConfiguration(an Apify proxy or your own). The actor never retries forever: 408/429/5xx responses are retried twice with backoff, a 404 is not retried, a stalled connection is dropped after 30 seconds without data, and any sitemap that still fails becomes a free error row. - Size guardrails: 100 MB is the decompressed ceiling for a single sitemap (Google's own limit
is 50 MB), 5 levels of nested indexes,
maxSitemaps200 by default, andmaxUrls1,000 by default so a first run costs cents. - Redirects: sitemap and
robots.txtrequests follow up to 5 redirects;statusCodechecks do not.
What it does not do
- It does not crawl your pages: it reads sitemaps. A site with no sitemap has no URLs to return.
- Sitemaps that only exist behind JavaScript rendering are not discovered (sitemap files are XML, not rendered pages).
checkStatusis a HEAD request (GET only when the server refuses HEAD), so it reports the HTTP status, not the page content, its title or whether the page is soft-404 content.- hreflang alternates are reported on the row of the URL that declares them; they are not expanded into extra rows, and they are not charged as URLs.
- Sitemaps are decoded as UTF-8, which the sitemap protocol requires; a sitemap served in a legacy 8-bit encoding (ISO-8859-1) would deliver mangled accented characters in its URLs.
- Sitemap entries are reported as published. If a sitemap lists a URL that has since been deleted,
the row still appears — use
checkStatusto see the 404.
Development
npm installnpm test # offline: node --test, fixtures only, no networknode test/live.mjs https://blog.apify.com 25 # live smoke test, not part of npm testnode test/live.mjs https://blog.apify.com 10 --check-status
The offline suite covers sitemap dialects (test/fixtures/), chunk boundaries, gzip, plain text,
50,000-URL documents, the discovery and traversal rules, the billing order (push before charge,
failed pushes never charged, error rows free, cap counted in delivered rows), input validation and
the manifest/ dataset-schema limits Apify enforces at deploy time. test/live.mjs runs the real
fetch-and-parse path against one live site and prints the row count, one full row and peak memory;
it exits non-zero when nothing is delivered.
Run monitoring and privacy
If a run fails or times out, this Actor tells its developer the run ID and status, so we can fix problems fast. It never sends your input, your results or your account data.