Sitemap Extractor: All URLs from XML Sitemaps

Get every URL a website lists in its XML sitemaps: one dataset row per URL with lastmod, changefreq, priority, the sitemap it came from and counts of image, video, news and hreflang entries. Give a site URL and the Actor finds the sitemaps itself, or give a sitemap URL. No browser, no proxy, no key.

Price: $0.40 per 1,000 URLs ($0.0004 per URL row), plus Apify's default Actor-start event. Sitemap files, robots.txt and error rows are free.

• Finds the sitemaps for you: reads the Sitemap: lines of robots.txt, then tries /sitemap.xml and /sitemap_index.xml.
• Follows sitemap indexes to any depth up to 5 (default 3), reads .xml.gz files and plain-text sitemaps.
• Filters before you pay: keep only URLs containing a text or * pattern, skip others, keep only URLs modified on or after a date. Filtered URLs and duplicates are not charged.
• Streams large sitemaps: a 50,000-URL file is read as it downloads and rows are written in batches; 50,000 URLs from apify.com's sitemap index took about 9 s.
• Quick start: press Start. The prefilled input reads 20 URLs from https://crawlee.dev/ (Apify's open-source crawling library; its robots.txt allows all crawlers). That is 2 requests, about 3 seconds and $0.008.

Polite by design. robots.txt is obeyed for the sitemap requests themselves, Crawl-delay is honoured in full, at most one request per second per host, an honest User-Agent, and blocks (401, 403, 429, challenge pages) are reported as free rows, never worked around.

What it does

1. For each start URL: a site URL (https://crawlee.dev/, https://example.org/blog/) is looked up at the origin. The Actor reads robots.txt; its Sitemap: lines are the sitemaps. Without such a line it tries /sitemap.xml, then /sitemap_index.xml. A sitemap URL (path ending in .xml, .xml.gz, .gz or .txt, or containing sitemap) is read directly.
2. Each sitemap is downloaded and parsed as it streams in. A <urlset> gives URL rows; a <sitemapindex> gives child sitemaps, read depth first in file order down to Max index depth.
3. Filters, then de-duplication, decide which URLs become rows. Rows are written every 1,000 URLs or 30 seconds, and the run stops cleanly at Max URLs, at your max charge, or at a safety cap.

Every request, robots.txt included, is rate-limited per host (one per second, or slower if robots.txt asks for it). Sitemaps on other hosts (for example a CDN listed in robots.txt) are allowed; the robots.txt of that host is read and obeyed too. Social-media and Google hosts (YouTube, Facebook, Instagram, X, TikTok, LinkedIn, Reddit, Google domains and similar) are never contacted: such start URLs, and redirects to them, give a free row with errorCode: "platform".

Input

Field	What it does
Site or sitemap URLs (startUrls)	Up to 100. A site URL is discovered through robots.txt, then /sitemap.xml and /sitemap_index.xml. A sitemap URL is read directly. A bare host (crawlee.dev) gets https://.
Max URLs (maxUrls)	URL rows for the whole run. Default 50,000, hard cap 200,000 (100,000 when the run has less than 512 MB of memory). The run stops cleanly when it is reached (stoppedBy: "max-urls").
Only URLs containing (includePattern)	Optional text, case-insensitive; * means "anything". /blog/ keeps blog URLs; /products/*/reviews keeps product review pages. No regular expressions.
Skip URLs containing (excludePattern)	Optional, same rules, for example /tag/ or ?page=. Wins over the include pattern.
Modified on or after (lastmodAfter)	Optional date, YYYY-MM-DD or a full date-time with zone. Keeps URLs whose lastmod is on or after it. URLs without a lastmod are dropped while this is set. Child sitemaps whose index entry has an older lastmod are not downloaded at all.
Max index depth (maxDepth)	Levels of sitemap indexes followed. 0 reads only the sitemaps you give or robots.txt lists; default 3; maximum 5. Deeper child sitemaps are counted in one free depth-cap row.
Remove duplicates (dedupe)	Default on: a URL in several sitemaps is output and charged once (the first sitemap wins). Off: one row per listing.

Example inputs

1. All URLs of a site (sitemaps found automatically):

{ "startUrls": ["https://crawlee.dev/"] }

2. Only blog posts changed this year, from a known sitemap index:

{
  "startUrls": ["https://www.example.org/sitemap_index.xml"],
  "includePattern": "/blog/",
  "excludePattern": "/blog/tag/",
  "lastmodAfter": "2026-01-01"
}

3. Several sites, first 1,000 URLs each run, top-level sitemaps only:

{
  "startUrls": ["https://crawlee.dev/", "https://www.example.org/", "https://shop.example.com/sitemap.xml.gz"],
  "maxUrls": 1000,
  "maxDepth": 0
}

Output

One row per URL:

{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "videoCount": 0,
  "newsCount": 0,
  "alternateCount": 0,
  "sourceSitemap": "https://crawlee.dev/sitemap.xml",
  "site": "https://crawlee.dev",
  "status": "ok"
}

• lastmod and changefreq are given exactly as the sitemap writes them (lastmod is a W3C date or date-time such as 2026-10-01T08:00:00+00:00); null when absent. priority is a number from 0 to 1, or null.
• imageCount, videoCount, newsCount and alternateCount count the Google image, video and news extension entries and the xhtml:link rel="alternate" hreflang entries of that URL (0 when there are none). Their contents (image URLs, titles) are not output.
• url is resolved to an absolute URL; entries with an invalid or non-http(s) <loc>, or longer than 2,048 characters, are skipped and counted in RUN_SUMMARY.invalidEntries.

Problems are free rows with status: "error":

{
  "url": "https://www.example.org/sitemap-old.xml",
  "status": "error",
  "errorCode": "http",
  "message": "The server answered HTTP 404.",
  "site": "https://www.example.org",
  "sourceSitemap": "https://www.example.org/sitemap_index.xml",
  "httpStatus": 404
}

Error codes

errorCode	Meaning
platform	A social-media, video or Google host (also when reached by a redirect). Never contacted.
invalid-url, ssrf	Not an http(s) URL, or a private, loopback or internal address.
dns, network, timeout, redirects	Host not found, connection failed, no data for 30 s or a file took over 5 minutes, more than 5 redirects.
http	Any other non-2xx answer (httpStatus says which, for example 404).
refused	401, 403, 429, 503, 999 or a bot challenge. Not retried, not worked around. After 2 refusals the host's other sitemaps are skipped (host-skipped, one summary row with count).
robots-blocked	robots.txt disallows the sitemap's path for this Actor. Not requested.
robots-unavailable	robots.txt answered 5xx or 429, or could not be reached, twice. No sitemap of that site is fetched without its rules (RFC 9309). A 4xx robots.txt means "no rules".
slow-host	robots.txt asks for a Crawl-delay over 30 s, or honouring the delays would take more than 10 minutes of waiting in this run. Skipped; the delay is never shortened.
no-sitemap	A site URL with no Sitemap: line in robots.txt and no sitemap at /sitemap.xml or /sitemap_index.xml.
not-sitemap	The file is an HTML page, an RSS or Atom feed (not supported), another XML document or empty.
malformed-xml	The XML is broken or ends early (a tag left open), or a tag, comment or CDATA section is too long. URLs read before the problem are kept (and charged); the message says how many.
gzip-error, too-large	A damaged .gz file; a file over 60 MB downloaded or 60 MB once decompressed (gzip bombs stop here). URLs read before are kept.
entry-cap	A file with more than 50,000 entries (the protocol limit); the rest of it was not read.
depth-cap, sitemap-cap	Summary rows (with count): child sitemaps deeper than Max index depth, or beyond 500 sitemap files per run.
internal	An unexpected error in this Actor, not a problem with your site.

RUN_SUMMARY (key-value store record)

URL rows and charged events, error rows, sitemap files read and failed, indexes read, entries read, duplicates, entries filtered by pattern or date, invalid entries, child sitemaps skipped (by date, depth, cap or refusing host), bytes downloaded (and the part that led to no URL row), seconds spent waiting for Crawl-delay, errors by code, and stoppedBy: finished, max-urls, max-charge, unpaid-work-cap, time-limit, migrating, aborted or error.

Pricing

Pay per event: $0.0004 per URL row (event url-extracted), plus Apify's standard Actor-start event. No other fee from this Actor.

URL rows	Cost
20 (prefilled example)	$0.008
1,000	$0.40
10,000	$4.00
50,000 (default Max URLs)	$20.00
200,000 (hard cap; needs 512 MB)	$80.00

Free: robots.txt and sitemap files, error rows, URLs removed by your filters, duplicates (with Remove duplicates on) and invalid entries. Set Max URLs, or Max total charge in the run options, to cap spending: the run checks the budget before each sitemap and before each write, and never writes a row it cannot charge. Apify subscription discounts apply on top.

To keep free work bounded, a run stops (stoppedBy: "unpaid-work-cap") after 50 failed sitemaps or start URLs, 100 MB downloaded for sitemaps that gave no URL row, or 1,000,000 sitemap entries that were filtered out, duplicates or invalid. Everything found up to then is saved.

Use it via API, n8n or Make

Get an API token in Apify Console under Settings, API & Integrations, and keep it in an environment variable (APIFY_TOKEN below). Never paste it into shared code.

cURL (starts a run, waits and returns the rows; the synchronous endpoint waits up to 5 minutes, so use it for small runs and the clients below for big ones):

curl -X POST "https://api.apify.com/v2/acts/ataism~sitemap-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/"], "maxUrls": 100}'

Python (pip install "apify-client>=3"):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("ataism/sitemap-extractor").call(run_input={
    "startUrls": ["https://crawlee.dev/"],
    "includePattern": "/docs/",
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    if row["status"] == "ok":
        print(row["url"], row["lastmod"])

(With apify-client versions before 3.0, call() returns a dict: use run["defaultDatasetId"].)

JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('ataism/sitemap-extractor').call({
    startUrls: ['https://crawlee.dev/'],
    lastmodAfter: '2026-01-01',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const row of items.filter((r) => r.status === 'ok')) console.log(row.url, row.lastmod);

n8n:

1. Add a Schedule trigger (for example weekly) or any trigger that gives you a site URL.
2. Add the Apify node, operation Run an Actor and get dataset, Actor ataism/sitemap-extractor, input {"startUrls": ["{{ $json.siteUrl }}"], "lastmodAfter": "{{ $now.minus({days: 7}).toFormat('yyyy-MM-dd') }}"}.
3. Send the new or changed URLs to a sheet, a crawler or Slack; filter on status = ok.

Make:

1. Add the Apify module Run an Actor, choose ataism/sitemap-extractor, put your site URL in the input JSON and set Run synchronously to Yes.
2. Add the Apify module Get Dataset Items and map the run's default dataset ID (one bundle per URL).
3. Send the rows to Google Sheets, Airtable, a database or a webhook.

Privacy and terms

• The Actor reads robots.txt and the sitemap files of the sites you enter, nothing else: it never opens the pages the sitemaps list. It outputs the URLs and dates the sitemaps contain. A sitemap that puts personal data into its URLs (for example user profile URLs) puts it into your dataset too.
• Nothing is sent to any AI or third-party service. No logins, no proxies, no CAPTCHA solving.
• Your dataset belongs to your Apify account and follows your retention settings.
• You are responsible for having the right to read and use the sites you enter (their terms, your agreement with the owner). Sitemaps are published for crawlers, and the Actor obeys robots.txt, but it does not know your agreement with the site.
• User-Agent: sitemap-extractor-actor/0.1 (https://apify.com/ataism/sitemap-extractor). In robots.txt, User-agent: sitemap-extractor-actor targets this Actor (exact match; otherwise the * group applies).

Safety

Only http and https to public addresses: private, loopback, link-local and internal addresses and other schemes are blocked, also after every redirect (at most 5) and at connection time. Each sitemap is capped at 60 MB downloaded and 60 MB decompressed, 50,000 entries and 5 minutes; robots.txt at 512 KB. The XML parser is a streaming scanner with bounded buffers: it builds no tree (deep nesting costs nothing), never expands DTD entities (no "billion laughs"), and caps every tag, value and comment.

Limits and honest caveats

• Limits: 100 start URLs, 200,000 URL rows (100,000 below 512 MB of memory), 500 sitemap files, 5 index levels and 60 minutes per run; 50,000 entries per sitemap file.
• Default memory is 256 MB, enough for the default 50,000 URLs. Below 512 MB a run is capped at 100,000 URLs (with a warning in the log); for up to 200,000 URLs in one run, start it with 512 MB (the de-duplication list and buffers grow with the URL count).
• Only what the sitemaps say: a page missing from the sitemaps is not found (this is not a crawler), and lastmod is only as accurate as the site makes it.
• RSS and Atom feeds used as sitemaps are not supported (not-sitemap). Image, video and news extensions are counted, not extracted.
• With Modified on or after set, URLs without lastmod are dropped, and child sitemaps whose index says they are older are skipped without downloading them. A site with wrong index dates can make you miss URLs; leave the date empty if in doubt.
• Rows are written in batches (every 1,000 URLs or 30 s). If the platform migrates the run, buffered rows are written and progress is saved; the resumed run reads the dataset and never writes or charges a URL twice.
• Up to one request per second per host, slower when robots.txt asks for a Crawl-delay (honoured in full up to 30 s; at most 10 minutes of such waiting per run). A site with many small sitemap files and a long delay therefore takes long, or ends in free slow-host rows.

FAQ

Do I pay for sitemap files or robots.txt? No. Only URL rows (status: "ok") are charged.

The site has no sitemap. What happens? You get one free no-sitemap row for it. Nothing is charged.

Why is a sitemap robots-blocked? The site's robots.txt disallows that path for crawlers in general or for this Actor. The Actor respects that; ask the site owner or use a sitemap they allow.

Can I get only new pages each week? Yes: schedule a run with Modified on or after set to last week's date (see the n8n example), or compare runs on url.

Does the default run work without setup? Yes. It reads 20 URLs from https://crawlee.dev/ (2 requests, about 3 seconds, $0.008).

Open-source components

The Actor uses the Apify SDK (apify 3.7.2, Apache-2.0). The sitemap scanner, robots.txt parser, filters and fetching are plain code in this Actor, with no other dependencies.