Sitemap URL Extractor

Returns: one row per page URL with url, lastmod, changefreq, priority, images, alternates, news, sitemapUrl. Input: {"startUrls": [{"url": "https://www.gov.uk"}, {"url": "https://nodejs.org/en/learn"}], "maxItems": 1000} Best for: crawler and RAG seed lists, SEO audits of indexable pages, monitoring new or changed URLs on a site.

{
    "url": "https://www.gov.uk/",
    "site": "www.gov.uk",
    "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
    "parentSitemapUrl": "https://www.gov.uk/sitemap.xml",
    "discoveredVia": "robots",
    "sitemapDepth": 1,
    "priority": 0.5
}

What does Sitemap URL Extractor do?

Sitemap URL Extractor is a sitemap scraper for any website. Give it a site root, a bare domain, any page URL on the site, a robots.txt URL or a direct sitemap URL, and it returns every page URL the site publishes in its XML sitemaps, one dataset row per URL. Each input is read by what it contains, not by its file extension: a body that is a sitemap or feed is walked directly, a robots.txt body gives its Sitemap: lines, and any other page (an article, a category, a 404) sends the Actor to its site to find the sitemaps. Discovery tries, in order, the Sitemap: lines in robots.txt, 18 common locations such as /sitemap.xml, /sitemap_index.xml, /sitemap.xml.gz and /wp-sitemap.xml (on the site root and under the first path segment of a page URL, so https://nodejs.org/en/learn also checks /en/sitemap.xml), a <link rel="sitemap"> tag or a sitemap link on the homepage, and finally the www. or bare-host twin of the domain.

It walks sitemap indexes level by level, reads gzip files (.xml.gz), plain-text sitemaps and RSS or Atom feeds, and keeps the extra data that sitemap extensions carry: lastmod, changefreq and priority, image entries, video entries, Google News entries and hreflang language alternates. Every row also says which sitemap file listed the URL, which index listed that file, and how the root sitemap was found.

Filters let you keep only the URLs you need: a URL regex (/products/|/blog/), an exclude regex, a regex for which child sitemaps to follow, and a "modified on or after" date. An optional page check sends a HEAD request to each returned URL and stores its HTTP status. Sitemaps are public files meant for crawlers, so no login is needed.

What data can you get from a sitemap?

Field	Description	Example
id	The page URL; unique per site after deduplication	https://nodejs.org/en/download/current
url	Page URL exactly as written in the sitemap's <loc>	https://nodejs.org/en/download/current
site	Hostname of the input this row belongs to	nodejs.org
sitemapUrl	Sitemap file that listed the URL	https://nodejs.org/sitemap.xml
parentSitemapUrl	Sitemap index that listed sitemapUrl (null for root sitemaps)	https://www.gov.uk/sitemap.xml
sitemapLastmod	<lastmod> of the sitemap file in its parent index	2026-10-12T02:50:02+00:00
discoveredVia	How the root sitemap was found: robots, common_path, html_link or input	robots
sitemapDepth	0 for a root sitemap, +1 per index level	1
lastmod	The <lastmod> text as written (RSS pubDate, Atom updated)	2026-10-13T03:03:31+02:00
lastmodIso	lastmod converted to UTC ISO 8601	2026-10-13T01:03:31.000Z
changefreq	<changefreq> value	always
priority	<priority> as a number	0.5
images	Image entries [{url, title, caption}]	see the output example
imageCount	Number of image entries	1
videos	Video entries [{title, description, thumbnailUrl, contentUrl, playerUrl, durationSeconds, publicationDate}]	see the output example
news	Google News entry {publicationName, language, publicationDate, title, keywords}	see the output example
alternates	hreflang alternates [{hreflang, url}]	[{"hreflang": "es", "url": "https://nodejs.org/es/download/current"}]
alternateCount	Number of hreflang alternates	13
pageStatus	Final HTTP status of the page when checkPageStatus is on	200
scrapedAt	When the row was created (UTC)	2026-10-13T01:16:00.051Z
sourceUrl	The sitemap file the row was read from	https://www.lemonde.fr/sitemap_news.xml

Fields a sitemap doesn't contain are null (many sites leave out changefreq and priority; Apple's newsroom sitemap has no lastmod). imageCount and alternateCount are always numbers.

How to use Sitemap URL Extractor

1. Open Sitemap URL Extractor in Apify Console and go to the Input tab.
2. Add one or more sources to Websites or sitemap URLs: https://www.gov.uk, https://www.bbc.com/mundo, https://developer.mozilla.org/robots.txt or https://www.lemonde.fr/sitemap_news.xml. Bare domains such as allbirds.com go in the urls or domains field, because startUrls entries must be full URLs.
3. Set Maximum results (all sites). If you give several sites, also set Maximum URLs per site so one big site doesn't use the whole budget.
4. Optionally add filters: a URL regex, an exclude regex, a child-sitemap regex or a "modified on or after" date.
5. Click Start. Rows appear in the dataset as each sitemap file is read.
6. Download the results as JSON, CSV, Excel, XML or HTML, or read them through the API.

A few tips. Sites are processed in the order you list them, and maxItems counts across all of them. Two inputs that point at the same site (http://nodejs.org and nodejs.org/en, say) return each URL once; the log says how many URLs of the second input were already returned for the first. Large sites can publish thousands of sitemap files; maxSitemapsPerSite (default 500) and maxDepth (default 5) keep a run bounded. For a Shopify store, sitemapUrlPattern: "products" reads only the product sitemaps; for a multilingual site, a pattern like /es/ picks one locale. The key-value store's OUTPUT record lists, for every input, how its sitemaps were found, how many files were read and every file that failed (404, blocked, too large, or not a sitemap).

Input

Field	Type	Required	Description
startUrls	array of {url}	One source field is needed	Site roots, page URLs, robots.txt URLs or direct sitemap URLs (full URLs with a scheme; bare domains go in urls or domains)
urls	array of strings	No	Same forms as startUrls, as plain strings; merged with it
sitemapUrls	array of strings	No	Direct sitemap or sitemap index URLs; merged with startUrls
websites	array of strings	No	Website or page URLs; merged with startUrls
domains	array of strings	No	Bare domains (https:// is added); merged with startUrls
url, startUrl, website, domain, sitemapUrl	string	No	Single-value forms of the fields above, for API calls that pass one target
maxItems	integer	No	Maximum rows across all sites, 0 = unlimited, default 1000
maxUrlsPerSite	integer	No	Maximum rows per input site, 0 = no per-site limit, default 0
maxSitemapsPerSite	integer	No	Maximum sitemap files fetched per site, default 500
maxDepth	integer	No	Sitemap index levels to follow, default 5
maxSitemapBytes	integer	No	Largest uncompressed sitemap file read, default 60 MB
includeUrlPattern	string	No	Keep only page URLs matching this regex (case-insensitive)
excludeUrlPattern	string	No	Drop page URLs matching this regex
sitemapUrlPattern	string	No	Follow only child sitemaps whose URL matches this regex
modifiedAfter	string	No	Keep only URLs whose lastmod is on or after this ISO date
deduplicate	boolean	No	One row per URL per site, default true
checkPageStatus	boolean	No	HEAD every returned URL and fill pageStatus, default false
requestTimeoutSecs	integer	No	Timeout per sitemap download, default 60
proxyConfiguration	object	No	Proxy settings, default Apify Proxy (datacenter)

{
    "startUrls": [
        { "url": "https://nodejs.org" },
        { "url": "https://www.gov.uk" }
    ],
    "urls": ["allbirds.com", "https://www.bbc.com/mundo"],
    "maxItems": 450,
    "maxUrlsPerSite": 150,
    "includeUrlPattern": "",
    "modifiedAfter": "2026-01-01",
    "checkPageStatus": false,
    "proxyConfiguration": { "useApifyProxy": true }
}

Output

Each row is one page URL. These two rows come from local test runs: a French news sitemap given directly, and the hreflang-rich sitemap that nodejs.org lists in its robots.txt.

[
    {
        "id": "https://www.lemonde.fr/culture/article/2026/10/13/l-acteur-rj-mitte-revele-par-son-role-du-fils-de-walter-white-dans-breaking-bad-est-mort-a-l-age-de-34-ans_6792909_3246.html",
        "url": "https://www.lemonde.fr/culture/article/2026/10/13/l-acteur-rj-mitte-revele-par-son-role-du-fils-de-walter-white-dans-breaking-bad-est-mort-a-l-age-de-34-ans_6792909_3246.html",
        "site": "www.lemonde.fr",
        "sitemapUrl": "https://www.lemonde.fr/sitemap_news.xml",
        "parentSitemapUrl": null,
        "sitemapLastmod": null,
        "discoveredVia": "input",
        "sitemapDepth": 0,
        "lastmod": "2026-10-13T03:03:31+02:00",
        "lastmodIso": "2026-10-13T01:03:31.000Z",
        "changefreq": null,
        "priority": null,
        "images": [
            {
                "url": "https://img.lemde.fr/2026/10/13/1666/0/3334/1667/1440/720/60/0/c9ef90b_upload-1-pvlygytpeh5e-063-2245066351.jpg",
                "title": null,
                "caption": "RJ Mitte assiste à la première mondiale de la série « Pluribus » d’Apple TV à Los Angeles (Californie), le 4 novembre 2025."
            }
        ],
        "imageCount": 1,
        "videos": null,
        "news": {
            "publicationName": "Le Monde",
            "language": "fr",
            "publicationDate": "2026-10-13T03:03:31+02:00",
            "title": "L’acteur RJ Mitte, révélé par son rôle dans « Breaking Bad », est mort à l’âge de 34 ans",
            "keywords": null
        },
        "alternates": null,
        "alternateCount": 0,
        "pageStatus": null,
        "scrapedAt": "2026-10-13T01:16:00.051Z",
        "sourceUrl": "https://www.lemonde.fr/sitemap_news.xml"
    },
    {
        "id": "https://nodejs.org/en/download/current",
        "url": "https://nodejs.org/en/download/current",
        "site": "nodejs.org",
        "sitemapUrl": "https://nodejs.org/sitemap.xml",
        "parentSitemapUrl": null,
        "sitemapLastmod": null,
        "discoveredVia": "robots",
        "sitemapDepth": 0,
        "lastmod": "2026-10-12T12:24:43.918Z",
        "lastmodIso": "2026-10-12T12:24:43.918Z",
        "changefreq": "always",
        "priority": null,
        "images": null,
        "imageCount": 0,
        "videos": null,
        "news": null,
        "alternates": [
            { "hreflang": "es", "url": "https://nodejs.org/es/download/current" },
            { "hreflang": "fr", "url": "https://nodejs.org/fr/download/current" },
            { "hreflang": "ja", "url": "https://nodejs.org/ja/download/current" }
        ],
        "alternateCount": 13,
        "pageStatus": null,
        "scrapedAt": "2026-10-13T01:17:17.944Z",
        "sourceUrl": "https://nodejs.org/sitemap.xml"
    }
]

The nodejs.org row lists 13 alternates in the dataset; three are shown here. The Overview tab shows the columns url, site, lastmod, changefreq, priority, imageCount, alternateCount, pageStatus, discoveredVia and sitemapUrl.

How much does it cost to extract sitemap URLs?

Sitemap URL Extractor is charged per result: one result is one page URL row in the dataset. You pay only for rows returned, and maxItems caps how many there can be. Sitemap files that fail, sites without a sitemap and URLs dropped by your filters are not charged. See the Pricing tab for the current price.

Platform usage is light because sitemaps are plain files fetched over HTTP, without a browser. The default datacenter proxy is enough for most sites; if you switch to residential proxies, remember that one sitemap file can be tens of megabytes.

Use with AI agents (MCP)

Connect Sitemap URL Extractor to Claude, Cursor or any MCP client through the Apify MCP server:

{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com/?tools=axiomworks/sitemap-url-extractor"
        }
    }
}

Example prompts:

1. "List all product URLs in the sitemap of allbirds.com and tell me how many were modified this month."
2. "Get the sitemap URLs of nodejs.org and show which pages have Spanish and Japanese alternates."
3. "Extract the 200 most recent URLs from the Le Monde news sitemap with their headlines."
4. "Check whether the pages in yoast.com's page sitemap all return HTTP 200."

Agents should pass the target in startUrls (or urls) and set maxItems to what they need.

Integrations and API

Run Sitemap URL Extractor from your own code with the Apify API. Set APIFY_TOKEN to your Apify API token (send it in the Authorization header, never in the URL).

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("axiomworks/sitemap-url-extractor").call(
    run_input={"startUrls": [{"url": "https://www.gov.uk"}], "maxItems": 500}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["lastmod"])

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });
const run = await client.actor('axiomworks/sitemap-url-extractor').call({
    urls: ['allbirds.com'],
    includeUrlPattern: '/products/',
    maxItems: 200,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

cURL

curl -X POST "https://api.apify.com/v2/acts/axiomworks~sitemap-url-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sitemapUrls": ["https://www.lemonde.fr/sitemap_news.xml"], "maxItems": 100}'

You can also connect the Actor to Make, Zapier, n8n, Google Sheets or webhooks from the Integrations tab, and schedule it; a scheduled run with modifiedAfter set to a recent date returns only the pages that changed.

FAQ

Which inputs work? A site root (https://www.gov.uk), a bare domain with or without www. (in urls, websites or domains; startUrls entries need a scheme), an http:// URL, any page URL on the site (https://nodejs.org/en/learn), a robots.txt URL, or a direct sitemap URL of any shape: .xml, .xml.gz, .txt, an RSS or Atom feed, a Shopify file with a ?from=...&to=... query string, or a sitemap without an extension. A URL with a path is fetched once and read by content: a sitemap or feed body becomes the root sitemap, a robots.txt body gives its Sitemap: lines, and any other page starts discovery on its site, so https://nodejs.org/en/learn returns the sitemaps of nodejs.org with site set to nodejs.org.

Why did a site return 0 results? Some sites publish no sitemap at all: no Sitemap: line in robots.txt, nothing at the 18 common paths, no sitemap link on the homepage and nothing on the www. or bare-host twin. The run still ends successfully, and the status message and the OUTPUT record name the site. A domain that doesn't resolve is reported the same way within seconds, and the other sites in the run still return their rows.

What happens with blocked or broken sitemap files? A sitemap that answers 403, 429 or a 5xx error is retried twice on a new proxy IP. If it still fails, or returns 404, an HTML page or a file larger than maxSitemapBytes, it is recorded in the OUTPUT record's failures list and the walk goes on with the next file. The Actor never tries to solve a CAPTCHA or bot challenge; files behind one stay listed as failures.

Does it read gzip and huge sitemaps? Yes. .xml.gz files are decompressed, also when a server labels them as XML. Files up to 60 MB uncompressed are read by default; change maxSitemapBytes as needed.

Why are some URLs missing when a site lists them twice? With deduplicate on (the default), a URL that appears in several sitemaps of the same site is returned once, from the first sitemap that listed it. Turn it off to get one row per listing.

Can I check that the pages are live? Turn on checkPageStatus. Each returned URL gets a HEAD request (a GET without reading the body when HEAD is refused), redirects are followed, and the final status goes into pageStatus. This adds one request per URL, so it is off by default.

Is it legal to scrape sitemaps?

Sitemaps are public files that websites publish so that search engines and other crawlers can find their pages. Sitemap URL Extractor reads only those public files (and, if you ask for it, the HTTP status of the listed pages); it doesn't log in or open any page behind a login. Some sitemaps, such as news sitemaps, contain headlines or image captions; you are responsible for how you use the data and for following the laws that apply to you, such as copyright and data-protection rules. This is not legal advice.

Sitemap URL Extractor is an independent tool. It is not affiliated with, endorsed by or sponsored by any of the websites whose sitemaps it reads.

Related Actors

Actor	What it does
Semrush SEO Metrics Scraper	Domain-level SEO metrics for the sites whose pages you list here
Local Business Email Finder	Finds public contact emails for businesses and their websites

Feedback

Found a bug or need a field that isn't here? Open an issue on the Issues tab and we'll look at it.