Sitemap URL Extractor

Extract clean URL inventories from XML sitemaps and sitemap indexes.

Use this actor to turn public sitemap files into a structured dataset of URLs with source sitemap metadata, crawl depth, last modified dates, change frequency, priority, HTTP status, and discovery timestamps.

Ready-to-run examples

• Compare Multiple Sitemap Url Inventories
• Extract Product Urls From Sitemap
• Build Docs Crawler Seed List
• Extract Blog Urls From Sitemap
• Extract Apify Sitemap Urls

View all ready-to-run examples

What does Sitemap URL Extractor do?

Sitemap URL Extractor reads website sitemap files and saves every discovered URL as a dataset item.

It supports both direct sitemap URLs and website root URLs.

If you enter a root such as https://apify.com, the actor tries common sitemap locations automatically.

If the sitemap is a sitemap index, the actor follows child sitemaps recursively until it reaches your limits.

You can filter URLs with include and exclude patterns.

You can deduplicate URLs across many sitemap files.

You can keep extraction limited to the same hostname as your original input.

Who is it for?

• 🔎 SEO teams auditing indexable URL inventories.
• 🧪 Technical SEO consultants checking sitemap coverage before crawls.
• 🕷️ Crawler operators seeding downstream crawling jobs with verified URLs.
• 📈 Growth engineers monitoring new pages by section.
• 🧰 Data teams collecting public URL lists for enrichment pipelines.
• 🧾 Content teams exporting article or documentation URLs.
• 🛠️ QA teams building URL regression test sets.

Why use this actor?

Sitemaps are one of the fastest ways to understand a public website structure.

Manual sitemap inspection gets slow when a site has dozens or hundreds of sitemap files.

This actor automates the repetitive parts:

• 🧭 Discover sitemap indexes.
• 🔁 Follow recursive sitemap files.
• 🧹 Normalize and deduplicate URLs.
• 🎯 Keep only sections you care about.
• 📦 Export results in JSON, CSV, Excel, XML, RSS, or HTML from Apify datasets.

Typical use cases

• Build a seed list for a crawler.
• Export all blog post URLs from a site.
• Compare sitemap URL counts over time.
• Find URLs that changed recently by using lastmod.
• Audit whether product, category, help, or documentation pages are present.
• Feed URLs into rank tracking, monitoring, or enrichment workflows.
• Create a clean URL list for link checking.

Input overview

The main input is startUrls.

You can provide one or more sitemap URLs.

You can also provide website roots.

Example sitemap input:

{
  "startUrls": [
    { "url": "https://apify.com/sitemap.xml" }
  ],
  "maxUrls": 100,
  "maxSitemaps": 25
}

Example root input:

{
  "startUrls": [
    { "url": "https://apify.com" }
  ],
  "maxUrls": 100,
  "maxSitemaps": 25
}

Input settings

Setting	JSON key	Type / default	Description
Sitemap or website URLs	startUrls	array, default [{"url":"https://apify.com/sitemap.xml"}]	Enter XML sitemap URLs such as https://example.com/sitemap.xml or website roots such as https://example.com. Root URLs try common sitemap paths automatically.
Maximum URLs	maxUrls	integer, default 20	Stop after saving this many URL records.
Maximum sitemap files	maxSitemaps	integer, default 10	Maximum number of sitemap files to fetch, including recursively discovered child sitemaps.
Include URL patterns	includePatterns	array, default []	Optional regular expressions. If provided, only URLs matching at least one include pattern are saved. Example: /blog/|/docs/
Exclude URL patterns	excludePatterns	array, default []	Optional regular expressions for URLs to skip. Example: /tag/|/author/
Same domain only	sameDomainOnly	boolean, default true	Keep only sitemap files and URLs on the same hostname as the original input URL.
Deduplicate URLs	dedupe	boolean, default true	Save each normalized URL only once across all processed sitemaps.
Proxy configuration	proxyConfiguration	object, default {"useApifyProxy":false}	Optional proxy settings. Most public sitemaps work without proxies; enable Apify Proxy only for sites that block direct requests.

Output data

Each dataset item is an extracted sitemap URL.

Field	Description
url	Normalized extracted URL.
sourceSitemapUrl	Sitemap file where the URL was found.
sitemapDepth	Recursive sitemap depth. Root sitemap is depth 0.
loc	URL from the sitemap <loc> value.
lastmod	Optional sitemap last modified value.
changefreq	Optional sitemap change frequency.
priority	Optional sitemap priority as a number.
contentType	HTTP content type returned for the sitemap file.
statusCode	HTTP status code returned for the sitemap file.
discoveredAt	ISO timestamp when the URL was saved.

Example input

{
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": [],
  "sameDomainOnly": true,
  "dedupe": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Example output

{
  "url": "https://apify.com/store",
  "sourceSitemapUrl": "https://apify.com/sitemap.xml",
  "sitemapDepth": 0,
  "loc": "https://apify.com/store",
  "lastmod": "2026-07-01",
  "changefreq": null,
  "priority": null,
  "contentType": "application/xml",
  "statusCode": 200,
  "discoveredAt": "2026-07-03T20:30:00.000Z"
}

Output fields

JSON key	Label	Type	Description
url	URL	string	Normalized extracted URL.
loc	Sitemap loc	string	URL from the sitemap
sourceSitemapUrl	Source sitemap URL	string	Sitemap file where the URL was found.
sitemapDepth	Sitemap depth	integer	Recursive sitemap depth from the original input.
lastmod	Last modified	string / null	Optional sitemap last modified value.
changefreq	Change frequency	string / null	Optional sitemap change frequency value.
priority	Priority	number / null	Optional sitemap priority value.
contentType	Content type	string / null	HTTP content type of the source sitemap.
statusCode	HTTP status	integer / null	HTTP status code of the source sitemap.
discoveredAt	Discovered at	string	ISO timestamp when the URL was saved.

Pricing

This Actor uses Apify pay-per-event pricing. The prices below come from the current Actor pricing configuration. Apify public plans map to Store discount tiers, so the table shows both the user-facing plan context and the pricing tier name. The final price shown in Apify depends on the user account plan and any custom agreement.

Event	What is charged	Price
start	One-time fee charged when a run starts. Covers fixed startup cost (init, first HTTP setup).	$0.005

Event	What is charged	Free / no discount	Starter / Bronze	Scale / Silver	Business / Gold	Custom / Platinum	Custom / Diamond
result	Charged per sitemap URL extracted.	$0.028 / 1,000	$0.024 / 1,000	$0.019 / 1,000	$0.015 / 1,000	$0.012 / 1,000	$0.01 / 1,000

Apify may also charge platform usage for compute, storage, proxies, or data transfer outside this Actor pricing. Check the Actor run and the Apify Pricing tab for the exact cost shown to your account.

How to run

1. Open the actor on Apify.
2. Add one or more sitemap URLs or website root URLs.
3. Keep the default maxUrls for a quick test.
4. Add include or exclude patterns if you only need part of a site.
5. Run the actor.
6. Export the dataset in your preferred format.

Filtering examples

Save only blog and docs URLs:

{
  "includePatterns": ["/blog/", "/docs/"]
}

Skip tag, author, and category pages:

{
  "excludePatterns": ["/tag/", "/author/", "/category/"]
}

Keep only product URLs:

{
  "includePatterns": ["/products/"]
}

Tips for best results

• Start with a small maxUrls value.
• Use direct sitemap URLs when you know them.
• Use root URLs when you want the actor to try common sitemap paths.
• Keep sameDomainOnly enabled unless you expect cross-domain sitemap entries.
• Use include patterns to reduce noise before exporting data.
• Use exclude patterns to remove archive, tag, or parameterized pages.
• Disable dedupe only when you need to see duplicate sitemap coverage.

Limits and safeguards

maxUrls stops the run after enough URL records are saved.

maxSitemaps stops recursive sitemap discovery after enough sitemap files are fetched.

These limits protect your budget and make large websites easier to test.

Some websites expose millions of URLs across many sitemap files.

For those sites, run section-specific sitemaps or use include patterns.

Handling sitemap indexes

Many websites use a sitemap index.

A sitemap index is a sitemap file that points to other sitemap files.

This actor follows those child sitemap files automatically.

The sitemapDepth field shows how far a URL was from the original sitemap.

Handling website root URLs

When you enter a root URL, the actor tries common sitemap locations.

For example, https://example.com may expand to common sitemap paths such as /sitemap.xml.

If you already know the exact sitemap file, enter it directly for faster runs.

Integrations

Use the extracted URL dataset with:

• Apify crawlers.
• SEO audit tools.
• Link checkers.
• Content inventory spreadsheets.
• Data enrichment workflows.
• Monitoring jobs that compare datasets over time.
• Webhook automations that trigger when a new run finishes.

API usage

Run Sitemap URL Extractor from your own code with the Apify API.

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": []
};

const run = await client.actor('fetch_cat/sitemap-url-extractor').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient
import os

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("fetch_cat/sitemap-url-extractor").call(run_input={
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": []
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

cURL

curl -X POST "https://api.apify.com/v2/acts/fetch_cat~sitemap-url-extractor/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com/sitemap.xml"}],"maxUrls":20,"maxSitemaps":10,"includePatterns":[],"excludePatterns":[]}'

Use with AI agents via MCP

Sitemap URL Extractor can be used by AI assistants through the hosted Apify MCP server.

Claude Code setup

$claude mcp add --transport http apify "https://mcp.apify.com?tools=fetch_cat/sitemap-url-extractor"

Claude Desktop, Cursor, or VS Code JSON config

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/sitemap-url-extractor"
    }
  }
}

Example prompts

• "Run Sitemap URL Extractor with this input JSON and summarize the dataset."
• "Export the latest Sitemap URL Extractor results to a table I can review."
• "Schedule this Actor for monitoring and tell me what changed between runs."

Legality and ethics

This actor reads public sitemap files that websites publish for crawlers.

Only extract data you are allowed to use.

Respect website terms, robots policies, and applicable laws.

Do not use extracted URLs for spam, abuse, or unauthorized access.

FAQ

Can I enter a website URL instead of a sitemap URL?

Yes. You can enter a website root URL and the actor will try common sitemap locations.

Can I extract only one section of a website?

Yes. Use includePatterns and excludePatterns to keep or skip URL paths.

Troubleshooting

Why did I get fewer URLs than expected?

Check maxUrls, maxSitemaps, include patterns, exclude patterns, and sameDomainOnly.

The website may also have multiple sitemap indexes not linked from the first file.

Why did a root URL return no results?

The site may not use a common sitemap path.

Try entering the exact sitemap URL from the site's robots.txt file or documentation.

Why are some metadata fields empty?

lastmod, changefreq, and priority are optional sitemap fields.

Many websites omit them.

Related actors

• Apple App Store Apps Scraper
• Apple App Store Reviews Scraper
• Bing SERP & Search Results Scraper
• DuckDuckGo Search Results Scraper
• Google Ads Transparency Scraper

Support

Report bugs, wrong output, blocked runs, or missing fields from the Actor page. Include the Apify run ID or run URL, your input JSON, what you expected, what the Actor returned, and one reproducible public URL so the issue can be tested quickly.