Sitemap URL Extractor

Extract URL inventories from public XML sitemaps and sitemap indexes with Sitemap URL Extractor. Export URLs, their source sitemap, depth and optional last-modified metadata for SEO audits, crawler seed lists and content inventories.

This extracts sitemap entries, not page content or proof that each listed page is reachable. Website roots try common sitemap paths rather than discovering every sitemap on a site.

Quick start

1. Open the Actor and paste a public XML sitemap URL into Sitemap or website URLs.
2. Start with a small URL limit. For example:

{
  "startUrls": [{"url": "https://apify.com/sitemap.xml"}],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "proxyConfiguration": {"useApifyProxy": false}
}

3. Run the Actor, then open the dataset to review and export the URLs.

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

Ready-to-run examples

Choose an existing public example, review its settings and run it in your own account.

• Extract URLs from an XML sitemap — Export a bounded URL inventory from Apify's sitemap.
• Build a docs crawler seed list — Start from a documentation website root and try common sitemap locations.
• Extract only blog URLs — Keep URLs matching the blog path pattern.

What you get

• URL inventories: One row for each matching sitemap entry, up to your limits.
• Sitemap indexes: Follow child sitemap files and retain their depth and source.
• Useful metadata: Keep optional last-modified dates, change frequency and priority when supplied.
• Focused exports: Include or exclude URL patterns, restrict hostnames and remove duplicate normalized URLs.

Who is it for?

User	Useful workflow
SEO teams	Review sitemap coverage and URL inventories
Crawler operators	Build seed lists for downstream page extraction
Content teams	Export article or documentation URLs
Data teams	Compare exported inventories in their own workflow

Input settings

Setting	JSON key	Type / default	Description
Sitemap or website URLs	startUrls	array, required (form prefill only)	Enter XML sitemap URLs such as https://example.com/sitemap.xml or website roots such as https://example.com. Root URLs try common sitemap paths automatically.
Maximum URLs	maxUrls	integer, default 20	Stop after saving this many URL records.
Maximum sitemap files	maxSitemaps	integer, default 10	Maximum number of sitemap files to fetch, including recursively discovered child sitemaps.
Maximum run time	maxRunSeconds	integer, default 270	Safety cap for requests and saving results. If a run is interrupted, only the matching unfinished checkpoint is resumed. Completed runs clear it.
Include URL patterns	includePatterns	array, default []	Optional regular expressions. If provided, only URLs matching at least one include pattern are saved. Example: /blog/|/docs/
Exclude URL patterns	excludePatterns	array, default []	Optional regular expressions for URLs to skip. Example: /tag/|/author/
Same domain only	sameDomainOnly	boolean, default true	Keep only sitemap files and URLs on the same hostname as the original input URL.
Deduplicate URLs	dedupe	boolean, default true	Save each normalized URL only once across all processed sitemaps.
Proxy configuration	proxyConfiguration	object, default {"useApifyProxy":false}	Optional proxy settings. Most public sitemaps work without proxies. Enable Apify Proxy only for sites that block direct requests.

The form defaults are 20 URLs and 10 sitemap files. If a raw call reaches the runtime without those values, its fallback limits are 1,000 URLs and 100 sitemap files. Supply explicit limits for predictable batches. startUrls is required, and its form prefill is not an API default.

Filter a site section

Use a complete input with the desired patterns. Patterns are case-insensitive regular expressions. Invalid patterns fail the run.

{
  "startUrls": [{"url": "https://apify.com/sitemap.xml"}],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": ["/blog/", "/docs/"],
  "excludePatterns": ["/tag/", "/author/"]
}

For product inventories, use the relevant public sitemap URL and an include pattern such as /products/. The Actor does not inspect pages to classify their content.

Output data

Each dataset row is a URL listed in a fetched sitemap. statusCode and contentType describe the source sitemap response, not the extracted page.

JSON key	Label	Type	Description
url	URL	string	Normalized extracted URL.
loc	Sitemap loc	string	URL from the sitemap
sourceSitemapUrl	Source sitemap URL	string	Sitemap file where the URL was found.
sitemapDepth	Sitemap depth	integer	Recursive sitemap depth from the original input.
lastmod	Last modified	string / null	Optional sitemap last modified value.
changefreq	Change frequency	string / null	Optional sitemap change frequency value.
priority	Priority	number / null	Optional sitemap priority value.
contentType	Content type	string / null	HTTP content type of the source sitemap.
statusCode	HTTP status	integer / null	HTTP status code of the source sitemap.
discoveredAt	Discovered at	string	ISO timestamp when the URL was saved.

Illustrative shape only, not a measured result:

{
  "url": "https://example.com/article",
  "loc": "https://example.com/article",
  "sourceSitemapUrl": "https://example.com/sitemap.xml",
  "sitemapDepth": 0,
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "contentType": "application/xml",
  "statusCode": 200,
  "discoveredAt": "2026-10-25T00:00:00.000Z"
}

loc is the same normalized, absolute URL as url. Optional sitemap metadata is null when absent. The key-value store's RUN_SUMMARY records saved and processed counts, remaining work and whether the time limit was reached.

Download your results

Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See Apify's download and export guide for the available formats and field options.

Pricing

This Actor uses pay-per-event pricing. The start event is charged once when extraction starts.

The result event charges for each extracted URL in an output batch. Sitemap files without matching URLs do not produce result charges. The Actor does not save paid status or error rows.

Use maxUrls, maxSitemaps and the time limit to bound a run. See the live Pricing tab for current rates and discounts.

Tips for best results

• Start small. Review a short export before increasing the limits.
• Use the exact sitemap URL. Root inputs only try common locations.
• Filter useful sections. Include blog, docs or product paths and exclude unwanted archives.
• Keep hostname filtering on. Disable it only when you expect cross-host sitemap entries.
• Keep deduplication on. Turn it off only to inspect repeated URL coverage.

Limits and caveats

• Only public sitemap entries are extracted. The Actor does not fetch the listed pages, generate sitemaps or guarantee indexation.
• Child sitemaps and output URLs are restricted to the original hostname by default. Subdomains count as different hostnames.
• Missing optional metadata stays null. An empty or filtered sitemap can produce no rows. A missing common sitemap path does not prove a website has no sitemap.
• Request failures can leave partial results. If every attempted request fails before any response arrives, the run fails. HTTP error responses are skipped.
• URL, sitemap and time limits can stop a large inventory early. The Actor stops accepting new files before its time limit to allow results to be saved.
• Matching unfinished checkpoints can resume only when the same key-value storage is reused. Saving a task alone does not share unfinished storage between runs.

Automation and integrations

Save your settings for next time

Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.

Run on a schedule

Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

Send results to your other tools

Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.

For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.

For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.

cURL

curl -X POST "https://api.apify.com/v2/acts/fetch_cat~sitemap-url-extractor/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com/sitemap.xml"}],"maxUrls":20,"maxSitemaps":10,"includePatterns":[],"excludePatterns":[]}'

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": []
};

const run = await client.actor('fetch_cat/sitemap-url-extractor').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient
import os

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("fetch_cat/sitemap-url-extractor").call(run_input={
  "startUrls": [
    {
      "url": "https://apify.com/sitemap.xml"
    }
  ],
  "maxUrls": 20,
  "maxSitemaps": 10,
  "includePatterns": [],
  "excludePatterns": []
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

MCP and AI agents

You can ask your AI assistant to collect data with Sitemap URL Extractor and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.

Easier setup with an Apify plugin

If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

Use Apify AI in Console

Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's Sitemap URL Extractor. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

Choose your assistant

• Claude and Claude Code (Anthropic): Apify setup guide.
• ChatGPT and Codex (OpenAI): Apify setup guides.
• Grok Bot (xAI): Apify plugin guide.
• Gemini CLI (Google): MCP server setup guide. Use the server URL below with the HTTP configuration supported by Gemini CLI.

Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/sitemap-url-extractor"
    }
  }
}

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

Use Apify's general server

Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use Sitemap URL Extractor by Fetch Cat, or give it the identifier fetch_cat/sitemap-url-extractor.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

Example requests

• “Get up to 20 URLs from Apify's public sitemap and show their source sitemap and last-modified values.”
• “Extract blog URLs from this public sitemap and organize the returned URLs in a table.”
• “Collect URLs from these two sitemap files, then compare their source groups using the returned records.”

Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.

Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.

Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

Other ways for agents to pay

Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.

FAQ

Can I enter a website URL instead of a sitemap URL?

Yes. A website root tries common sitemap paths. If those return nothing, supply the exact sitemap URL from the site's documentation or robots.txt.

Can I extract only one section of a website?

Yes. Use includePatterns and excludePatterns to keep or skip matching URL paths. They do not classify page content.

Why did I get fewer URLs than expected?

Check the URL, sitemap and time limits, filters and hostname setting. The starting file might not link every sitemap. Some targets may be inaccessible or contain no matching entries.

Why are metadata fields empty?

lastmod, changefreq and priority are optional. The Actor does not infer missing values from the listed pages.

Can I use this as a sitemap API?

Yes. The API and MCP examples above return a run whose dataset contains the extracted entries. Download JSON, CSV or Excel, or retrieve the dataset in your application.

Related Actors

• Website Content Crawler Lite — Collect page content after building a URL inventory.
• Bulk URL Status Checker — Check whether listed pages respond.
• Website Contact Finder — Find public website contact details.
• Website Social Profile Finder — Find social profiles linked from a website.
• Bing Rank Tracker & SERP Scraper — Collect another search-based URL inventory.

Support and feedback

For custom additions or integrations, email hello@fetchcat.dev.

Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. Leave a review to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥