Sitemap XML URL Extractor

Turn public XML sitemaps into a clean, deduplicated URL inventory.

This sitemap URL extractor reads one or more sitemap.xml files, expands nested sitemap indexes, and exports every accepted page URL with its source sitemap, root sitemap, last-modified value, change frequency, priority, depth, and extraction status.

Use the resulting dataset for crawl planning, website migrations, content audits, spreadsheet exports, and downstream data pipelines.

What does Sitemap XML URL Extractor do?

The Actor fetches standards-based public XML sitemap documents over HTTP or HTTPS.

It recognizes both <urlset> documents and <sitemapindex> documents.

For sitemap indexes, it follows child sitemap locations recursively up to your configured depth and document limits.

It normalizes absolute and relative <loc> values, removes URL fragments, applies optional regular-expression filters, and saves each accepted URL once.

It does not crawl the listed pages or claim that they are currently reachable.

The status field means the URL was successfully extracted from valid sitemap XML.

Who is this Actor for?

• SEO specialists building crawl queues or coverage inventories.
• Migration teams comparing the URLs a site publishes before a cutover.
• Content teams reviewing last-modified signals across sections.
• Agencies exporting repeatable sitemap inventories for clients.
• Developers feeding declared URLs into crawlers, indexers, or RAG pipelines.
• Data teams loading URL inventories into Sheets, BigQuery, Snowflake, or a warehouse.

Choose this Actor when the sitemap itself is the source of truth.

Use a separate page-status or SEO-audit Actor when you also need to request every listed page.

Why use this sitemap URL extractor?

Recursive sitemap-index support

A single root sitemap can reference dozens or hundreds of child files.

The Actor expands them with explicit maxDepth and maxSitemaps limits.

Deduplicated inventory

A page can appear in multiple supplied or nested sitemap files.

Only the first normalized occurrence is saved and charged.

Sitemap metadata retained

Output includes <lastmod>, <changefreq>, and <priority> when the publisher supplies them.

Missing optional XML values remain null instead of being invented.

Source provenance

Every row identifies both the direct sourceSitemap and original rootSitemap.

That makes large inventory exports easier to segment and troubleshoot.

HTTP-only and bounded

No browser is launched and no page crawl occurs.

Requests have timeout, retry, redirect, XML-size, depth, and document-count limits.

What data can you extract?

Field	Description
url	Unique page URL declared by the sitemap.
sourceSitemap	Direct XML document containing the URL.
rootSitemap	Original sitemap URL from the input.
depth	Nested index depth where the URL was found.
lastModified	<lastmod>, normalized to ISO 8601 when parseable.
changeFrequency	Optional <changefreq> value.
priority	Optional numeric <priority> from 0 to 1.
status	extracted when the URL became an inventory row.
extractedAt	ISO 8601 extraction timestamp.

Sitemap publishers choose whether to provide optional metadata.

A null value is normal and does not mean parsing failed.

How to extract URLs from sitemap XML

1. Open the Actor input in Apify Console.
2. Add one or more exact public sitemap XML URLs.
3. Set a small maxItems for the first run.
4. Keep maxDepth and maxSitemaps at their defaults for normal indexes.
5. Add include or exclude patterns only when you need a section-specific inventory.
6. Start the run.
7. Open the URL inventory dataset.
8. Export JSON, CSV, Excel, XML, RSS, or an HTML table.

The prefilled https://apify.com/sitemap.xml input is a real nested sitemap and produces useful rows.

Input parameters

Sitemap XML URLs

startUrls is required.

Supply exact public HTTP or HTTPS sitemap URLs such as:

{
  "startUrls": [
    { "url": "https://apify.com/sitemap.xml" },
    { "url": "https://www.shopify.com/sitemap.xml" }
  ]
}

Website roots are not guessed or converted to /sitemap.xml.

This keeps the requested source explicit and predictable.

Maximum URLs

maxItems limits unique dataset rows across all sources.

The default is 1,000 and the maximum is 100,000.

The Actor stops extracting after the limit is reached.

Maximum sitemap files

maxSitemaps limits fetched XML documents, including root and nested files.

The default is 100 and the maximum is 5,000.

Maximum index depth

maxDepth controls recursive sitemap-index expansion.

A direct urlset has depth 0.

URLs in a child file referenced by the root index have depth 1.

Include URL patterns

includeUrlPatterns accepts case-insensitive JavaScript regular expressions.

When supplied, a URL must match at least one include pattern.

Example for Store URLs:

{ "includeUrlPatterns": ["/store/"] }

Exclude URL patterns

excludeUrlPatterns runs after include matching.

Example for tag, author, and image URLs:

{
  "excludeUrlPatterns": [
    "/tags?/",
    "/authors?/",
    "\\.(?:png|jpg|jpeg|gif|svg|webp)(?:\\?|$)"
  ]
}

Invalid regular expressions fail input validation before sitemap fetching.

Request timeout and retries

requestTimeoutSecs applies to each sitemap request.

maxRetries applies only to transient connection errors, timeouts, HTTP 429, and temporary 5xx responses.

Permanent HTTP failures and invalid XML are not retried blindly.

Output example

A current local run against Shopify's public sitemap produced rows in this shape:

{
  "url": "https://www.shopify.com/",
  "sourceSitemap": "https://www.shopify.com/sitemap.xml",
  "rootSitemap": "https://www.shopify.com/sitemap.xml",
  "depth": 0,
  "lastModified": "2026-09-07T00:00:00.000Z",
  "changeFrequency": "daily",
  "priority": 1,
  "status": "extracted",
  "extractedAt": "2026-09-07T07:12:15.671Z"
}

Dates and source content vary between runs.

Deduplication and filter behavior

URLs are resolved relative to the sitemap document that contains them.

Only HTTP and HTTPS URLs are accepted.

URL fragments are removed before deduplication.

The query string remains part of the URL because it can identify distinct published resources.

The first accepted occurrence determines sourceSitemap, rootSitemap, and metadata.

Include and exclude filters use the same normalized URL value.

Excluded and duplicate URLs are neither saved nor charged as items.

Reliability and failure behavior

Each initial URL and redirect target is checked to prevent access to local, private, reserved, and non-routable addresses.

Embedded URL credentials are rejected.

Redirects are limited to five hops.

Each decompressed XML response is limited to 25 MB.

The parser requires a valid sitemap <urlset> or <sitemapindex> shape.

One failed sitemap is logged while other independent sitemap sources can continue.

If every supplied source fails and no URL is extracted, the run fails with a non-zero status.

A valid sitemap with no accepted URLs can finish with an empty dataset, for example when filters exclude every URL.

How much does it cost to extract sitemap URLs?

The Actor uses pay-per-event pricing.

A one-time start event is charged for each run, and an item event is charged for each unique URL saved to the dataset.

The current start price is $0.001 per run.

The current BRONZE item price is $0.00093334 per extracted URL; lower volume and higher subscription tiers use the live tier prices shown by Apify Console.

Approximate BRONZE examples:

Unique URLs	Estimated total
100	$0.094334
1,000	$0.934340
10,000	$9.334400

These examples combine the start event with the BRONZE per-item price.

Failed, duplicate, malformed, filtered-out, and unsaved URL entries do not receive an item charge.

Crawl planning workflow

Run the Actor with your canonical root sitemap and a generous document limit.

Export url, sourceSitemap, and lastModified to your crawler queue.

Use sourceSitemap to assign crawl priorities by content section.

Use lastModified as a planning signal, not proof that page content actually changed.

Website migration workflow

Create an inventory before the migration and retain the exported dataset.

Run the same task against the post-migration sitemap.

Compare URLs in your data tool to find additions, removals, and changed paths.

This Actor does not persist historical diffs itself; Apify schedules and datasets provide the snapshots.

For live redirect and status verification, feed the exported URLs into a purpose-built URL status checker.

Content audit workflow

Use include patterns to target a site section such as /blog/, /docs/, or /products/.

Export last-modified, frequency, and priority fields.

Join the inventory with analytics, CMS, or crawl data in your own pipeline.

Treat sitemap metadata as publisher-provided hints rather than independently verified facts.

Integrations and exports

Apify datasets can be downloaded as JSON, CSV, Excel, XML, RSS, and HTML.

Connect completed runs to Google Sheets, Slack, webhooks, Make, Zapier, or cloud storage.

Use schedules to refresh a stable sitemap inventory daily, weekly, or before a release.

Use the API dataset URL to load rows into a database or data warehouse.

JavaScript API example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/sitemap-xml-url-extractor').call({
  startUrls: [{ url: 'https://apify.com/sitemap.xml' }],
  maxItems: 100,
  maxDepth: 5,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python API example

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/sitemap-xml-url-extractor').call(run_input={
    'startUrls': [{'url': 'https://apify.com/sitemap.xml'}],
    'maxItems': 100,
    'maxDepth': 5,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

cURL API example

curl -X POST \
  'https://api.apify.com/v2/acts/automation-lab~sitemap-xml-url-extractor/runs?token=YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"startUrls":[{"url":"https://apify.com/sitemap.xml"}],"maxItems":100}'

Store the token in a secret or environment variable instead of committing it.

MCP setup

Add the Actor to Claude Code through Apify MCP Server:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/sitemap-xml-url-extractor"

Claude Desktop, Cursor, and VS Code can use this JSON MCP setup:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/sitemap-xml-url-extractor"
    }
  }
}

Example prompts:

• "Extract the first 500 URLs from this sitemap index."
• "Build a WordPress sitemap URL inventory with last-modified dates."
• "Exclude tag and author pages from this sitemap migration inventory."

Tips for large sitemap indexes

Start with maxItems between 20 and 100 to confirm the source and filters.

Increase maxSitemaps when the root index references many child files.

Increase maxDepth only when logs show a deeper valid index structure.

Keep regexes specific and test them on a small run first.

Split unrelated domains into separate tasks when you want independent failure and scheduling behavior.

Limitations

Only anonymous public HTTP(S) XML sitemaps are supported.

Authenticated, local-network, private-IP, FTP, filesystem, and data URLs are rejected.

Robots.txt sitemap discovery is not part of this Actor; provide exact XML URLs.

The Actor does not execute JavaScript or solve anti-bot challenges.

The Actor does not request listed pages, check HTTP page status, validate canonicals, or detect redirects.

The Actor does not generate sitemaps from a website crawl.

A single XML document larger than 25 MB after decompression is rejected.

Malformed XML and nonstandard documents may require the publisher to correct the source.

Legality and responsible use

Process only public sitemap data that you are authorized to use.

Respect website terms, applicable laws, and reasonable request rates.

Do not use the Actor to probe private infrastructure.

Sitemap files can contain URLs that publishers did not intend as personal data exports; review downstream handling accordingly.

Troubleshooting

No URLs were extracted

Confirm the input points directly to XML containing <urlset> or <sitemapindex>.

Check whether include patterns excluded every URL.

Inspect the run log for HTTP, DNS, XML-shape, depth, and document-limit messages.

A nested sitemap was skipped

Increase maxDepth if the log reports the depth limit.

Increase maxSitemaps if the index contains more child files than the current cap.

Last-modified fields are null

The source sitemap did not supply <lastmod> for those URLs.

The Actor does not infer or fetch page modification dates.

A regular expression is rejected

Use JavaScript regular-expression syntax without surrounding /.../ delimiters.

Escape backslashes correctly inside JSON strings.

The run fails on a public-looking URL

The hostname may resolve to a private or reserved network address, or redirect there.

Only publicly routable sitemap targets are supported.

Related Automation Lab Actors

• Bulk URL Status Checker checks exported page URLs for HTTP status.
• Multi-page On-page SEO Audit Crawler extracts page-level SEO signals.
• Robots.txt Validator validates robots directives separately.

FAQ

Does it support nested sitemap indexes?

Yes. It expands nested <sitemapindex> documents up to maxDepth and maxSitemaps.

Does it preserve sitemap last-modified dates?

Yes. <lastmod> is normalized to ISO 8601 when it is a parseable date; otherwise the non-empty source value is retained.

Does it check whether every page returns HTTP 200?

No. status: extracted describes extraction from XML, not page availability.

Use the related Bulk URL Status Checker for live page responses.

Can it read WordPress and Shopify sitemaps?

Yes, when you supply their exact public XML sitemap URLs.

The included task examples use current public WordPress.com and Shopify sitemap sources.

Does it support several root sitemaps?

Yes. Add multiple entries to startUrls; deduplication applies across the complete run.

Are duplicate URLs charged twice?

No. Only unique URL rows successfully saved to the default dataset receive the item event.

Can I schedule recurring inventory exports?

Yes. Save the input as an Apify Task and attach a schedule or webhook.

Does it use a proxy?

No automatic proxy or residential fallback is enabled.

The direct structured XML route keeps runtime and transfer costs predictable.