# Sitemap XML URL Extractor (`automation-lab/sitemap-xml-url-extractor`) Actor Extract deduplicated URLs and metadata from public XML sitemaps and nested sitemap indexes for crawl planning, migrations, and content audits. - **URL**: https://apify.com/automation-lab/sitemap-xml-url-extractor.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** SEO tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.56 / 1,000 item extracteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap XML URL Extractor Turn public XML sitemaps into a clean, deduplicated URL inventory. This sitemap URL extractor reads one or more `sitemap.xml` files, expands nested sitemap indexes, and exports every accepted page URL with its source sitemap, root sitemap, last-modified value, change frequency, priority, depth, and extraction status. Use the resulting dataset for crawl planning, website migrations, content audits, spreadsheet exports, and downstream data pipelines. ### What does Sitemap XML URL Extractor do? The Actor fetches standards-based public XML sitemap documents over HTTP or HTTPS. It recognizes both `` documents and `` documents. For sitemap indexes, it follows child sitemap locations recursively up to your configured depth and document limits. It normalizes absolute and relative `` values, removes URL fragments, applies optional regular-expression filters, and saves each accepted URL once. It does **not** crawl the listed pages or claim that they are currently reachable. The `status` field means the URL was successfully extracted from valid sitemap XML. ### Who is this Actor for? - SEO specialists building crawl queues or coverage inventories. - Migration teams comparing the URLs a site publishes before a cutover. - Content teams reviewing last-modified signals across sections. - Agencies exporting repeatable sitemap inventories for clients. - Developers feeding declared URLs into crawlers, indexers, or RAG pipelines. - Data teams loading URL inventories into Sheets, BigQuery, Snowflake, or a warehouse. Choose this Actor when the sitemap itself is the source of truth. Use a separate page-status or SEO-audit Actor when you also need to request every listed page. ### Why use this sitemap URL extractor? #### Recursive sitemap-index support A single root sitemap can reference dozens or hundreds of child files. The Actor expands them with explicit `maxDepth` and `maxSitemaps` limits. #### Deduplicated inventory A page can appear in multiple supplied or nested sitemap files. Only the first normalized occurrence is saved and charged. #### Sitemap metadata retained Output includes ``, ``, and `` when the publisher supplies them. Missing optional XML values remain `null` instead of being invented. #### Source provenance Every row identifies both the direct `sourceSitemap` and original `rootSitemap`. That makes large inventory exports easier to segment and troubleshoot. #### HTTP-only and bounded No browser is launched and no page crawl occurs. Requests have timeout, retry, redirect, XML-size, depth, and document-count limits. ### What data can you extract? | Field | Description | | --- | --- | | `url` | Unique page URL declared by the sitemap. | | `sourceSitemap` | Direct XML document containing the URL. | | `rootSitemap` | Original sitemap URL from the input. | | `depth` | Nested index depth where the URL was found. | | `lastModified` | ``, normalized to ISO 8601 when parseable. | | `changeFrequency` | Optional `` value. | | `priority` | Optional numeric `` from 0 to 1. | | `status` | `extracted` when the URL became an inventory row. | | `extractedAt` | ISO 8601 extraction timestamp. | Sitemap publishers choose whether to provide optional metadata. A null value is normal and does not mean parsing failed. ### How to extract URLs from sitemap XML 1. Open the Actor input in Apify Console. 2. Add one or more exact public sitemap XML URLs. 3. Set a small `maxItems` for the first run. 4. Keep `maxDepth` and `maxSitemaps` at their defaults for normal indexes. 5. Add include or exclude patterns only when you need a section-specific inventory. 6. Start the run. 7. Open the URL inventory dataset. 8. Export JSON, CSV, Excel, XML, RSS, or an HTML table. The prefilled `https://apify.com/sitemap.xml` input is a real nested sitemap and produces useful rows. ### Input parameters #### Sitemap XML URLs `startUrls` is required. Supply exact public HTTP or HTTPS sitemap URLs such as: ```json { "startUrls": [ { "url": "https://apify.com/sitemap.xml" }, { "url": "https://www.shopify.com/sitemap.xml" } ] } ``` Website roots are not guessed or converted to `/sitemap.xml`. This keeps the requested source explicit and predictable. #### Maximum URLs `maxItems` limits unique dataset rows across all sources. The default is 1,000 and the maximum is 100,000. The Actor stops extracting after the limit is reached. #### Maximum sitemap files `maxSitemaps` limits fetched XML documents, including root and nested files. The default is 100 and the maximum is 5,000. #### Maximum index depth `maxDepth` controls recursive sitemap-index expansion. A direct urlset has depth 0. URLs in a child file referenced by the root index have depth 1. #### Include URL patterns `includeUrlPatterns` accepts case-insensitive JavaScript regular expressions. When supplied, a URL must match at least one include pattern. Example for Store URLs: ```json { "includeUrlPatterns": ["/store/"] } ``` #### Exclude URL patterns `excludeUrlPatterns` runs after include matching. Example for tag, author, and image URLs: ```json { "excludeUrlPatterns": [ "/tags?/", "/authors?/", "\\.(?:png|jpg|jpeg|gif|svg|webp)(?:\\?|$)" ] } ``` Invalid regular expressions fail input validation before sitemap fetching. #### Request timeout and retries `requestTimeoutSecs` applies to each sitemap request. `maxRetries` applies only to transient connection errors, timeouts, HTTP 429, and temporary 5xx responses. Permanent HTTP failures and invalid XML are not retried blindly. ### Output example A current local run against Shopify's public sitemap produced rows in this shape: ```json { "url": "https://www.shopify.com/", "sourceSitemap": "https://www.shopify.com/sitemap.xml", "rootSitemap": "https://www.shopify.com/sitemap.xml", "depth": 0, "lastModified": "2026-09-07T00:00:00.000Z", "changeFrequency": "daily", "priority": 1, "status": "extracted", "extractedAt": "2026-09-07T07:12:15.671Z" } ``` Dates and source content vary between runs. ### Deduplication and filter behavior URLs are resolved relative to the sitemap document that contains them. Only HTTP and HTTPS URLs are accepted. URL fragments are removed before deduplication. The query string remains part of the URL because it can identify distinct published resources. The first accepted occurrence determines `sourceSitemap`, `rootSitemap`, and metadata. Include and exclude filters use the same normalized URL value. Excluded and duplicate URLs are neither saved nor charged as items. ### Reliability and failure behavior Each initial URL and redirect target is checked to prevent access to local, private, reserved, and non-routable addresses. Embedded URL credentials are rejected. Redirects are limited to five hops. Each decompressed XML response is limited to 25 MB. The parser requires a valid sitemap `` or `` shape. One failed sitemap is logged while other independent sitemap sources can continue. If every supplied source fails and no URL is extracted, the run fails with a non-zero status. A valid sitemap with no accepted URLs can finish with an empty dataset, for example when filters exclude every URL. ### How much does it cost to extract sitemap URLs? The Actor uses pay-per-event pricing. A one-time `start` event is charged for each run, and an `item` event is charged for each unique URL saved to the dataset. The current start price is **$0.001 per run**. The current BRONZE item price is **$0.00093334 per extracted URL**; lower volume and higher subscription tiers use the live tier prices shown by Apify Console. Approximate BRONZE examples: | Unique URLs | Estimated total | | ---: | ---: | | 100 | $0.094334 | | 1,000 | $0.934340 | | 10,000 | $9.334400 | These examples combine the start event with the BRONZE per-item price. Failed, duplicate, malformed, filtered-out, and unsaved URL entries do not receive an item charge. ### Crawl planning workflow Run the Actor with your canonical root sitemap and a generous document limit. Export `url`, `sourceSitemap`, and `lastModified` to your crawler queue. Use `sourceSitemap` to assign crawl priorities by content section. Use `lastModified` as a planning signal, not proof that page content actually changed. ### Website migration workflow Create an inventory before the migration and retain the exported dataset. Run the same task against the post-migration sitemap. Compare URLs in your data tool to find additions, removals, and changed paths. This Actor does not persist historical diffs itself; Apify schedules and datasets provide the snapshots. For live redirect and status verification, feed the exported URLs into a purpose-built URL status checker. ### Content audit workflow Use include patterns to target a site section such as `/blog/`, `/docs/`, or `/products/`. Export last-modified, frequency, and priority fields. Join the inventory with analytics, CMS, or crawl data in your own pipeline. Treat sitemap metadata as publisher-provided hints rather than independently verified facts. ### Integrations and exports Apify datasets can be downloaded as JSON, CSV, Excel, XML, RSS, and HTML. Connect completed runs to Google Sheets, Slack, webhooks, Make, Zapier, or cloud storage. Use schedules to refresh a stable sitemap inventory daily, weekly, or before a release. Use the API dataset URL to load rows into a database or data warehouse. ### JavaScript API example ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/sitemap-xml-url-extractor').call({ startUrls: [{ url: 'https://apify.com/sitemap.xml' }], maxItems: 100, maxDepth: 5, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` ### Python API example ```python import os from apify_client import ApifyClient client = ApifyClient(os.environ['APIFY_TOKEN']) run = client.actor('automation-lab/sitemap-xml-url-extractor').call(run_input={ 'startUrls': [{'url': 'https://apify.com/sitemap.xml'}], 'maxItems': 100, 'maxDepth': 5, }) items = client.dataset(run['defaultDatasetId']).list_items().items print(items) ``` ### cURL API example ```bash curl -X POST \ 'https://api.apify.com/v2/acts/automation-lab~sitemap-xml-url-extractor/runs?token=YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{"startUrls":[{"url":"https://apify.com/sitemap.xml"}],"maxItems":100}' ``` Store the token in a secret or environment variable instead of committing it. ### MCP setup Add the Actor to Claude Code through Apify MCP Server: ```bash claude mcp add --transport http apify \ "https://mcp.apify.com?tools=automation-lab/sitemap-xml-url-extractor" ``` Claude Desktop, Cursor, and VS Code can use this JSON MCP setup: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=automation-lab/sitemap-xml-url-extractor" } } } ``` Example prompts: - "Extract the first 500 URLs from this sitemap index." - "Build a WordPress sitemap URL inventory with last-modified dates." - "Exclude tag and author pages from this sitemap migration inventory." ### Tips for large sitemap indexes Start with `maxItems` between 20 and 100 to confirm the source and filters. Increase `maxSitemaps` when the root index references many child files. Increase `maxDepth` only when logs show a deeper valid index structure. Keep regexes specific and test them on a small run first. Split unrelated domains into separate tasks when you want independent failure and scheduling behavior. ### Limitations Only anonymous public HTTP(S) XML sitemaps are supported. Authenticated, local-network, private-IP, FTP, filesystem, and data URLs are rejected. Robots.txt sitemap discovery is not part of this Actor; provide exact XML URLs. The Actor does not execute JavaScript or solve anti-bot challenges. The Actor does not request listed pages, check HTTP page status, validate canonicals, or detect redirects. The Actor does not generate sitemaps from a website crawl. A single XML document larger than 25 MB after decompression is rejected. Malformed XML and nonstandard documents may require the publisher to correct the source. ### Legality and responsible use Process only public sitemap data that you are authorized to use. Respect website terms, applicable laws, and reasonable request rates. Do not use the Actor to probe private infrastructure. Sitemap files can contain URLs that publishers did not intend as personal data exports; review downstream handling accordingly. ### Troubleshooting #### No URLs were extracted Confirm the input points directly to XML containing `` or ``. Check whether include patterns excluded every URL. Inspect the run log for HTTP, DNS, XML-shape, depth, and document-limit messages. #### A nested sitemap was skipped Increase `maxDepth` if the log reports the depth limit. Increase `maxSitemaps` if the index contains more child files than the current cap. #### Last-modified fields are null The source sitemap did not supply `` for those URLs. The Actor does not infer or fetch page modification dates. #### A regular expression is rejected Use JavaScript regular-expression syntax without surrounding `/.../` delimiters. Escape backslashes correctly inside JSON strings. #### The run fails on a public-looking URL The hostname may resolve to a private or reserved network address, or redirect there. Only publicly routable sitemap targets are supported. ### Related Automation Lab Actors - [Bulk URL Status Checker](https://apify.com/automation-lab/bulk-url-status-checker) checks exported page URLs for HTTP status. - [Multi-page On-page SEO Audit Crawler](https://apify.com/automation-lab/multi-page-on-page-seo-audit) extracts page-level SEO signals. - [Robots.txt Validator](https://apify.com/automation-lab/robots-txt-validator) validates robots directives separately. ### FAQ #### Does it support nested sitemap indexes? Yes. It expands nested `` documents up to `maxDepth` and `maxSitemaps`. #### Does it preserve sitemap last-modified dates? Yes. `` is normalized to ISO 8601 when it is a parseable date; otherwise the non-empty source value is retained. #### Does it check whether every page returns HTTP 200? No. `status: extracted` describes extraction from XML, not page availability. Use the related Bulk URL Status Checker for live page responses. #### Can it read WordPress and Shopify sitemaps? Yes, when you supply their exact public XML sitemap URLs. The included task examples use current public WordPress.com and Shopify sitemap sources. #### Does it support several root sitemaps? Yes. Add multiple entries to `startUrls`; deduplication applies across the complete run. #### Are duplicate URLs charged twice? No. Only unique URL rows successfully saved to the default dataset receive the item event. #### Can I schedule recurring inventory exports? Yes. Save the input as an Apify Task and attach a schedule or webhook. #### Does it use a proxy? No automatic proxy or residential fallback is enabled. The direct structured XML route keeps runtime and transfer costs predictable. # Actor input Schema ## `startUrls` (type: `array`): One or more public HTTP(S) sitemap.xml or sitemap-index URLs. ## `maxItems` (type: `integer`): Maximum number of unique page URLs to save across all supplied sitemaps. ## `maxSitemaps` (type: `integer`): Maximum XML documents to fetch while expanding nested sitemap indexes. ## `maxDepth` (type: `integer`): Maximum number of nested sitemap-index levels to follow. ## `includeUrlPatterns` (type: `array`): Case-insensitive regular expressions. A URL is saved when it matches at least one pattern. Leave empty to include all URLs. ## `excludeUrlPatterns` (type: `array`): Case-insensitive regular expressions. Matching URLs are excluded after include filtering. ## `requestTimeoutSecs` (type: `integer`): Maximum seconds to wait for each sitemap request. ## `maxRetries` (type: `integer`): Retries for timeouts, connection failures, HTTP 429, and temporary server errors. ## Actor input object example ```json { "startUrls": [ { "url": "https://apify.com/sitemap.xml" } ], "maxItems": 20, "maxSitemaps": 20, "maxDepth": 5, "includeUrlPatterns": [], "excludeUrlPatterns": [], "requestTimeoutSecs": 30, "maxRetries": 2 } ``` # Actor output Schema ## `dataset` (type: `string`): Deduplicated URLs extracted from supplied and nested sitemap XML files. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://apify.com/sitemap.xml" } ], "maxItems": 20, "maxSitemaps": 20, "maxDepth": 5 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/sitemap-xml-url-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://apify.com/sitemap.xml" }], "maxItems": 20, "maxSitemaps": 20, "maxDepth": 5, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/sitemap-xml-url-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://apify.com/sitemap.xml" } ], "maxItems": 20, "maxSitemaps": 20, "maxDepth": 5 }' | apify call automation-lab/sitemap-xml-url-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/sitemap-xml-url-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/uXMd6aV9Pds3RGNKf/builds/iY7mTgt5r9GlKmbrc/openapi.json