# Sitemap URL Extractor (`landlordtools/sitemap-extractor`) Actor Lists every URL in a website's XML sitemaps: finds them through robots.txt, follows sitemap indexes, reads gzip files and returns lastmod, changefreq, priority and hreflang alternates. - **URL**: https://apify.com/landlordtools/sitemap-extractor.md - **Developed by:** [LandlordTools](https://apify.com/landlordtools) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.50 / 1,000 urls This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap URL Extractor Give it a website or sitemap URL; get every page URL in its sitemaps, one row each, with `lastmod`, `changefreq`, `priority` and hreflang alternates. It finds sitemaps through robots.txt, follows indexes, reads gzip and streams huge files in flat memory. ### What it does - **Discovery**: a home page URL reads the `Sitemap:` lines in robots.txt [1][2]. If there are none, it tries `/sitemap.xml`. Any other URL is read as a sitemap. - **Formats**: ``, `` and plain-text sitemaps (one URL per line) [1]. Gzip is detected by its bytes, whatever the file is called. - **Indexes**: breadth-first, each file once (loop guard), with `maxDepth` and `maxSitemaps`. - **Fields**: - `lastmod` exactly as written, plus `lastmodUtc`: W3C Datetime [3] converted to UTC, or null if invalid; - `changefreq` and `priority`, kept only when valid per sitemaps.org; - `xhtml:link` alternates [4]. - **Faults**: malformed XML, HTML served as XML, feeds, damaged gzip and files over 50 MB each give an error row naming the sitemap. URLs read before the fault are kept. - **Politeness**: - at most 2 requests per site at once (up to 4 if you choose); - a User-Agent naming the Actor (`SitemapExtractor/1.0`); - robots.txt `Disallow` rules obeyed for every sitemap fetch, per RFC 9309 [2]. A robots.txt that returns 5xx or can't be reached means nothing is fetched. - **Safety**: only http and https. Private, loopback, link-local and cloud metadata addresses are refused [5][6], including after DNS and on every redirect. ### Quick start The prefilled input lists the first 10 URLs on The National Archives' website: ```json { "startUrls": ["https://www.nationalarchives.gov.uk/"], "maxUrls": 10 } ``` First row, from the run on 4 October 2026: ```json { "url": "https://www.nationalarchives.gov.uk/", "lastmod": "2026-10-01", "lastmodUtc": "2026-10-01", "changefreq": null, "priority": null, "alternates": [], "sitemap": "https://www.nationalarchives.gov.uk/sitemaps/sitemap_1.xml", "sitemapDepth": 1, "site": "https://www.nationalarchives.gov.uk", "error": null } ``` It read robots.txt, two indexes and one sitemap, then stopped at 10 URLs (`capped: true`). ### Input | Field | Default | Meaning | |---|---|---| | `startUrls` | (required) | Websites or sitemaps, up to 100 | | `maxUrls` | 1,000 | Stop after this many URLs (up to 200,000) | | `maxSitemaps` | 200 | Most sitemap files to read | | `maxDepth` | 5 | Deepest index nesting; 0 reads only the start files | | `includeAlternates` | true | Return hreflang alternates | | `removeDuplicates` | true | Return each URL once | | `maxConcurrencyPerHost` | 2 | Requests to one site at a time (1–4) | | `timeoutSecs` | 30 | Per file | ### Output | Field | Meaning | |---|---| | `url` | Page URL (null on error rows) | | `lastmod`, `lastmodUtc` | As written, and normalised | | `changefreq`, `priority` | Valid values, else null | | `alternates` | `[{hreflang, href}]` | | `sitemap`, `sitemapDepth`, `site` | Where the URL was listed | | `error` | Why a sitemap couldn't be read | `OUTPUT` has totals and every sitemap read or skipped. ### Use from code ```bash curl -X POST "https://api.apify.com/v2/acts/landlordtools~sitemap-extractor/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"startUrls": ["https://www.gov.uk/"], "maxUrls": 500}' ``` ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_API_TOKEN' }); const run = await client.actor('landlordtools/sitemap-extractor').call({ startUrls: ['https://www.gov.uk/'], maxUrls: 500 }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items.length, items[0].url); ``` ### Use with AI agents Agents can call it through Apify's MCP server (https://mcp.apify.com) to see a site's pages before reading them. ### Limits - RSS and Atom feeds aren't read; image, video and news sitemap extensions are ignored. - DTDs are never read, so a sitemap that relies on DTD entities is reported as malformed. - The maxUrls cap counts URLs after duplicates are removed. - Benchmarked against Python's standard-library parser on 101,759 live GOV.UK, National Archives, Met Office and legislation.gov.uk URLs: identical (`tests/benchmark/`). ### Integrations Power Platform, OpenAPI, Excel and Google Sheets files are in [`integrations/`](integrations/README.md). Apify's Zapier, Make and n8n apps work too. ### Pricing Pay per event: $0.50 per 1,000 results ($0.0005 each), plus Apify's minimal start fee. A result is one URL returned. Rows for sitemaps that can't be read are free, and the run stops at your spending limit. ### Related Actors - [Link and Citation Checker](https://apify.com/landlordtools/link-citation-checker): check that the URLs work. - [GOV.UK Guidance Fetcher](https://apify.com/landlordtools/govuk-guidance-fetcher): read GOV.UK pages as clean text. ### Sources 1. [sitemaps.org: Sitemaps XML format](https://www.sitemaps.org/protocol.html). Accessed 4 October 2026. 2. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309). Accessed 4 October 2026. 3. [W3C Date and Time Formats](https://www.w3.org/TR/NOTE-datetime). Accessed 4 October 2026. 4. [Google Search Central: localized versions in sitemaps](https://developers.google.com/search/docs/specialty/international/localized-versions#sitemap). Accessed 4 October 2026. 5. [IANA IPv4 Special-Purpose Address Registry](https://www.iana.org/assignments/iana-ipv4-special-registry/). Accessed 4 October 2026. 6. [IANA IPv6 Special-Purpose Address Registry](https://www.iana.org/assignments/iana-ipv6-special-registry/). Accessed 4 October 2026. [1]: https://www.sitemaps.org/protocol.html [2]: https://www.rfc-editor.org/rfc/rfc9309 [3]: https://www.w3.org/TR/NOTE-datetime [4]: https://developers.google.com/search/docs/specialty/international/localized-versions#sitemap [5]: https://www.iana.org/assignments/iana-ipv4-special-registry/ [6]: https://www.iana.org/assignments/iana-ipv6-special-registry/ # Actor input Schema ## `startUrls` (type: `array`): A home page such as https://www.example.org/ finds the sitemaps listed in robots.txt (or /sitemap.xml). Any other URL is read as a sitemap or sitemap index. Up to 100. ## `maxUrls` (type: `integer`): The run stops once this many URLs have been found. Default 1,000; up to 200,000. ## `maxSitemaps` (type: `integer`): Default 200. ## `maxDepth` (type: `integer`): 0 reads only the start sitemaps. Default 5. ## `includeAlternates` (type: `boolean`): Language versions listed with xhtml:link. Default true. ## `removeDuplicates` (type: `boolean`): A URL listed in more than one sitemap is returned once. Default true. ## `maxConcurrencyPerHost` (type: `integer`): Default 2, at most 4. ## `timeoutSecs` (type: `integer`): Default 30. ## Actor input object example ```json { "startUrls": [ "https://www.nationalarchives.gov.uk/" ], "maxUrls": 10 } ``` # Actor output Schema ## `result` (type: `string`): No description ## `output` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://www.nationalarchives.gov.uk/" ], "maxUrls": 10 }; // Run the Actor and wait for it to finish const run = await client.actor("landlordtools/sitemap-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://www.nationalarchives.gov.uk/"], "maxUrls": 10, } # Run the Actor and wait for it to finish run = client.actor("landlordtools/sitemap-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://www.nationalarchives.gov.uk/" ], "maxUrls": 10 }' | apify call landlordtools/sitemap-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,landlordtools/sitemap-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Qu59lPcapGIIgAWaJ/builds/TJtSwbPPcESf47PiJ/openapi.json