# Sitemap URL Extractor – Get All URLs of a Website (XML Sitemap) (`friedl/sitemap-url-extractor`) Actor Get all page URLs of a website from its XML sitemaps: finds sitemaps via robots.txt, follows sitemap indexes and .gz files, returns lastmod, changefreq and priority. Delta mode reports only new pages, e.g. new competitor pages or blog posts. Fast, cheap, no browser. - **URL**: https://apify.com/friedl/sitemap-url-extractor.md - **Developed by:** [Friedrich K](https://apify.com/friedl) (community) - **Categories:** SEO tools, Developer tools - **Security:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap URL Extractor – All Page URLs from Any Website Get the complete list of page URLs of any website from its XML sitemaps, in seconds and without a browser. - **Automatic discovery:** finds sitemaps through `robots.txt`, then tries `/sitemap.xml` and `/sitemap_index.xml`. You can also enter a sitemap URL directly. - **Complete:** follows sitemap indexes recursively and reads gzip-compressed (`.xml.gz`) and plain-text sitemaps. - **Metadata:** returns `lastmod`, `changefreq` and `priority` for every URL. - **Filters:** keep only URLs that contain a pattern (for example `/blog/`) or that were modified after a given date. - **Delta mode:** schedule it and receive only **new or changed pages**. This is ideal for monitoring competitors' new products, articles or landing pages. - **Polite and compliant:** at most 1 request per second per website. It respects `robots.txt` and identifies itself with a clear user agent. ### Use cases - **SEO audits:** compare the URLs in the sitemap with the indexed pages, and find stale `lastmod` dates. - **Competitor monitoring:** get a daily list of new pages on competitor websites (delta mode). - **Crawl seeding:** feed the URL list into other actors, such as content extractors or link checkers. - **Migrations:** export every URL before a relaunch to build redirect maps. ### Input examples All URLs of two websites: ```json { "startUrls": ["example.com", "https://www.another-site.com"] } ``` Only new blog posts since the last run (schedule daily): ```json { "startUrls": ["competitor.com"], "includePatterns": ["/blog/"], "mode": "delta" } ``` A direct sitemap with a date filter: ```json { "startUrls": ["https://www.gov.uk/sitemap.xml"], "modifiedSince": "2026-10-01" } ``` ### Output (one item) ```json { "id": "https://apify.com/about", "url": "https://apify.com/about", "lastmod": "2026-10-01T08:00:00Z", "changefreq": "weekly", "priority": 0.8, "sitemapUrl": "https://apify.com/sitemap/pages.xml", "website": "https://apify.com", "source": { "name": "XML sitemaps of the websites you provide" }, "fetchedAt": "2026-10-08T14:00:00.000Z" } ``` `changeType` (`new` or `changed`) is present only in delta mode. The run log reports how many sitemaps and URLs were found per website, and warns when a site has no sitemap. ### Pricing Pay per event: **USD 1.00 per 1,000 URLs** (event `result`), with lower prices on higher Apify subscription plans. There is no start fee, and websites without a sitemap cost nothing. Users on the Apify Free plan are limited to 50 results per run. Speed: about 50,000 URLs per sitemap file. A site with 200,000 URLs typically finishes in a few minutes. ### Limits - The actor reads only sitemaps. It does not crawl pages, so pages that are missing from the sitemap are not found. - Sitemaps that `robots.txt` disallows for all user agents are skipped and reported in the log. - Websites that block automated requests (for example with a bot challenge) cannot be read. The actor does not try to bypass such protection. - Each sitemap file can be up to 60 MB, and by default the actor reads up to 1,000 sitemap files per website (`maxSitemaps`). ### Responsibility The actor only reads files that website operators publish for automated reading (`robots.txt` and sitemaps), and it returns the URLs listed in them. You are responsible for using the results in line with the terms of the websites you choose. This actor is not affiliated with any of these websites. ### Impressum / Legal notice Anbieter dieses Actors (Angaben gemäß § 5 DDG) / Provider of this actor: Friedrich Köhler\ Spatzenweg 7\ 04720 Döbeln\ Deutschland\ E-Mail: actorcreater_010@web.de # Actor input Schema ## `startUrls` (type: `array`): Domains (e.g. "example.com"), homepages or direct sitemap URLs (e.g. "https://example.com/sitemap_index.xml"). For domains, sitemaps are discovered via robots.txt, then /sitemap.xml and /sitemap_index.xml. ## `includePatterns` (type: `array`): Keep only URLs that contain at least one of these strings, e.g. "/blog/" or "/products/". ## `modifiedSince` (type: `string`): Keep only URLs whose is on or after this date (YYYY-MM-DD). URLs without lastmod are excluded when set. ## `maxSitemaps` (type: `integer`): Upper limit of sitemap files (incl. index files) read per website. ## `mode` (type: `string`): 'all' returns every URL. 'delta' returns only URLs that are new or whose lastmod/changefreq/priority changed since the last run with the same input – ideal for monitoring new pages of competitors. ## `maxResults` (type: `integer`): Maximum number of URLs (= charged events) per run. Users on the Apify Free plan are limited to 50 results per run. ## `deltaStateName` (type: `string`): Name for the delta state, e.g. "competitor-pages". Leave empty to derive it from the input. Use different names to run independent monitors. ## Actor input object example ```json { "startUrls": [ "https://apify.com" ], "maxSitemaps": 1000, "mode": "all", "maxResults": 10000 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://apify.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("friedl/sitemap-url-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://apify.com"] } # Run the Actor and wait for it to finish run = client.actor("friedl/sitemap-url-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://apify.com" ] }' | apify call friedl/sitemap-url-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,friedl/sitemap-url-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/fWZSbqza7eIIgs4CA/builds/ApnSZZ1bUKet7kphM/openapi.json