# Sitemap URL Extractor – All URLs from XML Sitemaps (`innovative_cookie/sitemap-extractor`) Actor Extract every URL from any website’s XML sitemaps: robots.txt discovery, nested indexes, .gz, lastmod filters. Fast, no browser, CSV/JSON export. - **URL**: https://apify.com/innovative_cookie/sitemap-extractor.md - **Developed by:** [Innovative Cookie](https://apify.com/innovative_cookie) (community) - **Categories:** SEO tools, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Sitemap URL Extractor do? **Sitemap URL Extractor** pulls **every URL out of a website's XML sitemaps** and gives them back as a clean list with `lastmod`, `changefreq` and `priority`. Type a domain such as `example.com` and the Actor finds the sitemaps for you. It reads [robots.txt](https://www.sitemaps.org/protocol.html#submit_robots), tries the standard locations, follows **nested sitemap indexes**, unpacks **.xml.gz** files and also reads **plain-text, RSS and Atom sitemaps**. It is a light **sitemap parser API**. It uses plain HTTP requests, with no browser and no proxy, so it is fast and cheap. You can also check the **HTTP status of each URL** in the same run. ### Why use Sitemap URL Extractor? - πŸ”Ž **SEO audits and site migrations.** Get the full list of indexable URLs, compare the old and new sitemaps, and find stale `lastmod` dates. - πŸ€– **RAG, LLM and crawling pipelines.** Build a clean, deduplicated URL list before you run a heavier crawler such as [Website Content Crawler](https://apify.com/apify/website-content-crawler). You only pay for the pages you need. - πŸ“° **Content monitoring.** Schedule a daily run with **"Modified after: 1 day"** to get only new or updated pages, including Google News titles from news sitemaps. - 🧭 **Competitor research.** See how a site is organised, how many products or articles it has, and how often it publishes. Apify platform features work out of the box: **scheduling**, **REST API**, **webhooks**, and integrations with Make, Zapier, n8n, Google Sheets, Slack and more. You can export results to JSON, CSV, Excel or XML. #### What it handles that other sitemap tools miss | Problem | What this Actor does | |---|---| | Sitemap is not at `/sitemap.xml` | Reads every `Sitemap:` line in robots.txt, then tries `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap/sitemap.xml`, `/sitemaps.xml` and `/sitemap.txt` | | Sitemap index nested 2 or more levels deep | Follows indexes recursively, up to a cap you choose | | `.gz` file that lies about itself | Detects gzip from the file's magic bytes, so it works whether or not the server sends `Content-Encoding: gzip`, and also when the file is not compressed at all | | Index lists `http://` children that fail | Retries automatically over `https://` (for example, gnu.org returns HTTP 500 on http) | | URL returns 200 with an HTML page (soft 404, SPA fallback) | Reported as a failure with a clear reason, not as a silent empty success | | Text, RSS 2.0 and Atom sitemaps | All parsed, as allowed by the sitemaps.org protocol | | Broken XML (stray `&`) | The parser recovers and keeps the valid entries | | One bad site in a batch | The run continues. Every site gets a status and a reason in the `SUMMARY` record | | Run "succeeds" with 0 results | The run is marked **FAILED** with the reason, so your schedules and alerts notice it (you can turn this off) | | Huge sites | Per-site and total URL caps, a lastmod date filter that **skips old child sitemaps before downloading them**, include and exclude regexes, and deduplication | ### What data can Sitemap URL Extractor extract? | Field | Description | |---|---| | `url` | Page URL listed in the sitemap | | `lastmod` | Last modification date (normalised to ISO 8601) | | `changefreq` | `daily`, `weekly`… if present | | `priority` | 0.0 to 1.0 if present | | `sitemapUrl`, `sitemapDepth`, `sitemapFormat` | Where the URL was found (child sitemap, depth in the index tree, and format: urlset, text, rss or atom) | | `imageCount`, `images` | Image sitemap entries | | `newsTitle`, `newsPublicationDate` | Google News sitemap data | | `alternates` | hreflang alternates (multilingual sites) | | `statusCode`, `finalUrl` | HTTP status and redirect target (optional status check) | ### How to extract all URLs from a sitemap 1. Click **Try for free**. 2. In **Websites, domains or sitemap URLs**, enter one or more domains (`example.com`), homepages or direct sitemap URLs (`https://example.com/sitemap_index.xml`). 3. Optionally set **Max URLs per site**, a **Modified after** date or **include/exclude** patterns. 4. Click **Start** and wait a few seconds. 5. Download the URL list as **CSV, Excel or JSON**, or get it via the API. ### How much does it cost to extract sitemap URLs? The Actor itself is **free to use**: there is no Actor fee for now. You only pay the Apify platform usage of your runs (compute units and data transfer), which stays small because the Actor uses plain HTTP requests, without a browser or proxy. The Apify Free plan includes $5 of monthly credit, which covers a lot of test runs. To keep runs short, set **Max URLs per site** or **Max URLs in total**. ### Input See the **Input** tab for every option. Example: ```json { "urls": ["apify.com", "https://www.lemonde.fr", "https://www.gnu.org/sitemap.xml"], "maxUrlsPerSite": 5000, "lastmodFrom": "7 days", "excludePatterns": ["/tag/", "\\.pdf$"], "includeAlternates": false, "checkStatus": false } ``` ### Output You can download the dataset in various formats such as **JSON, HTML, CSV or Excel**. Example items: ```json [ { "url": "https://www.lemonde.fr/international/article/2026/10/28/example_6801641_3210.html", "lastmod": "2026-10-28T20:01:34.000Z", "changefreq": null, "priority": null, "site": "https://www.lemonde.fr", "sitemapUrl": "https://www.lemonde.fr/sitemap_news.xml", "sitemapDepth": 0, "sitemapFormat": "urlset", "imageCount": 1, "newsTitle": "Example headline", "newsPublicationDate": "2026-10-28T20:01:34.000Z" }, { "url": "https://wordpress.org/themes/example-theme/", "lastmod": "2026-10-20T09:12:00.000Z", "changefreq": "weekly", "priority": 0.6, "site": "https://wordpress.org", "sitemapUrl": "https://wordpress.org/themes/sitemap-1.xml", "sitemapDepth": 2, "sitemapFormat": "urlset", "imageCount": 0, "newsTitle": null, "newsPublicationDate": null } ] ``` The **`SUMMARY`** record in the key-value store contains one entry per input site: ```json { "origin": "https://www.gnu.org", "robotsTxtStatus": 200, "sitemapsDeclaredInRobots": 1, "sitemapsFetched": 2, "sitemapsFailed": [], "urlsFound": 5381, "urlsPushed": 5381, "status": "ok" } ``` `status` is one of `ok`, `partial`, `limit-reached`, `no-sitemap`, `blocked` or `error`. `blocked` means the site answered with a captcha or bot challenge, or its robots.txt returned HTTP 401/403/429/5xx: the actor treats that as "no bots allowed" and never tries to get past it. Every failed sitemap is listed with its reason (HTTP 404, HTML page instead of a sitemap, invalid gzip, disallowed by robots.txt…). ### Tips - **Monitoring new pages:** schedule the Actor daily with `lastmodFrom: "1 day"`. Old child sitemaps are skipped before download, so large news sites stay cheap. - **Speed:** requests to the same host are spaced by 250 ms by default, and more if robots.txt sets a `Crawl-delay`. Several sites run in parallel (`maxConcurrency`). - **Status check** sends one HEAD request per URL, so it is slower. Use it on filtered subsets. - **No sitemap found?** Look at `SUMMARY`. If the site really has no sitemap, use a link crawler instead. ### FAQ #### Does it crawl the pages listed in the sitemap? No. It downloads only robots.txt and the sitemap files. With **Check HTTP status** turned on, it also sends a HEAD request to each URL. It never downloads page content. #### Does it respect robots.txt? Yes. It identifies itself with a clear user agent (`SitemapUrlExtractor/1.0`), honours `Crawl-delay` (capped at 10 seconds by default), and never fetches guessed locations or checks URLs that robots.txt disallows. Sitemaps that the site itself declares in robots.txt are always allowed, because they are published for bots. Robots rules are checked again at every redirect. If robots.txt or a sitemap answers with a captcha or bot challenge (or robots.txt returns 401/403/429/5xx), the site is skipped and reported as `blocked`. #### Can it read sitemap.xml.gz files? Yes. It handles gzip whether or not the server sends `Content-Encoding`, double-compressed files, and `.gz` files that are really plain XML. #### What is the maximum sitemap size? The sitemaps.org protocol allows 50,000 URLs and 50 MB per file. The Actor accepts files up to 60 MB after decompression, and up to 500 files per site by default (you can raise this to 50,000). #### Can I use it through the API or from Python/JavaScript? Yes. See the **API** tab for ready-made `curl`, Python and Node.js snippets, or use the Apify MCP server from your AI agent. #### Is it legal to extract sitemaps? Sitemaps are public files that site owners publish so that robots can read them. This Actor collects **no personal data**, only page URLs and their metadata. > Our Actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. #### Something does not work? Open an issue in the **Issues** tab and include the site and the `SUMMARY` record. Issues are usually answered quickly. ### Related actors More Actors by the same author ([innovative_cookie on Apify](https://apify.com/innovative_cookie)): - [PageSpeed Insights Bulk Checker](https://apify.com/innovative_cookie/pagespeed-bulk): Core Web Vitals and Lighthouse scores for many URLs. - [Bulk Domain Availability Checker](https://apify.com/innovative_cookie/domain-availability): check whether domain names are free via RDAP. - [Logo & Favicon Extractor](https://apify.com/innovative_cookie/logo-favicon): get the logo and favicon of any website. - [Bulk Image Converter & Compressor](https://apify.com/innovative_cookie/image-converter): convert and compress images to WebP, AVIF, JPEG or PNG. - [Broken Link Checker](https://apify.com/innovative_cookie/broken-link-checker): find 404s and dead links on any website. - [Website Image Extractor](https://apify.com/innovative_cookie/website-image-extractor): list every image of a page with size, format and alt text. - [Dataset Merge, Dedup & Transform](https://apify.com/innovative_cookie/dataset-merge-dedup): merge datasets, remove duplicates and clean columns. # Actor input Schema ## `urls` (type: `array`): One per line. A domain (example.com) or homepage URL triggers automatic discovery (robots.txt, then standard locations such as /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml). A direct sitemap URL (.xml, .xml.gz, .txt, sitemap index or RSS feed) is read as is. ## `maxUrlsPerSite` (type: `integer`): Stop after this many URLs for each site (after filters). 0 = no limit. ## `maxUrls` (type: `integer`): Stop the whole run after this many URLs. 0 = no limit. Useful to cap cost. ## `maxSitemapsPerSite` (type: `integer`): Safety cap on the number of sitemap files (including nested index children) read per site. ## `includePatterns` (type: `array`): Keep only URLs matching at least one of these regular expressions (case-insensitive). Example: /blog/ ## `excludePatterns` (type: `array`): Drop URLs matching any of these regular expressions. Example: .pdf$ or /tag/ ## `lastmodFrom` (type: `string`): Keep only URLs whose is on or after this date (YYYY-MM-DD) or relative period (e.g. '7 days'). URLs without lastmod are dropped when a date filter is set. ## `lastmodTo` (type: `string`): Keep only URLs whose is on or before this date (YYYY-MM-DD). ## `skipOldSitemaps` (type: `boolean`): When a 'Modified after' date is set, do not download index children whose own is older. Saves time and cost on large sites. ## `dedupe` (type: `boolean`): Output each URL once per site, even if it appears in several sitemaps. ## `includeAlternates` (type: `boolean`): Add the xhtml:link hreflang alternates of each URL (multilingual sites). ## `includeImages` (type: `boolean`): Add the image:loc URLs listed for each page (image sitemaps). The imageCount field is always present. ## `checkStatus` (type: `boolean`): Send a lightweight HEAD request to every extracted URL and add statusCode and finalUrl (redirect target). Slower, billed as an extra event. Respects robots.txt. ## `respectRobotsTxt` (type: `boolean`): Do not fetch guessed sitemap locations or check URLs that robots.txt disallows. Sitemaps declared in robots.txt are always allowed. ## `guessSitemapLocations` (type: `boolean`): If robots.txt declares no working sitemap, try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap/sitemap.xml, /sitemaps.xml and /sitemap.txt. ## `maxConcurrency` (type: `integer`): Number of different sites processed at the same time. Requests to the same host are always spaced out. ## `requestDelayMs` (type: `integer`): Minimum pause between two requests to the same host. A robots.txt Crawl-delay is honoured if larger (capped by the next option). ## `maxCrawlDelaySecs` (type: `integer`): Upper bound for the robots.txt Crawl-delay directive. ## `requestTimeoutSecs` (type: `integer`): Timeout for each HTTP request. Failed requests (network, 429, 5xx) are retried up to 3 times with backoff. ## `failOnEmpty` (type: `boolean`): Mark the run as FAILED (instead of an empty SUCCEEDED run) when no input produced any URL, so schedules and integrations notice the problem. ## Actor input object example ```json { "urls": [ "apify.com", "https://www.gnu.org/sitemap.xml" ], "maxUrlsPerSite": 0, "maxUrls": 0, "maxSitemapsPerSite": 500, "includePatterns": [], "excludePatterns": [], "skipOldSitemaps": true, "dedupe": true, "includeAlternates": false, "includeImages": false, "checkStatus": false, "respectRobotsTxt": true, "guessSitemapLocations": true, "maxConcurrency": 5, "requestDelayMs": 250, "maxCrawlDelaySecs": 10, "requestTimeoutSecs": 30, "failOnEmpty": true } ``` # Actor output Schema ## `dataset` (type: `string`): Dataset with one row per URL ## `summary` (type: `string`): SUMMARY record with sitemaps read, failures and counts # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "apify.com", "https://www.gnu.org/sitemap.xml" ] }; // Run the Actor and wait for it to finish const run = await client.actor("innovative_cookie/sitemap-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "apify.com", "https://www.gnu.org/sitemap.xml", ] } # Run the Actor and wait for it to finish run = client.actor("innovative_cookie/sitemap-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "apify.com", "https://www.gnu.org/sitemap.xml" ] }' | apify call innovative_cookie/sitemap-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,innovative_cookie/sitemap-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/3ksVx5IeJBaGLtfpP/builds/eQqKWsSvvU5bMaJhe/openapi.json