# Sitemap Extractor: Every URL of Any Website, New Pages (`deriverge/sitemap-extractor`) Actor Extract every URL from any website's sitemaps: found via robots.txt, sitemap indexes, gzip and text sitemaps, with last modified date, priority, images, news and video titles and hreflang alternates. Filters by path and date. New-page alerts between runs. JSON, CSV, API. - **URL**: https://apify.com/deriverge/sitemap-extractor.md - **Developed by:** [deriverge s.r.o.](https://apify.com/deriverge) (community) - **Categories:** SEO tools, Developer tools, Automation - **Security:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.20 / 1,000 urls This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap Extractor Sitemap Extractor lists every URL a website publishes in its XML sitemaps, one row per URL, with the last modified date, change frequency, priority, image URLs, Google News and video titles, and the hreflang language versions when the sitemap has them. Type a domain such as `stripe.com` into **Websites or sitemaps** and click **Start**: sitemaps are found through robots.txt, and indexes and gzip files are opened for you. The actor reads sitemaps only; it does not visit the pages or check whether they still load. It costs $0.40 per 1,000 URLs on the Free plan ($0.20 on Business), and the $5 of monthly credit in Apify's Free plan covers about 12,000 URLs. ### What data does it return? | Field | Description | |---|---| | `url` | Page address exactly as the sitemap lists it in ``. | | `site` | Host name of the website, such as `ahrefs.com`. | | `sitemap` | The sitemap file that lists the URL, after any redirect, such as `https://ahrefs.com/writing-tools/sitemap.xml`. | | `lastModified` | The `` date as an ISO 8601 UTC timestamp, `null` when the site leaves it out. Many large sites do: Stripe and Ahrefs had none in October 2026. | | `changeFrequency`, `priority` | `` such as `daily` or `weekly`, and `` from 0 to 1. Both `null` when not given. | | `images` | Image URLs listed for the page in an image sitemap, up to 20 per page. | | `newsTitle`, `newsPublishedAt` | Article title and publication time from a Google News sitemap. | | `videoTitle` | Video title from a video sitemap. | | `alternates` | Language versions from `xhtml:link rel="alternate"` tags, each with `hreflang` (such as `de` or `en-AT`) and `url`. Stripe lists up to 76 per page. | | `isNew` | With a watch name, `true` when the URL was not in the site's sitemaps in the previous run and `false` when it was. `null` without a watch or in the first run. | | `checkedAt` | When the sitemap was read, as an ISO 8601 UTC timestamp. | ### How to extract URLs from a sitemap 1. Enter websites in **Websites or sitemaps**, one per line. A domain such as `wordpress.org` makes the actor read the `Sitemap:` lines of its robots.txt, or try 8 usual addresses (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.txt` and others) when robots.txt names none. To read one file only, paste its URL, such as `https://wordpress.org/news/sitemap.xml`. 2. Narrow the list with **URL contains (any of)**, such as `/blog/`, and **URL does not contain**, such as `/tag/`. Both ignore case. 3. To get recently changed pages, set **Modified on or after** to a date (`2026-10-01`) or a moving day (`today`, `-7`). URLs without a `` date are then left out. 4. Set **Maximum URLs per website** (100,000 by default) and **Maximum rows** to control the size of the run. 5. Click **Start**. URLs appear in the **Output** tab, ready to download as CSV, Excel or JSON. The `SUMMARY` record in the key-value store gives per website the number of sitemap files read, URLs found, new URLs and any error. For the API, this input returns blog and news pages that changed in the last 30 days on two sites: ```json { "websites": ["www.shopify.com", "wordpress.org"], "urlIncludes": ["/blog/", "/news/"], "modifiedAfter": "-30", "maxUrlsPerSite": 100000 } ``` ### Example output One URL from Ahrefs in a run on 11 October 2026. The Ahrefs sitemap gives no dates, so `lastModified` is `null`, and the page has 12 language versions. ```json { "url": "https://ahrefs.com/fr/writing-tools/img-alt-text-generator", "site": "ahrefs.com", "sitemap": "https://ahrefs.com/writing-tools/sitemap.xml", "lastModified": null, "changeFrequency": null, "priority": null, "images": [], "newsTitle": null, "newsPublishedAt": null, "videoTitle": null, "alternates": [ { "hreflang": "de", "url": "https://ahrefs.com/de/writing-tools/img-alt-text-generator" }, { "hreflang": "en", "url": "https://ahrefs.com/writing-tools/img-alt-text-generator" }, { "hreflang": "es", "url": "https://ahrefs.com/es/writing-tools/img-alt-text-generator" }, { "hreflang": "id", "url": "https://ahrefs.com/id/writing-tools/img-alt-text-generator" }, { "hreflang": "it", "url": "https://ahrefs.com/it/writing-tools/img-alt-text-generator" }, { "hreflang": "ja", "url": "https://ahrefs.com/ja/writing-tools/img-alt-text-generator" }, { "hreflang": "ko", "url": "https://ahrefs.com/ko/writing-tools/img-alt-text-generator" }, { "hreflang": "pl", "url": "https://ahrefs.com/pl/writing-tools/img-alt-text-generator" }, { "hreflang": "pt", "url": "https://ahrefs.com/pt/writing-tools/img-alt-text-generator" }, { "hreflang": "tr", "url": "https://ahrefs.com/tr/writing-tools/img-alt-text-generator" }, { "hreflang": "uk", "url": "https://ahrefs.com/uk/writing-tools/img-alt-text-generator" }, { "hreflang": "zh", "url": "https://ahrefs.com/zh/writing-tools/img-alt-text-generator" } ], "isNew": null, "checkedAt": "2026-10-11T16:01:38.385Z" } ``` ### How much does it cost to extract sitemap URLs? | | Free plan | Starter | Scale | Business | |---|---|---|---|---| | 1,000 URLs | $0.40 | $0.32 | $0.26 | $0.20 | You pay only for the events in the table. There is no start fee, and compute time and proxies are included. A website with 10,000 URLs costs $4.00 on the Free plan and $2.00 on Business; stripe.com listed 7,103 in October 2026. A URL listed in several sitemaps of the same site is returned and charged once, and URLs removed by your filters or skipped as already seen in new-only mode cost nothing. ### Limits - Speed depends on the number of sitemap files, which are read one at a time. In October 2026, stripe.com, ahrefs.com and apify.com gave 59,146 URLs from 21 sitemap files in 76 seconds. Shopify and WordPress.org took 67 seconds for 371 sitemap files. - The cap in **Maximum URLs per website** counts every unique URL read, before the filters. With filters on a large site, raise it: the Shopify and WordPress.org run above stopped at 100,000 URLs on each site and returned the 1,520 blog and news pages modified in the last 30 days among them. - Up to 5,000 sitemap files are opened per website. Each file request waits up to 60 seconds and is tried 3 times on a server error or HTTP 429. - The sitemap protocol allows 50,000 URLs per file, and the actor follows indexes inside indexes. Pages that the site leaves out of its sitemaps are not found, because the actor does not crawl links. - `lastModified`, `changeFrequency` and `priority` are whatever the site writes. Some sites stamp every URL with the date the sitemap was generated. - A website without any sitemap in robots.txt or at the usual addresses is reported in `SUMMARY` with the error `No sitemap found` and costs nothing. When no website could be read, the run fails, unless robots.txt was the reason. ### robots.txt The actor follows robots.txt of each website, using the rules for all user agents. A sitemap file that robots.txt disallows is skipped and counted under `sitemapsSkippedByRobots` in `SUMMARY`, and redirects are checked the same way. A website whose robots.txt does not answer (a server error, HTTP 429 or a timeout) is skipped for that run, while a missing robots.txt (HTTP 404) allows everything. ### Alerts for new pages Give the run a **Watch name**, such as `shop-blog`, or save it as a task, and schedule it daily or weekly. From the second run on, every URL carries `isNew`, and `SUMMARY` gives `newUrls` per website. Turn on **Return only URLs new since the last run** to receive and pay for only the new ones: new products, articles or landing pages of the site. The actor stores a 64-bit fingerprint of each URL, never the page itself. When a run stops early at a limit, its fingerprints are added to the stored ones, so the unread part of the site is not forgotten. New URLs that were read but did not fit under **Maximum rows** or the spending limit are not stored, so they stay new and come in the next run: three runs with **Maximum rows** at 5 on apify.com returned 15 different new URLs. ### FAQ #### Is it legal to extract URLs from a sitemap? Sitemaps are files that websites publish so that search engines and other programs can find their pages, and the actor reads them without logging in. It honors robots.txt and returns URLs and the metadata of the sitemap, no page content and no personal data. You are responsible for what you do with the pages you then visit. #### Does it check the status code of each URL? No. It returns what the sitemap lists. To find URLs that redirect or return 404, pass the output to a URL status checker. #### Can it read a sitemap that is not at the usual address? Yes. Paste the full sitemap URL, including `.xml`, `.xml.gz` or `.txt` files and any URL whose path contains `sitemap`; it is read directly, and an index found there is opened down to its last file. ### Related scrapers - [URL Status Checker](https://apify.com/deriverge/url-status-checker) - [Broken Link Checker](https://apify.com/deriverge/broken-link-checker) - [Tech Stack Detector](https://apify.com/deriverge/tech-stack-detector) - [Article Extractor](https://apify.com/deriverge/article-extractor) - [RSS Feed Scraper](https://apify.com/deriverge/rss-feed-scraper) ### Support This actor is built and maintained by deriverge s.r.o., a software company based in the Czech Republic. If a run fails or a field you need is missing, please open an issue in the **Issues** tab or write to us at info@deriverge.com. We respond in English and Czech. Runs can be scheduled in Apify Console or started from the **API** tab, which has examples for Python, JavaScript and cURL and works with Make, Zapier, n8n and the Apify MCP server. If the actor saves you time, a short review helps other people find it. # Changelog This Actor's version history is a separate document: https://apify.com/deriverge/sitemap-extractor/changelog.md # Actor input Schema ## `websites` (type: `array`): Domains (example.com) or direct sitemap URLs (https://example.com/sitemap.xml), one per line. For a domain, sitemaps are found in robots.txt or at the usual addresses. ## `maxUrlsPerSite` (type: `integer`): Stop reading a website after this many unique URLs. ## `urlIncludes` (type: `array`): Keep only URLs containing one of these, for example /blog/ or /products/. ## `urlExcludes` (type: `array`): Drop URLs containing any of these, for example /tag/ or ?page=. ## `modifiedAfter` (type: `string`): Keep only URLs whose last modified date is on or after this day: YYYY-MM-DD, today, or -7 for the last week. URLs without a date are dropped. ## `newOnly` (type: `boolean`): With a watch name or a saved task, return only URLs that were not in the sitemaps last time: new pages, products, articles. ## `watchName` (type: `string`): Give the run a name such as "competitor-blog" and schedule it. Every later run marks each URL with isNew, and new-only mode returns just the new ones. ## `maxItems` (type: `integer`): Hard cap on returned URLs in the run. ## Actor input object example ```json { "websites": [ "apify.com" ], "maxUrlsPerSite": 100000, "newOnly": false, "maxItems": 100 } ``` # Actor output Schema ## `urls` (type: `string`): One row per URL with last modified date, priority, images, news and language versions. ## `summary` (type: `string`): Sitemaps read, URLs and new URLs per website, and websites without a sitemap. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "websites": [ "apify.com" ], "maxItems": 100 }; // Run the Actor and wait for it to finish const run = await client.actor("deriverge/sitemap-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "websites": ["apify.com"], "maxItems": 100, } # Run the Actor and wait for it to finish run = client.actor("deriverge/sitemap-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "websites": [ "apify.com" ], "maxItems": 100 }' | apify call deriverge/sitemap-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,deriverge/sitemap-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/rOulwZNPkyVZ4nhf6/builds/IGmpiEJZrM16hcG8N/openapi.json