# Sitemap Extractor Pro - All URLs, Images, News & Hreflang (`mbabbu/sitemap-extractor-pro`) Actor Extract every URL from any website's sitemaps. Auto-discovers sitemaps (robots.txt, WordPress, Shopify, Yoast, gzip, indexes, RSS/Atom, text), outputs lastmod, priority, images, videos, Google News and hreflang, with date/regex filters and optional HTTP status checks. Never fails on a bad site. - **URL**: https://apify.com/mbabbu/sitemap-extractor-pro.md - **Developed by:** [RoseFormulaLab](https://apify.com/mbabbu) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.50 / 1,000 url extracteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap Extractor Pro: Extract All URLs from Any Website's Sitemap **Sitemap Extractor Pro** finds and reads every sitemap of any website and gives you a clean list of all its URLs. Each URL comes with `lastmod`, `changefreq`, `priority`, **image, video and Google News data** and **hreflang alternates**. Give it a domain. It discovers the sitemaps on its own: robots.txt, WordPress, Yoast, Shopify, Wix, Squarespace, gzip files, nested sitemap indexes, RSS/Atom feeds and plain-text sitemaps. You can filter by date or regex, and optionally check the HTTP status of every URL to find broken pages and redirects. It is built to **never fail your run because one website is broken**. A site with no sitemap, broken XML, a timeout or a 403 is reported clearly in the run summary, and every other site in your batch still gets extracted. ### What can it do? - **Automatic sitemap discovery.** It checks: - `robots.txt` `Sitemap:` lines - about 20 common and CMS paths (`/sitemap.xml`, `/sitemap_index.xml` (Yoast, Rank Math, AIOSEO), `/wp-sitemap.xml` (WordPress core), `/sitemap.xml.gz`, `/sitemap.txt`, news sitemaps, Magento/Drupal/PrestaShop paths) - homepage `` tags - `www`/non-www and https/http variants - **Every sitemap format.** XML `urlset`, **sitemap indexes nested to any depth** (loops are detected), **`.xml.gz` gzip** (even when the server double-compresses it or labels it wrongly), RSS 2.0, RSS 1.0/RDF, Atom and plain-text URL lists. - **Tolerant parsing.** Broken XML is common: unescaped `&`, unclosed tags, truncated or corrupt gzip, wrong encodings, an HTML "soft 404" served as `sitemap.xml`. The Actor recovers every URL it can and tells you what was wrong. - **Rich output.** - `url`, `lastmod` (normalised to ISO 8601), `changefreq`, `priority` - which sitemap the URL came from and how that sitemap was found - `image:image` entries, `video:video` entries, `news:news` (publication, language, date, headline) and `xhtml:link` hreflang alternates - **Feed fallback.** If a site has no XML sitemap, URLs are taken from its RSS/Atom feed. - **Filters.** - "Changed since" with an absolute date or a relative period like `7 days`. Child sitemaps that haven't changed are skipped too, which makes it much faster on big sites. - include/exclude regex - same-domain only - de-duplication - max URLs, overall and per site - **Optional HTTP status check.** - HEAD request with an automatic GET fallback, so servers that reject HEAD don't produce false "broken" results - records the first status code, the redirect count, the final URL and the final status - polite concurrency limits per site - **Reliable and polite.** - retries with exponential backoff and `Retry-After` - timeouts for stalled servers - an honest User-Agent - respects robots.txt when fetching sitemaps - per-host concurrency limits - optional proxy ### Who is it for? Use cases - **SEO audits.** List every indexable URL, find sitemap URLs that return 404, 5xx or redirects, and check `lastmod` freshness, hreflang coverage and image sitemaps. - **Content inventory.** Export every page, product, post and category of a site to CSV or Excel in seconds. - **Website migrations.** Snapshot all URLs before a redesign or domain move, then check after launch that every old URL redirects (use the status check). - **Change monitoring.** Schedule the Actor daily with "Changed since: 1 day" to get only new or updated pages, such as competitor product launches, new blog posts or news articles. - **Feeding AI / RAG pipelines.** Get a complete, deduplicated list of a site's URLs, then pass it to a crawler such as [Website Content Crawler](https://apify.com/apify/website-content-crawler) to build knowledge bases or train chatbots. - **Crawler seeding.** Start large crawls with the full URL list instead of following links. - **Competitive research.** See how many products, categories or articles a competitor has and how often they are updated. ### How to use it 1. Enter one or more **websites** (`example.com`) or **sitemap URLs** (`https://example.com/sitemap_index.xml`). You can also upload a file with thousands of domains. 2. Optionally set **filters** ("Changed since", include/exclude regex) and **limits**. 3. Optionally turn on **Check HTTP status of each URL** for an SEO audit. 4. Click **Start**. Download the results as JSON, CSV, Excel or HTML, or get them through the Apify API or integrations (Google Sheets, Zapier, Make, n8n, Slack, webhooks). #### Input example ```json { "startUrls": [ { "url": "https://www.allbirds.com" }, { "url": "wordpress.org/news" }, { "url": "https://www.nytimes.com/sitemaps/new/news.xml.gz" } ], "modifiedSince": "7 days", "excludePatterns": ["/tag/", "/author/"], "maxUrlsPerSite": 10000, "checkStatus": false } ``` ### Output Each URL is one row in the dataset. Here is a real row from a Shopify store: ```json { "url": "https://www.allbirds.com/products/mens-wool-runners", "lastmod": "2026-10-04T09:09:12.000Z", "changefreq": "daily", "priority": null, "title": null, "sitemapUrl": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808", "sitemapType": "urlset", "discoveredVia": "sitemap-index", "site": "https://www.allbirds.com", "imageCount": 1, "videoCount": 0, "hreflangCount": 0, "images": [ { "loc": "https://cdn.shopify.com/s/files/1/1104/4168/files/Allbirds_WL_RN_SF_PDP_Natural_Grey_LAT.png?v=1751143404", "title": "Men's Wool Runner - Natural Grey (Light Grey Sole)" } ], "videos": [], "news": null, "alternates": [] } ``` With **Check HTTP status** on, each row also gets: ```json "statusCode": 301, "finalUrl": "https://example.com/new-page/", "finalStatusCode": 200, "redirectCount": 1, "checkMethod": "HEAD", "statusError": null ``` Rows from a Google News sitemap carry the publication data (real row from theguardian.com): ```json "title": "NRL grand final 2026: Sydney Roosters v Newcastle Knights – live", "news": { "publicationName": "The Guardian", "language": "en", "publicationDate": "2026-10-04T09:13:45Z", "title": "NRL grand final 2026: Sydney Roosters v Newcastle Knights – live", "keywords": "NRL, Sydney Roosters, Newcastle Knights, Rugby league, Australia sport, Sport" } ``` Hreflang alternates (real row from spotify.com, 293 alternates) look like `"alternates": [{ "hreflang": "en-US", "href": "https://www.spotify.com/us/about-us/contact/" }, { "hreflang": "ca-AD", "href": "https://www.spotify.com/ad-ca/about-us/contact/" }, ...]`. The dataset has ready-made views: **All URLs**, **SEO status check**, **Google News**, **Images & videos** and **Hreflang alternates**. #### Run summary (key-value store, `SUMMARY`) Each run also saves a `SUMMARY` record with a report for every site: - status: `ok`, `partial`, `no-sitemap`, `unreachable`, `sitemap-error` or `empty` - how the sitemaps were found - every sitemap file read, with its type, URL count, HTTP status and any errors or recovered problems - totals: URLs found, output, duplicates, filtered, broken, redirected You can see exactly what happened, even for sites that failed. ### How much does it cost? The Actor uses **pay-per-event** pricing, so you pay only for results: | Event | Price (FREE plan) | |---|---| | Actor start | $0.003 per run | | URL extracted | **$0.50 per 1,000 URLs** | | URL status checked (optional) | $1.00 per 1,000 URLs | Paid Apify plans get lower prices. **Filtered and duplicate URLs are free**, and a site with no sitemap costs nothing beyond the start event. Set a **maximum cost per run** in the run options and the Actor stops cleanly when it reaches it. Examples: a 2,500-page store costs about $1.25. A daily "what changed today" check on a news site costs a few cents. ### FAQ **Do I need the sitemap URL?** No. Just enter the domain and the Actor finds the sitemaps itself. If you already know the sitemap URL, you can enter it directly. **What if the website has no sitemap?** The Actor tries robots.txt, about 20 common paths, homepage links and RSS/Atom feeds. If it finds nothing, the site is marked `no-sitemap` in the summary and the run still **succeeds**. To discover pages on sites without sitemaps, use a crawler such as Website Content Crawler. **Why are some URLs missing compared to the live site?** A sitemap contains what the site owner chose to list. Feeds usually include only the latest 10–50 items. **Does it handle huge sites?** Yes. It follows indexes with thousands of child sitemaps, handles files up to 200 MB, and has safety limits ("Max sitemap files per site", "Max URLs"). For very large sites, use "Changed since" and "Skip unchanged child sitemaps". **Can I monitor changes over time?** Yes. Create a [schedule](https://docs.apify.com/platform/schedules) with "Changed since: 1 day" (or 1 week) and connect a Google Sheets, Slack or webhook integration. **Is it legal?** Sitemaps and robots.txt exist so that machines can read them. The Actor reads only these public files, identifies itself with an honest User-Agent and respects robots.txt. The optional status check sends one lightweight request per URL with polite concurrency limits. **Does it need a proxy?** Usually not. If a site blocks datacenter IPs, enable Apify Proxy in the input. **Why does the status check use HEAD and then GET?** HEAD is the fastest way to check a URL, but many servers and CDNs wrongly answer HEAD with 403, 404 or 405. Every failed HEAD is re-checked with GET (the body is not downloaded), so you don't get false "broken page" alerts. ### Limitations - The Actor does not crawl HTML pages to find links. It reads sitemaps and feeds only. - Sitemaps that only appear after JavaScript runs are not supported (this is extremely rare). - `lastmod` is only as accurate as the website's own sitemap. ### Support Found a site that doesn't work? Open an issue on the **Issues** tab with the URL. We usually respond within 24 hours, and every reported site becomes a regression test. # Actor input Schema ## `startUrls` (type: `array`): Domains (e.g. example.com), homepages, or direct sitemap/feed URLs (e.g. https://example.com/sitemap_index.xml, .xml.gz, RSS/Atom, .txt). For a domain, sitemaps are discovered automatically from robots.txt, common CMS paths (WordPress, Yoast, Shopify, Wix, Squarespace...), homepage links and feeds. You can also upload a text file or link a URL list. ## `discoveryMode` (type: `string`): Auto: use robots.txt sitemaps; if there are none, probe common sitemap paths and homepage links. Thorough: always probe all common paths too (finds sitemaps that robots.txt forgot). robots.txt only: only sitemaps declared in robots.txt. None: treat every start URL as a sitemap URL. ## `feedMode` (type: `string`): Fallback: if a site has no XML sitemap, extract URLs from its RSS/Atom feed instead. Always: also add feed URLs for sites that have sitemaps. Never: ignore feeds. ## `modifiedSince` (type: `string`): Only output URLs whose lastmod is on or after this date. Accepts an absolute date (2026-09-01, 2026-09-01T12:00:00Z) or a relative period (24 hours, 7 days, 2 weeks, 3 months). Great for scheduled change monitoring. Leave empty for all URLs. ## `includeUrlsWithoutLastmod` (type: `boolean`): When 'Changed since' is set, URLs that have no lastmod date are dropped by default. Enable to keep them. ## `skipStaleSitemaps` (type: `boolean`): When 'Changed since' is set, skip child sitemaps whose own lastmod (in the sitemap index) is older than the date. Makes date-filtered runs on big news and e-commerce sites up to 10x faster and cheaper. ## `includePatterns` (type: `array`): Only keep URLs matching at least one of these regular expressions (case-insensitive). Example: /blog/ or /products/.\*. ## `excludePatterns` (type: `array`): Drop URLs matching any of these regular expressions (case-insensitive). Example: /tag/, ?page=. ## `sameDomainOnly` (type: `boolean`): Drop URLs on other domains than the start URL (subdomains are kept). ## `dedupe` (type: `boolean`): Output each URL only once per run, even if it appears in several sitemaps or sites. ## `maxUrls` (type: `integer`): Stop after this many URLs in total across all sites. 0 = no limit (all URLs). ## `maxUrlsPerSite` (type: `integer`): Stop each site after this many URLs. 0 = no limit. ## `maxSitemapsPerSite` (type: `integer`): Safety limit on how many sitemap files are downloaded per site (very large sites can have thousands). ## `maxDepth` (type: `integer`): How many levels of nested sitemap indexes to follow. Loops are detected automatically. ## `checkStatus` (type: `boolean`): Sends a HEAD request to every URL (falls back to GET when the server rejects HEAD) and records the status code, redirect count, final URL and final status. Slower and charged per checked URL. Off by default. ## `statusConcurrency` (type: `integer`): Maximum parallel status-check requests per website. Keep low to be polite to the server. ## `statusTimeoutSecs` (type: `integer`): Timeout for each status-check request. ## `includeExtensions` (type: `boolean`): Output Google image/video/news sitemap extensions and xhtml:link hreflang alternates for each URL. ## `respectRobotsTxt` (type: `boolean`): Do not fetch sitemap files that robots.txt disallows (sitemaps declared in robots.txt are always allowed). ## `requestTimeoutSecs` (type: `integer`): Maximum wait for a server to start responding (and between data chunks). Large sitemaps that keep streaming are allowed to take longer. ## `maxRetries` (type: `integer`): Retries for failed requests (network errors, timeouts, 429 and 5xx) with exponential backoff and Retry-After support. ## `siteConcurrency` (type: `integer`): How many websites to process at the same time. ## `userAgent` (type: `string`): User-Agent header for all requests. Leave empty for the default, which identifies this Actor. ## `proxyConfiguration` (type: `object`): Optional. Sitemaps are public and usually do not need a proxy; use one only if a site blocks datacenter IPs. ## Actor input object example ```json { "startUrls": [ { "url": "https://crawlee.dev" } ], "discoveryMode": "auto", "feedMode": "fallback", "includeUrlsWithoutLastmod": false, "skipStaleSitemaps": true, "sameDomainOnly": false, "dedupe": true, "maxUrls": 500, "maxUrlsPerSite": 0, "maxSitemapsPerSite": 1000, "maxDepth": 10, "checkStatus": false, "statusConcurrency": 10, "statusTimeoutSecs": 20, "includeExtensions": true, "respectRobotsTxt": true, "requestTimeoutSecs": 30, "maxRetries": 3, "siteConcurrency": 3, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `urls` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://crawlee.dev" } ], "maxUrls": 500, "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("mbabbu/sitemap-extractor-pro").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://crawlee.dev" }], "maxUrls": 500, "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("mbabbu/sitemap-extractor-pro").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://crawlee.dev" } ], "maxUrls": 500, "proxyConfiguration": { "useApifyProxy": false } }' | apify call mbabbu/sitemap-extractor-pro --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,mbabbu/sitemap-extractor-pro" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/2zCg4Nyn3K1uwrnSo/builds/Nm6SNLfaQkyoHqLQa/openapi.json