# Sitemap URL Extractor: All URLs, Status Codes & New Pages (`genesis.labs.co/sitemap-url-extractor-monitor`) Actor Get every URL of a website from its sitemaps: auto-discovery via robots.txt, nested indexes, .gz, RSS/Atom, lastmod filters, optional status-code checks and monitoring of new, changed and removed URLs. Failed sitemaps are free. - **URL**: https://apify.com/genesis.labs.co/sitemap-url-extractor-monitor.md - **Developed by:** [Nicholas Powers](https://apify.com/genesis.labs.co) (community) - **Categories:** SEO tools, Developer tools, Automation - **Security:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.08 / 1,000 url saveds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap URL Extractor: All URLs, Status Codes & New Pages **Get every URL of any website from its XML sitemaps in seconds**, as a clean CSV, Excel or JSON list. Paste a domain and the Actor finds the sitemaps itself (robots.txt, `sitemap.xml`, `sitemap_index.xml`, WordPress, Shopify...), follows nested sitemap indexes, unzips `.xml.gz`, reads RSS/Atom feeds and text sitemaps, and gives you one row per URL with `lastmod`, `changefreq`, `priority` and the sitemap it came from. Turn on **status checks** to find 404s, redirects and `noindex` headers among the URLs your sitemap promotes, or turn on **monitoring** to get only the **new, changed or removed URLs** since the last run: new products, articles, job pages or landing pages on any site, on a schedule. **Failed sitemaps are free**, and the Actor stops before going over your max cost. ### What it does - ๐Ÿ”Ž **Sitemap auto-discovery**: `Sitemap:` lines in robots.txt, then `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt`. For `https://example.com/blog/` it first tries sitemaps under `/blog/`. Or paste any sitemap or feed URL directly. - ๐Ÿ—‚๏ธ **Every sitemap format**: ``, nested `` (configurable depth), gzip files, RSS 2.0 and Atom feeds, plain-text URL lists. Malformed XML (unescaped `&`, truncated files) is recovered instead of failing. - ๐Ÿ–ผ๏ธ **Sitemap extensions**: image URLs, video titles/content/thumbnail URLs, Google News titles and publication dates, and `hreflang` alternates. - ๐Ÿงน **Filters**: include/exclude patterns (text, `*` wildcards or `/regex/`), **modified after/before** (`2026-01-01` or `7 days`), same-site only, max URLs per site and in total. Duplicate URLs across sitemaps are removed. - ๐Ÿšฆ **Status checks (optional)**: HTTP status, final URL, redirect chain, content type, `X-Robots-Tag` and response time for each URL. HEAD first with automatic GET fallback; retries for timeouts, 5xx and 429. Save **only broken URLs** for a quick sitemap audit. - ๐Ÿ”” **Monitoring: only new URLs**: remembers each site's URLs between runs and outputs only new ones, or new + updated (newer `lastmod`), plus optionally removed URLs. Schedule it and connect an integration (email, Slack, webhook, Google Sheets, Zapier, Make) to get alerts. - ๐Ÿ”— **Integration friendly**: read websites from another Actor's dataset (`{{resource.defaultDatasetId}}`), runs with **limited permissions**, stable error codes, a run summary in the `OUTPUT` record. - ๐Ÿ›ก๏ธ **Safe and polite**: one global requests-per-second limit, honest User-Agent, private/internal addresses blocked, no browser and no login needed. ### Pricing Pay per event. Compute and storage are included. | Event | Free & Bronze | Silver | Gold & above | |---|---|---|---| | URL saved to the dataset | $0.00015 (**$0.15 per 1,000**) | $0.0001 | $0.00008 | | Status check (only when "Check status codes" is on) | $0.0002 ($0.20 per 1,000) | $0.00015 | $0.00012 | | Run start | $0.00005 per GB of memory (512 MB default = 1 event) | same | same | **Not charged**: sitemaps that fail to download or parse, URLs removed by your filters, duplicate URLs, unchanged URLs in monitoring mode, and status checks that end in a network error (DNS, timeout, refused connection) or a 429 rate limit after retries. **Examples (Free/Bronze prices):** | Job | Cost | |---|---| | Full URL list of a 5,000-page site | about **$0.75** | | 1,000 URLs from a Shopify store's product sitemaps, modified in the last 90 days | about **$0.15** | | Sitemap audit of 10,000 URLs with status checks, saving only the 40 broken ones | about **$2.01** | | 2,000 URLs with status codes and redirects | about **$0.70** | | Daily "new pages" monitor of 3 competitor sites, ~20 new URLs a day | about **$0.10 a month** | Set **Max cost per run** in the run options: the Actor checks the budget before saving or checking URLs and stops cleanly when it would go over. Running with empty input starts a **free offline demo** (a sample site with a sitemap index, 2 sitemaps, 7 URLs and status checks; nothing is charged), so you can see the output first. ### How to use it #### List every URL of a website 1. Add the website in **Websites or sitemap URLs** (`https://www.example.com`), or paste bare domains in **Domains or URLs (plain text)**. 2. Start the run. Download the list from the **Output** tab as CSV, Excel or JSON, or use the **plain URL list** link (one URL per line). ```json { "urls": ["crawlee.dev", "allbirds.com"] } ``` #### Only some sections, only recent pages ```json { "startUrls": [{ "url": "https://www.example.com" }], "includeUrlPatterns": ["/blog/", "/guides/*"], "excludeUrlPatterns": ["/tag/", "/author/"], "lastmodAfter": "30 days" } ``` #### Find broken URLs in a sitemap ```json { "startUrls": [{ "url": "https://www.example.com/sitemap_index.xml" }], "checkStatus": true, "onlyBroken": true } ``` Each row gets `status`, `isBroken`, `finalUrl`, `redirectCount`, `redirectChain`, `contentType`, `xRobotsTag` and `responseMs`. URLs that redirect are not marked broken, but `redirectCount > 0` shows sitemap entries you should update. #### Monitor sites for new pages ```json { "urls": ["competitor-one.com", "competitor-two.com/blog/"], "monitorMode": "new", "monitorName": "competitors", "silentFirstRun": true } ``` Create a **schedule** (daily or hourly) for this input. The first run just remembers the current URLs (with "Silent first run"), and every later run outputs only URLs that weren't there before, with `changeType: "new"`. Use `newAndChanged` to also get URLs whose `lastmod` got newer (`changeType: "changed"`, with `previousLastmod`), and **Also report removed URLs** for pages that disappeared. Removals are only reported when the whole sitemap was read without errors or limits, so a failed download never looks like a mass deletion. The memory lives in a key-value store named `sitemap-monitor-` on your account. Use different monitor names for different jobs, or **Reset memory** to start over. #### Websites from another Actor (integration) In a task or integration, set **Input dataset** to `{{resource.defaultDatasetId}}` and **URL field** to the field that holds the website (`url`, `website`, `company.website`). Each row's site is processed once. ### Output One item per URL: ```json { "url": "https://www.example.com/blog/sitemap-seo-checklist", "lastmod": "2026-10-28T09:30:00.000Z", "changefreq": "weekly", "priority": 0.8, "site": "https://www.example.com", "sitemap": "https://www.example.com/sitemap-blog.xml", "imageCount": 1, "images": ["https://www.example.com/img/checklist.png"], "alternates": [{ "hreflang": "de", "url": "https://www.example.com/de/blog/sitemap-seo-checklist" }], "changeType": "new", "status": 200, "isBroken": false, "finalUrl": "https://www.example.com/blog/sitemap-seo-checklist", "redirectCount": 0, "contentType": "text/html; charset=utf-8", "responseMs": 184, "checkedWith": "HEAD" } ``` Fields that don't apply are left out (no status fields without status checks, no `changeType` without monitoring). RSS/Atom items also get `title`; news sitemaps add `news.title`, `news.publicationDate`, `news.publication`; video sitemaps add `videos`. The `OUTPUT` record has a run summary: sites, sitemap files read and failed, URLs found/saved/filtered/duplicate, monitor counts (new, changed, unchanged, removed), status codes by count, per-site details (how the sitemap was found) and export links. ### Error codes Problems are listed in the `ERRORS` record and in `OUTPUT.failures` with a `code`. A run succeeds when at least one sitemap was read (turn on **Fail the run on any error** for strict integrations). | Code | Meaning | |---|---| | `NO_SITEMAP_FOUND` | robots.txt has no `Sitemap:` line and none of the usual locations exist. Paste the sitemap URL directly. | | `HTTP_403`, `HTTP_401` | The site refused access, often because it blocks datacenter IPs. Try **Proxy configuration**. | | `HTTP_404`, `HTTP_410` | A sitemap listed in robots.txt or an index doesn't exist. | | `HTTP_429`, `HTTP_5xx` | Rate limit or server error that persisted after retries. | | `NOT_A_SITEMAP` | The file is HTML or another format, not a sitemap or feed. | | `BAD_XML`, `BAD_GZIP` | The file couldn't be parsed or unzipped, even with recovery. | | `TOO_LARGE` | The file is bigger than **Max sitemap file size**. | | `TIMEOUT`, `CONNECTION`, `DNS`, `TLS` | Network problems (timeouts and connection resets are retried). | | `BLOCKED_HOST` | The URL points to a private or internal address. | | `INPUT`, `DATASET_ACCESS` | Invalid input, or the input dataset can't be read. | Status checks use `checkError`: `RATE_LIMITED` (429 after retries, not charged, status unknown) or a network code above. ### FAQ **Does it crawl the pages?** No. It reads only sitemaps (and, with status checks, sends one lightweight HEAD/GET per URL). That's why it's fast and cheap: 50,000 URLs take about 10 seconds. **What if a site has no sitemap?** You get `NO_SITEMAP_FOUND` and nothing is charged. A crawler is the right tool for such sites. **How big can a site be?** Hundreds of thousands of URLs work at the default 512 MB. For millions of URLs in monitoring mode, use 1-2 GB of memory. **Is it legal?** Sitemaps are public files that sites publish specifically so that machines can read them. The Actor reads only public URLs, needs no login and collects no personal data. Respect each site's terms when you use the results. **Can I get just a plain list of URLs?** Yes: the "Plain URL list" link in the Output tab, or `.../items?format=csv&fields=url&skipHeaderRow=1`. Questions or a sitemap that doesn't parse? Open an issue on the Actor's **Issues** tab with the sitemap URL. # Actor input Schema ## `startUrls` (type: `array`): Website addresses (example.com, https://example.com/blog/) or direct sitemap/feed URLs (https://example.com/sitemap_index.xml, .xml.gz, RSS/Atom, .txt). For a website, sitemaps are found via robots.txt and the usual locations. Leave empty to run a free demo. ## `urls` (type: `array`): Same as above, as plain text: bare domains work here (example.com, shop.example.org). Handy for pasting a list or passing an array of strings from an integration. ## `datasetId` (type: `string`): Read website/sitemap URLs from a dataset, e.g. the output of another Actor. In integrations pass {{resource.defaultDatasetId}}. Only this dataset is readable (limited permissions). ## `urlField` (type: `string`): Field holding the website or sitemap URL in each input row. Dot paths work, e.g. company.website. ## `maxInputItems` (type: `integer`): Read at most this many rows from the input dataset. 0 = all. ## `discoverSitemaps` (type: `boolean`): For website URLs, look up Sitemap: lines in robots.txt, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz and /sitemap.txt. Turn off to treat every input URL as a sitemap file. ## `includeUrlPatterns` (type: `array`): Keep only URLs that match at least one pattern. Plain text matches anywhere (/blog/), \* is a wildcard (https://example.com/products/\*), or use /regex/. ## `excludeUrlPatterns` (type: `array`): Drop URLs that match any pattern (same syntax), e.g. /tag/ or *?replytocom=*. ## `lastmodAfter` (type: `string`): Keep URLs whose is on or after this date: 2026-01-31 or a relative time like "7 days" or "24 hours". Old child sitemaps are skipped entirely. ## `lastmodBefore` (type: `string`): Keep URLs whose is on or before this date (same formats). Useful to find stale pages. ## `keepUrlsWithoutLastmod` (type: `boolean`): When a date filter is set, also keep URLs that have no . ## `sameSiteOnly` (type: `boolean`): Drop URLs on other hostnames than the site (www. is ignored). ## `includeSitemapExtras` (type: `boolean`): Add image URLs, video info, Google News fields and hreflang alternates when the sitemap has them. ## `maxUrls` (type: `integer`): Stop after saving this many URLs in total. 0 = no limit (use Max cost per run to cap spend). ## `maxUrlsPerSite` (type: `integer`): Per website or sitemap input. 0 = no limit. ## `maxSitemapsPerSite` (type: `integer`): Safety limit for huge sitemap indexes. ## `maxDepth` (type: `integer`): How many levels of nested sitemap indexes to follow. ## `monitorMode` (type: `string`): Remembers each site's URLs between runs (in a key-value store named sitemap-monitor- on your account) and outputs only what is new. Schedule it daily to get new pages, products or articles. ## `monitorName` (type: `string`): Separate memories for separate jobs, e.g. competitor-blogs. Letters, digits and dashes. ## `includeRemoved` (type: `boolean`): Output URLs that were in the sitemap last time but are gone now (changeType "removed"). Only reported when the whole sitemap was read without errors or limits. ## `silentFirstRun` (type: `boolean`): On the first run of a monitor, just remember the URLs and output nothing (nothing charged per URL), so later runs return only real changes. ## `resetMonitor` (type: `boolean`): Forget the remembered URLs of these sites before the run (everything counts as new). ## `checkStatus` (type: `boolean`): Request every saved URL and add its HTTP status, final URL after redirects, redirect chain, content type, X-Robots-Tag and response time. Finds 404s and redirects listed in your sitemap. ## `checkMethod` (type: `string`): HEAD is fastest; servers that reject HEAD (405/501/403/404) are re-checked with GET automatically. ## `onlyBroken` (type: `boolean`): Save only URLs with status 4xx/5xx or a network error. All checks are still counted. ## `checkConcurrency` (type: `integer`): Parallel status checks. ## `requestsPerSecond` (type: `integer`): Global politeness limit for all requests (sitemaps and status checks). ## `timeoutSecs` (type: `integer`): Per request. Timeouts and 5xx are retried twice. ## `maxSitemapMB` (type: `integer`): Larger files are skipped with TOO_LARGE (the sitemap protocol limit is 50 MB uncompressed). ## `proxyConfiguration` (type: `object`): Optional. Use a proxy if a site blocks datacenter IPs. By default requests go out directly. ## `failOnSiteError` (type: `boolean`): By default the run succeeds when at least one sitemap was read, and problems are listed in the ERRORS record. Turn on to make integrations notice any failed sitemap. ## Actor input object example ```json { "startUrls": [ { "url": "https://crawlee.dev" } ], "urlField": "url", "maxInputItems": 0, "discoverSitemaps": true, "keepUrlsWithoutLastmod": false, "sameSiteOnly": false, "includeSitemapExtras": true, "maxUrls": 0, "maxUrlsPerSite": 0, "maxSitemapsPerSite": 1000, "maxDepth": 5, "monitorMode": "off", "monitorName": "default", "includeRemoved": false, "silentFirstRun": false, "resetMonitor": false, "checkStatus": false, "checkMethod": "HEAD", "onlyBroken": false, "checkConcurrency": 10, "requestsPerSecond": 10, "timeoutSecs": 30, "maxSitemapMB": 60, "proxyConfiguration": { "useApifyProxy": false }, "failOnSiteError": false } ``` # Actor output Schema ## `urls` (type: `string`): No description ## `csv` (type: `string`): All URLs and fields as CSV. ## `xlsx` (type: `string`): No description ## `plain` (type: `string`): No description ## `summary` (type: `string`): No description ## `errors` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://crawlee.dev" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("genesis.labs.co/sitemap-url-extractor-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://crawlee.dev" }] } # Run the Actor and wait for it to finish run = client.actor("genesis.labs.co/sitemap-url-extractor-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://crawlee.dev" } ] }' | apify call genesis.labs.co/sitemap-url-extractor-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,genesis.labs.co/sitemap-url-extractor-monitor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/B0fXSkR9CDrMlPdqi/builds/wiarrIJF8ggY2THMe/openapi.json