# Sitemap URL Extractor (`openquarry/sitemap-url-extractor`) Actor Returns every page URL listed in the sitemaps of one or more websites, with lastmod, changefreq and priority. Finds sitemaps via robots.txt and common paths, and follows sitemap indexes and .gz files. No proxy or login needed. - **URL**: https://apify.com/openquarry/sitemap-url-extractor.md - **Developed by:** [OpenQuarry](https://apify.com/openquarry) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.25 / 1,000 url returneds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap URL Extractor Returns every page URL listed in the sitemaps of one or more websites, with `lastmod`, `changefreq` and `priority` for each URL. Give it websites or sitemap URLs. It finds the sitemaps through robots.txt and common paths, and follows sitemap indexes, `.gz` files and plain-text sitemaps. No proxy, login or API key needed. ### What you get One dataset item per page URL: | Field | Type | Description | |---|---|---| | `url` | string | Page URL listed in the sitemap | | `lastmod` | string or null | Last modification time from the sitemap (ISO 8601), or null if not given | | `changefreq` | string or null | Change frequency hint (`always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, `never`), or null | | `priority` | number or null | Priority hint from 0.0 to 1.0, or null | | `sitemapUrl` | string | The sitemap file that listed this URL | | `inputUrl` | string | The input entry this URL came from | Each URL appears once per input, even if several sitemaps list it. ### Input ```json { "urls": ["https://wordpress.org", "https://crawlee.dev/sitemap.xml"], "maxUrlsPerSite": 1000, "includeUrlPatterns": ["/blog/"], "excludeUrlPatterns": ["/tag/"], "modifiedAfter": "2026-01-01" } ``` - **`urls`** (required): websites (`example.com`) or direct sitemap URLs. For a website, the Actor reads the `Sitemap:` lines in robots.txt. If there are none, it tries `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`. - **`maxUrlsPerSite`**: maximum URLs per input (default 1,000; `0` means no limit). - **`includeUrlPatterns`** / **`excludeUrlPatterns`**: keep or drop URLs that contain any of these texts (case-insensitive). - **`modifiedAfter`**: keep only URLs with a `lastmod` on or after this date. URLs without a `lastmod` are dropped when this is set. ### Sample output ```json [ { "url": "https://crawlee.dev/blog", "lastmod": null, "changefreq": "weekly", "priority": 0.5, "sitemapUrl": "https://crawlee.dev/sitemap.xml", "inputUrl": "https://crawlee.dev/sitemap.xml" }, { "url": "https://www.sitemaps.org/protocol.html", "lastmod": "2022-12-15T00:00:00.000Z", "changefreq": null, "priority": null, "sitemapUrl": "https://www.sitemaps.org/sitemap.xml", "inputUrl": "https://www.sitemaps.org/sitemap.xml" } ] ``` ### Use cases - **RAG and LLM pipelines:** get a clean list of a site's pages to feed a crawler or embedding job. - **SEO audits:** see what a site tells search engines it has, and when each page last changed. - **Change monitoring:** run on a schedule with `modifiedAfter` to get only new or updated pages. - **Crawl planning:** scope a crawl to `/blog/` or `/docs/` before you spend money crawling. ### How it handles problems The run never fails because one site is broken. URLs from the other sites are still returned, and the run's status message lists every input that produced no URLs, with the reason: - `no sitemap found`: robots.txt lists no sitemap and none of the common paths exist. - `site unreachable`: DNS failure, refused connection or timeout. - `site blocked the request (HTTP 403 …)` or `sitemap unreadable: … HTTP 403`: the site refuses automated clients. - `invalid URL`: the input isn't a website or sitemap URL. ### Limits - **No browser disguise.** Requests identify themselves honestly as `openquarry-sitemap-url-extractor`. A few sites block unknown bots, and for those you'll get a "blocked" message instead of URLs. We don't get around such blocks. - **Sitemaps only.** The Actor doesn't crawl pages or check whether listed URLs still work. If a site has no sitemap, use a crawler instead. - **Big sites take longer.** A sitemap index with tens of thousands of URLs takes a minute or more with no limit. The default limit of 1,000 per site keeps runs to seconds. - At most 2,000 sitemap files are read per input. ### Pricing Pay per result: $0.25 per 1,000 URLs returned (67,000 URLs from developer.mozilla.org cost about $17). Apify adds a small start fee of $0.00005 per run. Sites with no sitemap cost you nothing beyond that start fee. Use `maxUrlsPerSite` or the run's maximum cost setting to cap spending. The Actor stops cleanly when your limit is reached. ### Changelog - **0.1** (2026-10): First release. # Actor input Schema ## `urls` (type: `array`): Websites (e.g. "example.com") or direct sitemap URLs (e.g. "https://example.com/sitemap.xml"). For a website, sitemaps are found via robots.txt, then /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. Sitemap indexes, .gz and .txt sitemaps are followed automatically. ## `maxUrlsPerSite` (type: `integer`): Maximum URLs returned for each input. 0 means no limit. Large sites can list millions of URLs, so keep this low for fast runs. ## `includeUrlPatterns` (type: `array`): Return only URLs that contain at least one of these texts (case-insensitive), e.g. "/blog/". Leave empty to return all URLs. ## `excludeUrlPatterns` (type: `array`): Skip URLs that contain any of these texts (case-insensitive), e.g. "/tag/". ## `modifiedAfter` (type: `string`): Return only URLs whose sitemap lastmod date is on or after this date (YYYY-MM-DD). URLs with no lastmod are skipped when this is set. ## Actor input object example ```json { "urls": [ "https://wordpress.org", "https://crawlee.dev/sitemap.xml" ], "maxUrlsPerSite": 1000, "includeUrlPatterns": [], "excludeUrlPatterns": [] } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://wordpress.org", "https://crawlee.dev/sitemap.xml" ] }; // Run the Actor and wait for it to finish const run = await client.actor("openquarry/sitemap-url-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://wordpress.org", "https://crawlee.dev/sitemap.xml", ] } # Run the Actor and wait for it to finish run = client.actor("openquarry/sitemap-url-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://wordpress.org", "https://crawlee.dev/sitemap.xml" ] }' | apify call openquarry/sitemap-url-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,openquarry/sitemap-url-extractor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/csBMe9wWKnjnooGdc/builds/m2prDSzQHgro3Wc48/openapi.json