# Sitemap URL Extractor (`seemuapps/sitemap-extractor`) Actor Extract every URL from a website's sitemap.xml. Recursively walks nested sitemap indexes and returns loc, lastmod, changefreq, and priority for each page. - **URL**: https://apify.com/seemuapps/sitemap-extractor.md - **Developed by:** [Andrew](https://apify.com/seemuapps) (community) - **Categories:** Agents, Developer tools, Lead generation - **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $0.30 / 1,000 url extracteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap URL Extractor Pull every URL from any website's `sitemap.xml` — automatically walks nested sitemap indexes and returns a clean dataset with `loc`, `lastmod`, `changefreq`, and `priority` for each page. ### What you get - **URL** (`loc`) for every page listed in the site's sitemap - **Last modified** date (`lastmod`) — when each page was last updated - **Change frequency** (`changefreq`) — `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, `never` - **Priority** (`priority`) — relative importance of each URL (0.0 - 1.0) - **Source sitemap** — which sitemap file the URL came from (useful when a site splits its sitemap by section) - **Auto-discovery** — point at a homepage and the actor finds the sitemap via `robots.txt` or `/sitemap.xml` - **Gzipped sitemap support** — handles `.xml.gz` files transparently - **Recursive sitemap index walking** — follows nested `` files up to 5 levels deep ### Use cases - **SEO audits** — pull a full URL inventory before running site-wide checks (broken links, missing meta tags, schema validation) - **Content migration** — build a complete URL list when moving a site between platforms - **Crawl budget planning** — see how many URLs a site exposes and how recently each was updated - **Competitor research** — map out every page a competitor publishes - **Sitemap validation** — verify that your published sitemap actually contains the pages you expect - **Bulk URL scraping pipelines** — feed the output into another actor for screenshots, content extraction, or AI summarization ### How to use 1. Enter a **Website or Sitemap URL** — either a homepage like `https://www.example.com` (the actor auto-discovers the sitemap) or a direct sitemap URL like `https://www.example.com/sitemap.xml` 2. Set **Max Items** — `0` returns every URL in the entire sitemap tree 3. Choose whether to **Follow Sitemap Index** — on by default, so a single run pulls every URL from every child sitemap 4. Run the actor — results land in the **Dataset** tab 5. Export to JSON, CSV, Excel, or Google Sheets directly from the Apify console #### Extract every URL on a site ```json { "websiteUrl": "https://www.apify.com", "maxItems": 0, "followSitemapIndex": true } ```` #### Extract only the top-level sitemap ```json { "websiteUrl": "https://www.apify.com/sitemap.xml", "maxItems": 0, "followSitemapIndex": false } ``` ### Output format One dataset record per URL: ```json { "loc": "https://www.apify.com/store", "lastmod": "2024-08-12", "changefreq": "daily", "priority": "0.8", "sourceSitemap": "https://www.apify.com/sitemap.xml" } ``` Fields not present in the sitemap entry come back as `null`. ### Parameters | Field | Default | Description | |---|---|---| | **Website or Sitemap URL** | `https://www.apify.com` | Homepage URL (auto-discovered) or direct `.xml` / `.xml.gz` sitemap URL | | **Max Items** | `0` | Maximum URLs to return per run. `0` = unlimited | | **Follow Sitemap Index** | `true` | Recurse into child sitemaps when the top-level file is a sitemap index | ### Notes - Sitemap discovery first looks for `Sitemap:` directives in `/robots.txt`, then falls back to `/sitemap.xml` - Nested sitemap indexes are walked breadth-first; the actor de-duplicates sitemap URLs so circular references are safe - Recursion is capped at 5 levels deep and 1,000 total sitemaps as a safety net against runaway loops - Each fetched sitemap has a 30-second timeout — slow or unreachable child sitemaps are logged and skipped, the run continues - Gzip-compressed sitemaps (`*.xml.gz`) are decompressed automatically # Actor input Schema ## `websiteUrl` (type: `string`): Either a homepage URL (e.g. 'https://www.apify.com' — we'll auto-discover the sitemap via robots.txt or /sitemap.xml) or a direct sitemap URL (e.g. 'https://www.apify.com/sitemap.xml'). ## `maxItems` (type: `integer`): Maximum number of URLs to return. 0 = unlimited (walks the entire sitemap tree). ## `followSitemapIndex` (type: `boolean`): When the top-level sitemap is a sitemap index (a list of child sitemaps), recurse into each child sitemap. Disable to only extract URLs from the top-level file. ## Actor input object example ```json { "websiteUrl": "https://www.apify.com", "maxItems": 0, "followSitemapIndex": true } ``` # Actor output Schema ## `results` (type: `string`): Each record: { loc, lastmod, changefreq, priority, sourceSitemap } # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "websiteUrl": "https://www.apify.com", "maxItems": 0 }; // Run the Actor and wait for it to finish const run = await client.actor("seemuapps/sitemap-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "websiteUrl": "https://www.apify.com", "maxItems": 0, } # Run the Actor and wait for it to finish run = client.actor("seemuapps/sitemap-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "websiteUrl": "https://www.apify.com", "maxItems": 0 }' | apify call seemuapps/sitemap-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=seemuapps/sitemap-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap URL Extractor", "description": "Extract every URL from a website's sitemap.xml. Recursively walks nested sitemap indexes and returns loc, lastmod, changefreq, and priority for each page.", "version": "1.0", "x-build-id": "pAvJYLKNVDlqq4Ilm" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/seemuapps~sitemap-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-seemuapps-sitemap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/seemuapps~sitemap-extractor/runs": { "post": { "operationId": "runs-sync-seemuapps-sitemap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/seemuapps~sitemap-extractor/run-sync": { "post": { "operationId": "run-sync-seemuapps-sitemap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "websiteUrl" ], "properties": { "websiteUrl": { "title": "Website or Sitemap URL", "type": "string", "description": "Either a homepage URL (e.g. 'https://www.apify.com' — we'll auto-discover the sitemap via robots.txt or /sitemap.xml) or a direct sitemap URL (e.g. 'https://www.apify.com/sitemap.xml')." }, "maxItems": { "title": "Max Items", "minimum": 0, "type": "integer", "description": "Maximum number of URLs to return. 0 = unlimited (walks the entire sitemap tree).", "default": 0 }, "followSitemapIndex": { "title": "Follow Sitemap Index", "type": "boolean", "description": "When the top-level sitemap is a sitemap index (a list of child sitemaps), recurse into each child sitemap. Disable to only extract URLs from the top-level file.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```