# Website Sitemap Extractor (`glassventures/website-sitemap-extractor`) Actor Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range. - **URL**: https://apify.com/glassventures/website-sitemap-extractor.md - **Developed by:** [Glass Ventures](https://apify.com/glassventures) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website Sitemap Extractor Extract all URLs from any website's XML sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compressed sitemaps. Filter by URL pattern or date range. ### What does Website Sitemap Extractor do? Website Sitemap Extractor is a universal tool that pulls every URL listed in a website's XML sitemap. Unlike other sitemap tools, you don't need to know the sitemap location -- just provide the website URL and the actor finds sitemaps automatically by checking robots.txt, then falling back to common sitemap paths like /sitemap.xml. The actor handles the full complexity of real-world sitemaps: sitemap index files that reference dozens of child sitemaps, .gz compressed sitemaps, and deeply nested sitemap hierarchies. It follows every reference until all URLs are collected. Each extracted URL includes its metadata from the sitemap: last modification date, change frequency, priority score, and which specific sitemap file it came from. You can filter results by URL pattern (glob matching) or by date range to get exactly the data you need. ### Use Cases - **SEO auditors** -- Quickly inventory all indexed pages on a website, check lastmod dates, and identify orphan pages not in the sitemap - **Content marketers** -- Discover all blog posts, landing pages, and content URLs on competitor sites for content gap analysis - **Web crawl planning** -- Get a complete URL list before running expensive browser-based scrapers, so you can plan crawl budgets accurately - **Migration teams** -- Extract the full URL map of a website before domain migration to ensure every page gets a proper redirect - **AI training data discovery** -- Find all publicly listed URLs on a domain to build targeted web datasets for model training - **Developers** -- Validate sitemap structure, check for broken sitemap references, and audit sitemap completeness ### Features - Auto-discovers sitemaps from robots.txt (no need to know the sitemap URL) - Falls back to /sitemap.xml and other common locations if robots.txt has no sitemap directive - Follows sitemap index files (nested sitemaps) recursively - Supports .gz compressed sitemaps - Extracts full metadata: URL, lastmod, changefreq, priority, source sitemap - Filter by URL glob pattern (e.g., `*/blog/*`, `*.html`) - Filter by date range (lastmod-based) - Works on any website with an XML sitemap - Handles CDATA-wrapped content in sitemaps - Exports to JSON, CSV, Excel, or connect via API - No proxies needed for most sites (sitemaps are public) - Proxy support available for restricted networks ### How much will it cost? This actor is **free** to use. You only pay for Apify platform usage (compute and storage). | URLs Extracted | Estimated Cost | |---------------|---------------| | 1,000 | ~$0.01 | | 10,000 | ~$0.05 | | 100,000 | ~$0.25 | | 1,000,000 | ~$1.50 | | Cost Component | Per 100,000 URLs | |----------------|-------------------| | Platform compute (256 MB) | ~$0.15 | | Storage | ~$0.05 | | Proxy (if used) | ~$0.00 (not needed) | | **Total** | **~$0.20** | Sitemaps are lightweight XML files, so extraction is extremely fast and cheap compared to page-by-page scraping. ### How it discovers sitemaps 1. **robots.txt** -- The actor first fetches `{website}/robots.txt` and looks for `Sitemap:` directives. Most well-configured websites list their sitemaps here. 2. **Common paths** -- If robots.txt has no sitemap references, the actor tries standard locations: `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap.xml.gz`, `/sitemaps.xml`, `/sitemap/sitemap.xml`. 3. **Sitemap index files** -- When a sitemap turns out to be an index (containing references to other sitemaps), the actor follows every child sitemap recursively. 4. **Compressed sitemaps** -- `.gz` compressed sitemaps are automatically decompressed. ### How to use 1. Go to the Website Sitemap Extractor page on Apify Store 2. Click "Start" or "Try for free" 3. Enter website URLs (e.g., `https://www.apple.com`) or direct sitemap URLs 4. Optionally set URL filter pattern and date range 5. Set the maximum number of URLs to extract 6. Click "Start" and wait for the results ### Input parameters | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | startUrls | array | Website URLs to discover sitemaps from | - | | sitemapUrls | array | Direct sitemap XML URLs | - | | urlFilterPattern | string | Glob pattern to filter URLs (e.g., `*/blog/*`) | - | | dateFrom | string | Only include URLs modified after this date | - | | dateTo | string | Only include URLs modified before this date | - | | maxItems | number | Maximum URLs to extract (0 = unlimited) | 10000 | | proxyConfig | object | Proxy settings (not needed for most sites) | - | ### Output The actor produces a dataset with the following fields: ```json { "url": "https://www.apple.com/shop/buy-iphone", "loc": "https://www.apple.com/shop/buy-iphone", "lastmod": "2025-12-01T00:00:00.000Z", "changefreq": "daily", "priority": "0.8", "sitemapUrl": "https://www.apple.com/sitemap.xml", "scrapedAt": "2026-04-23T10:30:00.000Z" } ```` | Field | Type | Description | |-------|------|-------------| | url | string | The URL found in the sitemap | | loc | string | Raw loc value from the sitemap XML | | lastmod | string/null | Last modification date (if provided in sitemap) | | changefreq | string/null | Change frequency: always, hourly, daily, weekly, monthly, yearly, never | | priority | string/null | Priority value from 0.0 to 1.0 | | sitemapUrl | string | The sitemap file this URL was extracted from | | scrapedAt | string | ISO 8601 timestamp of extraction | ### Integrations Connect Website Sitemap Extractor with other tools: - **Apify API** -- REST API for programmatic access - **Webhooks** -- get notified when a run finishes - **Zapier / Make** -- connect to 5,000+ apps - **Google Sheets** -- export directly to spreadsheets #### API Example (Node.js) ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const run = await client.actor('YOUR_USERNAME/website-sitemap-extractor').call({ startUrls: [{ url: 'https://www.apple.com' }], maxItems: 1000, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(`Found ${items.length} URLs`); ``` #### API Example (Python) ```python from apify_client import ApifyClient client = ApifyClient('YOUR_TOKEN') run = client.actor('YOUR_USERNAME/website-sitemap-extractor').call(run_input={ 'startUrls': [{'url': 'https://www.apple.com'}], 'maxItems': 1000, }) items = client.dataset(run['defaultDatasetId']).list_items().items print(f'Found {len(items)} URLs') ``` #### API Example (cURL) ```bash curl "https://api.apify.com/v2/acts/YOUR_USERNAME~website-sitemap-extractor/runs" \ -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"startUrls": [{"url": "https://www.apple.com"}], "maxItems": 1000}' ``` ### Tips and tricks - Start with a small `maxItems` (100) to test before running large extractions - Use `urlFilterPattern` to extract only specific sections, e.g., `*/blog/*` for blog posts only - If you already know the sitemap URL, use `sitemapUrls` for faster processing (skips robots.txt discovery) - Date filters work on the `lastmod` field -- URLs without lastmod are always included - For very large sites (1M+ URLs), increase the actor memory to 1024 MB or higher ### FAQ **Q: Does this actor need login credentials?** A: No. XML sitemaps are publicly accessible files. No authentication is required. **Q: How fast is the extraction?** A: Very fast. The actor processes XML files, not rendered web pages. Expect 10,000-50,000 URLs per minute depending on sitemap size and server response times. **Q: What if a website has no sitemap?** A: The actor will check robots.txt and common sitemap paths. If no sitemap is found, the dataset will be empty and a warning is logged. Not all websites have sitemaps. **Q: Does it work with non-standard sitemap formats?** A: The actor handles standard XML sitemaps (urlset), sitemap index files (sitemapindex), .gz compressed sitemaps, and CDATA-wrapped content. Non-XML formats (e.g., plain text sitemap lists) are not supported. **Q: Why are some fields null?** A: The sitemap standard only requires the `` (URL) element. Fields like `lastmod`, `changefreq`, and `priority` are optional and many websites don't include them. **Q: Do I need proxies?** A: Usually no. Sitemaps are meant to be publicly accessible. Only use proxies if the website blocks direct access. ### Is it legal to scrape sitemaps? XML sitemaps are explicitly published by website owners for search engines and web crawlers to consume. They are public files designed to be read by automated tools. Web scraping of publicly available data is generally legal based on precedents like the LinkedIn v. HiQ Labs case. Always review and respect the target site's Terms of Service. For more information, see [Apify's blog on web scraping legality](https://blog.apify.com/is-web-scraping-legal/). ### Limitations - Only processes standard XML sitemaps (not plain text URL lists or HTML sitemaps) - URLs without `lastmod` cannot be filtered by date range (they are included by default) - Very large sitemaps (10M+ URLs) may require increased memory allocation - Some websites may rate-limit or block rapid sitemap requests (enable proxies in this case) ### Changelog - **v0.1** (2026-04-23) -- Initial release with auto-discovery, sitemap index support, .gz compression, URL and date filtering # Actor input Schema ## `startUrls` (type: `array`): Website URLs to extract sitemaps from. The actor will auto-discover sitemaps by checking robots.txt and /sitemap.xml. ## `sitemapUrls` (type: `array`): Direct URLs to sitemap XML files. Use this if you already know the sitemap location. Supports sitemap index files and .gz compressed sitemaps. ## `urlFilterPattern` (type: `string`): Glob pattern to filter extracted URLs. For example, '*/blog/*' will only include URLs containing /blog/. Leave empty to include all URLs. ## `dateFrom` (type: `string`): Only include URLs with lastmod date on or after this date. ## `dateTo` (type: `string`): Only include URLs with lastmod date on or before this date. ## `maxItems` (type: `integer`): Maximum number of URLs to extract. Use 0 or leave empty for unlimited. ## `proxyConfig` (type: `object`): Proxy settings. Most sitemaps are publicly accessible and don't require proxies. ## Actor input object example ```json { "startUrls": [ { "url": "https://www.apple.com" } ], "sitemapUrls": [], "maxItems": 10000 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://www.apple.com" } ], "sitemapUrls": [] }; // Run the Actor and wait for it to finish const run = await client.actor("glassventures/website-sitemap-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://www.apple.com" }], "sitemapUrls": [], } # Run the Actor and wait for it to finish run = client.actor("glassventures/website-sitemap-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://www.apple.com" } ], "sitemapUrls": [] }' | apify call glassventures/website-sitemap-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=glassventures/website-sitemap-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website Sitemap Extractor", "description": "Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.", "version": "0.1", "x-build-id": "qujnNVsFjZF31X9UE" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/glassventures~website-sitemap-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-glassventures-website-sitemap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/glassventures~website-sitemap-extractor/runs": { "post": { "operationId": "runs-sync-glassventures-website-sitemap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/glassventures~website-sitemap-extractor/run-sync": { "post": { "operationId": "run-sync-glassventures-website-sitemap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Website URLs", "type": "array", "description": "Website URLs to extract sitemaps from. The actor will auto-discover sitemaps by checking robots.txt and /sitemap.xml.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "sitemapUrls": { "title": "Direct Sitemap URLs", "type": "array", "description": "Direct URLs to sitemap XML files. Use this if you already know the sitemap location. Supports sitemap index files and .gz compressed sitemaps.", "items": { "type": "string" } }, "urlFilterPattern": { "title": "URL Filter Pattern", "type": "string", "description": "Glob pattern to filter extracted URLs. For example, '*/blog/*' will only include URLs containing /blog/. Leave empty to include all URLs." }, "dateFrom": { "title": "Date From", "type": "string", "description": "Only include URLs with lastmod date on or after this date." }, "dateTo": { "title": "Date To", "type": "string", "description": "Only include URLs with lastmod date on or before this date." }, "maxItems": { "title": "Max Items", "minimum": 0, "type": "integer", "description": "Maximum number of URLs to extract. Use 0 or leave empty for unlimited.", "default": 10000 }, "proxyConfig": { "title": "Proxy Configuration", "type": "object", "description": "Proxy settings. Most sitemaps are publicly accessible and don't require proxies." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```