# Sitemap Generator (`automation-lab/sitemap-generator`) Actor Crawl any website and automatically generate a standards-compliant XML sitemap. Discovers all internal pages, extracts last-modified dates and page titles, and lets you download sitemap.xml directly. Configurable crawl depth and URL filters. Ideal for SEO audits and site migrations. - **URL**: https://apify.com/automation-lab/sitemap-generator.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** SEO tools, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Sitemap Generator **Generate a complete XML sitemap for any website in minutes — no technical setup required.** Automatically crawl every page of a website and produce a standards-compliant XML sitemap ready for Google Search Console, Bing Webmaster Tools, and any other SEO platform. Each URL is also saved to a structured dataset for further analysis or integration. --- ### What Does It Do? Sitemap Generator crawls your website starting from the URL you provide. It follows every internal link it discovers, collects page metadata (title, content type, last-modified date), and compiles all of this into a well-formed XML sitemap that follows the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). Key capabilities: - Crawls up to 10,000 pages per run - Generates XML sitemap saved directly to the key-value store for download - Optionally includes image URLs as `` elements for Google Image Search - Extracts page titles and last-modified dates from HTTP headers - Tracks crawl depth per URL - Stays strictly within the same hostname (never follows external links) - Normalizes URLs to avoid duplicates - Skips non-page assets: PDFs, images, CSS, JS, zip files, and more --- ### Who Is It For? **SEO professionals** who need a fresh sitemap to submit to search engines after a site rebuild or migration. **Developers** who want to generate a sitemap for a new site quickly without installing any software. **Content managers** who want an inventory of all pages on their site for auditing purposes. **Digital agencies** managing multiple client websites who need to automate sitemap generation at scale. --- ### Why Use This Actor? - **Zero setup** — no software to install, no command line required - **Fast** — uses lightweight HTTP crawling (no browser), processes up to 10 parallel requests - **Standards-compliant** — produces valid XML following the Sitemaps 0.9 protocol - **Downloadable output** — sitemap.xml is saved to the key-value store for direct download - **Structured data** — every URL also saved to a dataset for analysis in spreadsheets or databases - **Pay per use** — charged per page crawled, not a flat fee. Small sites cost fractions of a cent. - **Image sitemaps** — optional Google Image Sitemap extension support --- ### Data Fields Each URL found during the crawl produces one record in the dataset: | Field | Type | Description | |-------|------|-------------| | `url` | string | The full, normalized URL of the page | | `title` | string / null | The HTML `` of the page | | `statusCode` | number | HTTP status code returned (e.g., 200) | | `depth` | number | Crawl depth from the start URL (0 = start page) | | `changefreq` | string | The change frequency set for this URL | | `priority` | number | The priority value (start URL always 1.0, others configurable) | | `lastmod` | string / null | Date in YYYY-MM-DD format from Last-Modified header or today's date | | `contentType` | string / null | MIME type of the page (e.g., text/html) | | `images` | string[] | List of image URLs found on the page (only if includeImages is enabled) | --- ### Pricing This actor uses **Pay Per Event** billing: | Event | Cost | |-------|------| | Run started | $0.01 (one-time per run) | | Per page crawled | $0.001 | **Example costs:** - 50-page site: $0.01 + 50 × $0.001 = **$0.06** - 500-page site: $0.01 + 500 × $0.001 = **$0.51** - 5,000-page site: $0.01 + 5,000 × $0.001 = **$5.01** --- ### How to Use #### Step 1: Enter your website URL In the **Website URL** field, enter the full URL of the site you want to crawl, including the protocol: ```` https://example.com ```` #### Step 2: Set the page limit The **Max pages to crawl** field controls how many pages the actor visits. The default is 100. For large sites, increase this up to 10,000. #### Step 3: Configure sitemap options - **Include image URLs** — Enable this if you want Google Image Search indexing support. Each page's images will be listed in the sitemap. - **Change frequency** — Choose how often your pages typically change. This hint helps search engines schedule re-crawling. - **URL priority** — Set the relative importance of pages (0.0–1.0). The start URL is always set to 1.0 automatically. #### Step 4: Run the actor Click **Start**. The actor will begin crawling immediately. You can watch progress in the logs. #### Step 5: Download your sitemap When the run completes: 1. Click **Storage** in the run details 2. Go to **Key-value store** 3. Find `sitemap.xml` and click to download You can also access it directly via the Apify API. --- ### Input Configuration ```json { "startUrl": "https://example.com", "maxPages": 100, "includeImages": false, "changefreq": "weekly", "priority": 0.5 } ```` #### Input fields | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `startUrl` | string | Yes | — | Full website URL to crawl | | `maxPages` | integer | No | 100 | Maximum pages to crawl (1–10,000) | | `includeImages` | boolean | No | false | Include image URLs in sitemap | | `changefreq` | string | No | weekly | Change frequency for all URLs | | `priority` | number | No | 0.5 | Default URL priority (0–1) | **changefreq options:** `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, `never` *** ### Output #### Key-value store: `sitemap.xml` A complete XML sitemap following the Sitemaps 0.9 protocol: ```xml <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://example.com/</loc> <lastmod>2024-01-15</lastmod> <changefreq>weekly</changefreq> <priority>1</priority> </url> <url> <loc>https://example.com/about</loc> <lastmod>2024-01-15</lastmod> <changefreq>weekly</changefreq> <priority>0.5</priority> </url> </urlset> ``` #### Dataset Every URL is also stored as a structured record in the dataset. You can export this to JSON, CSV, or Excel directly from the Apify platform. *** ### Tips and Best Practices **Start with a small limit first.** Run with `maxPages: 20` to verify the crawler is finding the right pages before committing to a full crawl. **Enable image sitemaps for e-commerce and photography sites.** If you sell products with images or run a portfolio site, enabling `includeImages` can boost image search visibility. **Set changefreq based on your publishing cadence.** A news site updates daily; a brochure site might be monthly. Accurate hints help search engines prioritize. **Submit your sitemap to Search Console.** After downloading `sitemap.xml`, go to Google Search Console → Sitemaps and paste your sitemap URL, or upload the file. **For very large sites, increase timeout.** If your site has thousands of pages, the default 600-second timeout should be sufficient. The actor processes pages in parallel (10 concurrent requests). **The start URL always gets priority 1.0.** All other pages use the `priority` value you set. You do not need to manually set the homepage priority. **Robots.txt is respected by default.** The underlying CheerioCrawler respects robots.txt directives, so the actor will not crawl pages blocked by robots.txt. *** ### Integrations #### Zapier / Make Use the Apify integration to trigger sitemap generation on a schedule (weekly, monthly) and automatically download or email the resulting sitemap.xml file. #### Google Search Console API Combine this actor with the Search Console API to automatically submit your freshly generated sitemap after every crawl. #### Apify Scheduler Create a scheduled run in Apify to regenerate your sitemap automatically — for example, every Monday morning. *** ### API Usage #### Node.js ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_API_TOKEN' }); const run = await client.actor('automation-lab/sitemap-generator').call({ startUrl: 'https://example.com', maxPages: 100, includeImages: false, changefreq: 'weekly', priority: 0.5, }); console.log('Run finished:', run.status); console.log('Dataset ID:', run.defaultDatasetId); // Download the sitemap const store = client.keyValueStore(run.defaultKeyValueStoreId); const sitemap = await store.getRecord('sitemap.xml'); console.log(sitemap.value); ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient('YOUR_API_TOKEN') run = client.actor('automation-lab/sitemap-generator').call(run_input={ 'startUrl': 'https://example.com', 'maxPages': 100, 'includeImages': False, 'changefreq': 'weekly', 'priority': 0.5, }) print(f"Run finished: {run['status']}") print(f"Dataset ID: {run['defaultDatasetId']}") ## Download the sitemap store = client.key_value_store(run['defaultKeyValueStoreId']) record = store.get_record('sitemap.xml') print(record['value']) ``` #### cURL ```bash ## Start a run curl -X POST \ "https://api.apify.com/v2/acts/automation-lab~sitemap-generator/runs?token=YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "startUrl": "https://example.com", "maxPages": 100, "includeImages": false, "changefreq": "weekly", "priority": 0.5 }' ## Get run status (replace RUN_ID) curl "https://api.apify.com/v2/actor-runs/RUN_ID?token=YOUR_API_TOKEN" ## Download sitemap.xml (replace STORE_ID) curl "https://api.apify.com/v2/key-value-stores/STORE_ID/records/sitemap.xml?token=YOUR_API_TOKEN" ``` *** ### Use with Claude AI (MCP) This actor is available as a tool in Claude AI through the Model Context Protocol (MCP). Add it to Claude Desktop, Cursor, Windsurf, or any MCP-compatible client. #### Setup for Claude Code ```bash claude mcp add --transport http apify "https://mcp.apify.com" ``` #### Setup for Claude Desktop, Cursor, or VS Code Add this to your MCP config file: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com" } } } ``` #### Example prompts - "Generate an XML sitemap for example.com crawling up to 200 pages and give me the download link." - "Create a sitemap for my e-commerce site at shop.example.com with image URLs included for Google Image Search." - "Crawl https://docs.example.com and generate a sitemap with weekly change frequency for all pages found." Learn more in the [Apify MCP documentation](https://docs.apify.com/platform/integrations/mcp). *** ### Legality Web crawling for sitemap generation is generally considered legitimate use and is commonly practiced by SEO tools, search engines, and webmasters. The actor: - Respects `robots.txt` directives by default - Crawls only the site you specify (stays on-domain) - Uses standard HTTP requests without bypassing login walls or paywalls - Does not store any personal data Always ensure you have permission to crawl a website you do not own. Some websites' terms of service restrict automated crawling. *** ### FAQ **Q: Why doesn't the actor find all pages on my site?** A: The actor discovers pages by following links. If your site uses JavaScript-rendered navigation (single-page apps), links may not appear in the HTML. Consider using a JavaScript-enabled crawling approach for SPAs. **Q: Can I crawl multiple websites in one run?** A: No, each run is scoped to one starting URL and hostname. Run the actor multiple times for multiple sites. **Q: The sitemap only has a few URLs — my site has many more.** A: Increase `maxPages` and run again. The actor stops when it reaches the page limit, not when the site is fully crawled. **Q: Does it follow redirects?** A: Yes. The actor follows HTTP redirects automatically and records the final URL in the sitemap. **Q: Can I set different priority values per page?** A: Not currently — the actor applies a uniform priority to all pages except the start URL (always 1.0). Priority per-page is a planned future feature. **Q: Is the XML output valid?** A: Yes. The output follows the Sitemaps 0.9 protocol and includes the correct XML namespace declarations. *** ### Related Tools - [Broken Link Checker](https://apify.com/automation-lab/broken-link-checker) — Find 404 errors and broken links across your site - [Canonical URL Checker](https://apify.com/automation-lab/canonical-url-checker) — Audit canonical tags and detect duplicate content issues - [Robots & Sitemap Analyzer](https://apify.com/automation-lab/robots-sitemap-analyzer) — Analyze your robots.txt and existing sitemap configuration - [Accessibility Checker](https://apify.com/automation-lab/accessibility-checker) — Check pages for WCAG accessibility compliance # Actor input Schema ## `startUrl` (type: `string`): The website to crawl and generate a sitemap for. Must be a full URL including the protocol (e.g., https://example.com). The crawler stays within this domain. ## `maxPages` (type: `integer`): Maximum number of internal pages to crawl and include in the sitemap. Increase for larger sites. Each page crawled triggers a per-event charge. ## `includeImages` (type: `boolean`): Extract and include image URLs found on each page as <image:image> elements in the sitemap. Useful for Google Image Search indexing. ## `changefreq` (type: `string`): Default <changefreq> value for all URLs in the sitemap. Tells search engines how often the page is likely to change. ## `priority` (type: `number`): Default <priority> value for all URLs (0.0–1.0). Higher values signal more importance to search engines. The start URL is always set to 1.0. ## Actor input object example ```json { "startUrl": "https://example.com", "maxPages": 50, "includeImages": false, "changefreq": "weekly", "priority": 0.5 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '<YOUR_API_TOKEN>' with your token const client = new ApifyClient({ token: '<YOUR_API_TOKEN>', }); // Prepare Actor input const input = { "startUrl": "https://example.com", "maxPages": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/sitemap-generator").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '<YOUR_API_TOKEN>' with your token. client = ApifyClient("<YOUR_API_TOKEN>") # Prepare the Actor input run_input = { "startUrl": "https://example.com", "maxPages": 50, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/sitemap-generator").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrl": "https://example.com", "maxPages": 50 }' | apify call automation-lab/sitemap-generator --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/sitemap-generator", "--header", "Authorization: Bearer <YOUR_API_TOKEN>" ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Sitemap Generator", "description": "Crawl any website and automatically generate a standards-compliant XML sitemap. Discovers all internal pages, extracts last-modified dates and page titles, and lets you download sitemap.xml directly. Configurable crawl depth and URL filters. Ideal for SEO audits and site migrations.", "version": "0.1", "x-build-id": "bnyFU7gmTszKhtG5b" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~sitemap-generator/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-sitemap-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~sitemap-generator/runs": { "post": { "operationId": "runs-sync-automation-lab-sitemap-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~sitemap-generator/run-sync": { "post": { "operationId": "run-sync-automation-lab-sitemap-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrl" ], "properties": { "startUrl": { "title": "Website URL", "type": "string", "description": "The website to crawl and generate a sitemap for. Must be a full URL including the protocol (e.g., https://example.com). The crawler stays within this domain." }, "maxPages": { "title": "Max pages to crawl", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of internal pages to crawl and include in the sitemap. Increase for larger sites. Each page crawled triggers a per-event charge.", "default": 100 }, "includeImages": { "title": "Include image URLs", "type": "boolean", "description": "Extract and include image URLs found on each page as <image:image> elements in the sitemap. Useful for Google Image Search indexing.", "default": false }, "changefreq": { "title": "Change frequency", "enum": [ "always", "hourly", "daily", "weekly", "monthly", "yearly", "never" ], "type": "string", "description": "Default <changefreq> value for all URLs in the sitemap. Tells search engines how often the page is likely to change.", "default": "weekly" }, "priority": { "title": "URL priority", "minimum": 0, "maximum": 1, "type": "number", "description": "Default <priority> value for all URLs (0.0–1.0). Higher values signal more importance to search engines. The start URL is always set to 1.0.", "default": 0.5 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```