🗺️ Sitemap Generator – XML & TXT Sitemap Builder with Website Crawler

🚀 Generate XML sitemaps, TXT sitemaps, and sitemap-index files by automatically crawling any website or from your own custom URL list. Supports image sitemaps with <image:image> tags, robots.txt compliance, smart priority & changefreq based on page depth, and automatic sitemap splitting for large sites. Download ready-to-submit sitemaps for Google Search Console, Bing Webmaster Tools, and Yandex. Built with Apify best practices for reliability, speed, and scale.

---

📋 Table of Contents

• ✨ Features
• 🎯 Use Cases
• ⚡ Quick Start
• 📊 Input Parameters
• 📤 Output Format
• ❓ FAQ
• 🛠️ Troubleshooting
• 🏷️ Tags
• 🚀 Get Started Now

---

✨ Features

• Website Crawler – Automatically crawl any website from a seed URL, discover all pages breadth-first, and build a complete sitemap
• Custom URL List – Generate sitemaps instantly from your own URL list without crawling — perfect for pre-built inventories
• XML & TXT Formats – Output sitemaps in standard XML, plain TXT, or both formats simultaneously
• Image Sitemaps – Collect <image:image> tags from <img> elements and og:image meta tags for Google Image Search indexing
• Sitemap Index Splitting – Automatically splits large sitemaps into numbered files with a sitemap-index.xml when URLs exceed the threshold
• robots.txt Compliance – Respects Disallow rules from robots.txt during crawling (configurable)
• Smart Priority & Changefreq – Auto-computed from page depth (home=1.0, depth-1=0.8, ...) or set fixed values for all URLs
• Same-Domain Crawling – Stays on the seed URL's domain during breadth-first link discovery
• Direct Cloud Storage – All sitemap files are automatically saved to Apify Key-Value Store with instant public download URLs
• Crawl Statistics – Get detailed stats: pages crawled, URLs found, images discovered, duration, and more

---

🎯 Use Cases

Use Case	Description	Key Benefits
SEO & Google Search Console	Generate XML sitemaps for submission to Google, Bing, and Yandex search engines	Faster indexing, better crawl coverage, improved SEO rankings
Website Audit & SEO Analysis	Crawl a website to discover all pages, check URL structure, and build a complete page inventory	Identify orphan pages, broken links, and crawl depth issues
E-commerce Product Indexing	Generate sitemaps for thousands of product pages with image tags for Google Shopping	Image search visibility, structured product indexing
Blog & Content Sitemap	Auto-generate sitemaps for WordPress, Ghost, or custom blogs with proper lastmod and changefreq	Fresh content indexed faster, proper content priority
Multi-site Sitemap Management	Generate sitemaps from pre-built URL lists for multiple domains or microsites	Centralized sitemap management, bulk generation
Website Migration	Crawl old site structure and generate a sitemap for the new domain to ensure nothing is missed	Complete URL discovery, migration validation
Automated Sitemap Updates	Schedule recurring runs to keep sitemaps fresh and up-to-date with new content	Always current sitemaps, automated SEO maintenance

---

⚡ Quick Start

1. Crawl a Website – Auto-discover all pages

{
  "mode": "crawl",
  "url": "https://example.com",
  "maxPages": 100,
  "includeImages": true,
  "format": "both"
}

2. Generate from URL List – No crawling needed

{
  "mode": "generate",
  "urls": [
    { "loc": "https://example.com/", "lastmod": "2026-10-01" },
    { "loc": "https://example.com/blog/post-1", "images": ["https://example.com/img/cover.jpg"] },
    "https://example.com/about",
    "https://example.com/contact"
  ],
  "siteUrl": "https://example.com",
  "changefreq": "weekly",
  "priority": "0.7",
  "format": "xml"
}

3. Large Website – With image sitemaps & custom settings

{
  "mode": "crawl",
  "url": "https://myshop.com",
  "maxPages": 500,
  "includeImages": true,
  "respectRobots": true,
  "format": "xml",
  "urlsPerFile": 10000,
  "changefreq": "daily",
  "priority": "auto"
}

---

📊 Input Parameters

Mode Selection

Parameter	Type	Default	Description	Example
mode	String	"crawl"	Choose how to generate the sitemap: crawl (auto-discover pages) or generate (from your URL list)	"crawl", "generate"

Crawl Mode Settings

Parameter	Type	Default	Description	Example
url	String	—	Seed URL to start crawling. The crawler stays on the same domain and discovers pages breadth-first. Required in crawl mode.	"https://example.com"
maxPages	Integer	30	Maximum number of pages to crawl (1–500). Server hard cap is 500.	50, 200, 500
respectRobots	Boolean	true	Follow robots.txt Disallow rules during crawling. Set to false to crawl all pages.	true, false
includeImages	Boolean	false	Add <image:image> tags to the sitemap, collected from <img> elements and og:image meta tags on each page.	true, false

Generate Mode Settings

Parameter	Type	Default	Description	Example
urls	Array	—	Your list of URLs to include in the sitemap (max 100,000). Each item can be a plain URL string or an object with { loc, images?, lastmod? }. Duplicates are auto-removed. Required in generate mode.	["https://example.com/page1", {"loc": "https://example.com/page2", "lastmod": "2026-01-01"}]
baseUrl	String	—	Base URL for resolving relative paths in your URL list.	"https://example.com"
lastmod	String	today	Default lastmod date (YYYY-MM-DD) applied to URLs without their own.	"2026-10-01"

Common Settings (Both Modes)

Parameter	Type	Default	Description	Example
format	String	"both"	Output format: xml (standard XML sitemap), txt (plain URL list), or both (generate both formats)	"xml", "txt", "both"
urlsPerFile	Integer	50000	Maximum URLs per sitemap file. When exceeded, output is automatically split into numbered files with a sitemap-index.xml. Protocol limit: 50,000.	10000, 25000, 50000
siteUrl	String	seed origin	The origin where you will host the sitemap files. Used for absolute <loc> entries inside the sitemap-index.	"https://example.com"
changefreq	String	"auto"	How frequently pages change: auto (depth-based), always, hourly, daily, weekly, monthly, yearly, never	"auto", "weekly", "daily"
priority	String	"auto"	URL priority value (0.0–1.0). auto uses depth heuristic: home=1.0, depth-1=0.8, depth-2=0.6, etc.	"auto", "0.8", "0.5"

ℹ️ Note: In crawl mode, provide a url. In generate mode, provide a urls array. The mode parameter determines which settings are used.

---

📤 Output Format

Dataset Record Example

{
  "source": "sitemap_generator",
  "mode": "crawl",
  "source_url": "https://example.com",
  "total_urls": 47,
  "total_files": 3,
  "pages_crawled": 50,
  "duration_ms": 12450,
  "images_found": 23,
  "files": [
    {
      "filename": "sitemap.xml",
      "download_url": "https://api.apify.com/v2/key-value-stores/abc123/records/sitemap.xml",
      "content_type": "application/xml",
      "size_bytes": 8540,
      "url_count": 47
    },
    {
      "filename": "sitemap.txt",
      "download_url": "https://api.apify.com/v2/key-value-stores/abc123/records/sitemap.txt",
      "content_type": "text/plain",
      "size_bytes": 2350,
      "url_count": 47
    },
    {
      "filename": "crawled-entries.json",
      "download_url": "https://api.apify.com/v2/key-value-stores/abc123/records/crawled-entries.json",
      "content_type": "application/json",
      "size_bytes": 5120,
      "url_count": 47
    }
  ],
  "format": "both",
  "changefreq": "auto",
  "priority": "auto",
  "include_images": true,
  "respect_robots": true,
  "max_pages": 50,
  "urls_per_file": 50000,
  "key_value_store_id": "abc123-def456",
  "status": "success",
  "generated_at": "2026-10-25T14:30:00.000Z"
}

Output Fields Explained

Field	Type	Description
mode	String	Mode used: crawl or generate
source_url	String	The crawled URL or count of URLs provided
total_urls	Integer	Total unique URLs found and included in the sitemap
total_files	Integer	Number of sitemap files generated
pages_crawled	Integer	Number of pages actually crawled (crawl mode only)
duration_ms	Integer	Total crawl/generation time in milliseconds
images_found	Integer	Number of images discovered (crawl mode with includeImages)
files	Array	Array of generated file objects with download URLs
files[].filename	String	Generated file name (e.g., sitemap.xml, sitemap.txt, sitemap-index.xml)
files[].download_url	String	Direct public download URL from Apify Key-Value Store
files[].content_type	String	MIME type: application/xml, text/plain, or application/json
files[].size_bytes	Integer	File size in bytes
files[].url_count	Integer	Number of URLs contained in this file
status	String	Execution status: success or error
generated_at	String	ISO 8601 timestamp of sitemap generation

---

❓ FAQ

• Q: Where are my generated sitemap files saved?

  • All sitemap files (XML, TXT, sitemap-index) are automatically stored in the run's default Apify Key-Value Store. Direct download_url links are returned in the dataset for instant access and downloading.

• Q: Can I submit the generated sitemap directly to Google Search Console?

  • Yes! Download the sitemap.xml file using the provided download URL and upload it to Google Search Console, Bing Webmaster Tools, or add the URL to your robots.txt file.

• Q: What's the difference between Crawl and Generate modes?

  • Crawl mode automatically discovers all pages on a website by following links from your seed URL — perfect when you don't have a URL list. Generate mode builds a sitemap from URLs you provide — ideal for pre-built inventories, e-commerce catalogs, or migrated sites.

• Q: How does the image sitemap work?

  • When includeImages is enabled, the crawler collects image URLs from <img> elements and og:image meta tags on each page. These are added as <image:image> tags inside the XML sitemap, helping Google Image Search discover and index your images.

• Q: What happens when my site has more than 50,000 URLs?

  • The sitemap protocol limits each file to 50,000 URLs. When exceeded, the actor automatically splits the output into numbered sitemap files (sitemap-1.xml, sitemap-2.xml, ...) and generates a sitemap-index.xml that references them all. You can customize the split threshold with urlsPerFile.

• Q: Does the crawler follow robots.txt rules?

  • Yes, by default respectRobots is true, so the crawler honors Disallow directives in the site's robots.txt. Set it to false if you want to crawl all pages regardless.

• Q: How many pages can I crawl?

  • The maximum is 500 pages per run (server hard cap). The default is 30 pages. Increase maxPages to crawl larger sites.

• Q: Can I schedule automatic sitemap generation?

  • Absolutely. Use Apify's built-in scheduling feature to run this actor daily, weekly, or at any interval. Combined with webhooks, you can auto-upload fresh sitemaps to your server or CDN.

• Q: Can I use this with other Apify Actors or integrate via API?

  • Yes. You can call this actor via the Apify API, integrate it in Apify Workflows, or use webhooks to automatically process generated sitemaps — upload to S3, Google Cloud Storage, or trigger other automation tools like Zapier and Make.

---

🛠️ Troubleshooting

Issue	Cause	Solution
"No URL provided" error	url was not set in crawl mode	Provide a valid seed URL like "https://example.com"
"No URLs provided" error	urls array was empty or missing in generate mode	Provide an array of at least one URL
Only 1 page crawled	The website may not have internal links, or robots.txt is blocking	Check the site's link structure; try "respectRobots": false
url_count is 0 in output	The sitemap content might be empty	Verify the seed URL is accessible and has crawlable pages
400 Bad Request	Invalid or malformed input parameters	Check that url is a valid URL and mode is "crawl" or "generate"
413 Payload Too Large	More than 100,000 URLs in generate mode	Reduce your URL list to under 100,000 or split into multiple runs
422 No Valid URLs	No valid URLs were found during crawling or in your list	Check that URLs are properly formatted and accessible
429 Rate Limit Exceeded	Too many requests sent in a short time	Wait and retry — crawl: 10 req/min, generate: 30 req/min
Request timed out	Website is very large or the server is busy	Reduce maxPages or try again later

---

🏷️ Tags

sitemap generator, xml sitemap, txt sitemap, sitemap builder, website crawler, sitemap creator, google sitemap, seo sitemap, sitemap-index, image sitemap, sitemap xml generator, sitemap crawler, url sitemap, apify sitemap, search engine optimization, google search console, bing webmaster, robots.txt, website indexing, seo tools, apify actor, sitemap api, bulk sitemap, automated sitemap

---

🚀 Get Started Now

Ready to generate your sitemap?

1. Choose your mode — crawl to discover pages or generate from your URL list
2. Enter your website URL or provide your URL array
3. Configure format, priority, and changefreq settings
4. Click Start to run the actor
5. Download your sitemap files via the download URLs in the output
6. Click "Try for free" to test the actor

If this actor helps your SEO workflow, please give it a ⭐ star!

---

Built with Apify Platform