Sitemap Generator

Generate XML sitemaps by crawling any website. Automatically discover all pages, images, and videos with configurable crawl depth, URL filters, and multiple output formats.

Features

• Full Site Crawling: Automatically discovers and indexes all pages on your website
• Image Sitemap: Include images in your sitemap for Google Images indexing
• Video Sitemap: Include videos for Google Video indexing
• Multiple Output Formats: XML, XML Index (for large sites), JSON, or plain text
• URL Filtering: Include or exclude URLs using regex patterns
• Configurable Depth: Control how deep the crawler goes
• Last Modified Detection: Automatically detects lastmod from headers/meta tags
• Webhook Integration: Get notified when sitemap generation completes

Use Cases

• SEO Audits: Generate sitemaps to submit to Google Search Console
• Site Migration: Create a complete URL inventory before redesigning
• Content Inventory: Get a full list of all pages and their metadata
• Image SEO: Generate image sitemaps for better Google Images visibility
• Monitoring: Track how your site structure changes over time

Input Parameters

Parameter	Type	Description
startUrl	string	The website URL to crawl (required unless demoMode)
maxPages	integer	Maximum pages to crawl (default: 1000)
maxDepth	integer	Maximum link depth from start URL (default: 10)
includeImages	boolean	Include images in sitemap (default: true)
includeVideos	boolean	Include videos in sitemap (default: false)
includeLastmod	boolean	Detect last modified dates (default: true)
respectRobotsTxt	boolean	Follow robots.txt rules (default: true)
urlPatterns	array	Only include URLs matching these regex patterns
excludePatterns	array	Exclude URLs matching these regex patterns
outputFormat	string	xml, xml-index, json, or txt (default: xml)
changefreq	string	Default change frequency (default: weekly)
priority	number	Default priority 0.0-1.0 (default: 0.5)
webhookUrl	string	URL to receive completion notification
demoMode	boolean	Return sample data for testing

Output

The actor outputs a single result object containing:

{
  "url": "https://example.com",
  "pagesFound": 245,
  "imagesFound": 892,
  "videosFound": 12,
  "crawlTime": 45230,
  "format": "xml",
  "sitemap": "<?xml version=\"1.0\"...>",
  "pages": [
    {
      "url": "https://example.com/",
      "lastmod": "2024-01-15",
      "changefreq": "weekly",
      "priority": 1.0,
      "images": ["https://example.com/logo.png"],
      "title": "Home Page",
      "depth": 0
    }
  ]
}

Example Usage

Basic Sitemap

{
  "startUrl": "https://example.com",
  "maxPages": 500
}

Image Sitemap for E-commerce

{
  "startUrl": "https://shop.example.com",
  "maxPages": 10000,
  "includeImages": true,
  "urlPatterns": ["/products/", "/categories/"],
  "excludePatterns": ["/cart", "/checkout", "/account"]
}

JSON Export for Processing

{
  "startUrl": "https://example.com",
  "outputFormat": "json",
  "maxDepth": 3
}

With Webhook Notification

{
  "startUrl": "https://example.com",
  "webhookUrl": "https://hooks.zapier.com/your-webhook-id"
}

Pricing

This actor uses the pay-per-event pricing model:

• Base cost: $0.10 per run
• Per page: $0.001 per page crawled

Example: Crawling 1,000 pages costs approximately $1.10

Integration

Zapier/Make/n8n

Use the webhook URL to trigger automations when sitemap generation completes. The webhook payload includes:

• Number of pages found
• Number of images/videos found
• Crawl time
• Output format

Google Search Console

Download the XML sitemap from the dataset and submit it to Google Search Console for indexing.

Limitations

• Maximum 100,000 URLs per sitemap (XML standard limit)
• For larger sites, use xml-index format to generate a sitemap index
• Respects robots.txt by default (can be disabled)
• External links are not followed (stays within the same domain)

Support

For issues or feature requests, contact support.

---

Keywords

sitemap generator, xml sitemap, seo sitemap, website crawler, sitemap automation, zapier integration, make integration, n8n integration