Sitemap Generator - Crawl Website & Create XML Sitemap

Generate an XML sitemap (sitemap.xml) for any website by crawling internal pages from one or more start URLs.

This Actor is designed for:

• SEO audits (discover missing pages)
• Creating/refreshing sitemaps for search engines
• QA / monitoring of site URL coverage

What it does

• Crawls internal links (same hostname as the provided start URLs)
• Deduplicates URLs
• Stores sitemap.xml in the default key-value store
• If the site has more than 50,000 discovered URLs, it creates multiple sitemap-*.xml parts plus a sitemap-index.xml (and sitemap.xml will contain the index for compatibility)
• Writes a dataset item for each crawled page (included/excluded + reason)
• Writes a SUMMARY JSON report (counts, settings, sitemap URL count)

Input

• startUrls (required): Start URLs (request list)
• maxPages: Max pages to crawl (limits total requests)
• maxDepth: Max link depth from the start URLs
• ignoreUrlPatterns: Array of regex strings to exclude URLs
• includeQueryParams: Include ?query=params in sitemap URLs
• includeFragments: Include #fragments in sitemap URLs (usually disabled)
• includeLastModified: If enabled, uses the HTTP Last-Modified header for <lastmod> when available
• respectRobotsTxt: If enabled, skips URLs disallowed by robots.txt for User-agent: * (best-effort)
• robotsTxtTimeoutSecs: Timeout for downloading robots.txt
• changefreq, priority: Optional sitemap hints applied to all URLs

Output

Key-value store

• sitemap.xml (XML)
• sitemap-index.xml (XML, only for large sites)
• sitemap-1.xml, sitemap-2.xml, ... (XML parts, only for large sites)
• SUMMARY (JSON)

Dataset

Each item contains:

• url, normalizedUrl, statusCode, contentType
• depth, discoveredFrom
• includedInSitemap, exclusionReason
• lastModified, crawledAt

SEO keywords

sitemap generator, xml sitemap generator, website sitemap crawler, generate sitemap.xml, seo sitemap tool, internal link crawler

Quick start

Store page: https://apify.com/scrappy_garden/sitemap-generator

Paste this into Input and click Run:

{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Outputs (what you get)

• Dataset: Dataset items typically include fields like: url, statusCode, includedInSitemap, exclusionReason, depth, lastModified, crawledAt.
• Key-value store: SUMMARY, sitemap.xml

Tips (trust + predictable results)

• Start with 1–3 URLs to validate behavior, then scale up.
• If a target blocks requests, enable Proxy and/or slow down concurrency in Input.
• Use the SUMMARY / REPORT keys (when present) for automation pipelines and monitoring.

Related actors

• robots-txt-validator (https://apify.com/scrappy_garden/robots-txt-validator)
• broken-link-checker (https://apify.com/scrappy_garden/broken-link-checker)
• canonical-url-checker (https://apify.com/scrappy_garden/canonical-url-checker)
• web-page-change-monitor (https://apify.com/scrappy_garden/web-page-change-monitor)

Search keywords

sitemap generator, sitemap generator - crawl website & create xml sitemap, website audit, seo, sitemap