Sitemap Generator & Crawl-Based Site Audit

Crawl a website, discover every reachable page, and get a valid sitemap.xml plus a full indexability audit — for sites that never published a sitemap, or whose sitemap is out of date.

Point it at a homepage. It follows the links, respects robots.txt, works out which pages actually belong in a sitemap, and writes the file for you.

---

What it does

1. Builds a sitemap from a crawl

Give it a start URL. It walks the site breadth-first, tracks how many hops each page sits from the entry point, and writes generated-sitemap.xml to the run's key-value store — a real sitemap document with <loc>, <lastmod>, <changefreq> and <priority>.

• sitemapPriority derives from crawl depth: 1.0 at the root, decreasing outward
• changeFrequency follows the same signal (daily at the root down to yearly deep in)
• <lastmod> comes from each page's Last-Modified response header, and is simply omitted where the server does not send one rather than being invented
• URLs containing & are properly escaped, so the output is well-formed XML — the test suite parses every generated document to prove it

2. Decides what actually belongs in the sitemap

A sitemap listing redirects, 404s and duplicate pages is worse than no sitemap. Every page is assessed and non-indexable ones are excluded by default, with the reason recorded:

indexabilityIssues	Meaning
http_404, http_500, …	The page errors
redirect	The URL is a 3xx hop, not a destination
meta_noindex	<meta name="robots" content="noindex">
robots_disallowed	robots.txt disallows this path
non_canonical	A rel="canonical" points somewhere else — this is a duplicate

Canonical comparison is normalised, so a canonical differing only by a trailing slash or tracking parameter is correctly read as self-referential rather than flagged as a duplicate.

3. A site audit alongside the sitemap

Every crawled page becomes a row with title, meta description, status, canonical, robots directives, response time, size, word count, and inbound and outbound link counts — inbound counts are backfilled after the crawl finishes, so a page linked from somewhere visited later still gets the right number.

The summary reports pagesMissingTitle, pagesMissingMetaDescription, duplicateTitleCount, the status-code and depth spread, and how many URLs were skipped as off-site, disallowed or non-page assets.

4. Crawl scope you control

robots.txt is parsed properly — a group naming a specific user agent takes precedence over the wildcard group, which is easy to get backwards. Subdomains are excluded by default, and a lookalike domain (notexample.com against example.com) is never mistaken for a subdomain. Asset URLs are filtered by extension so images, PDFs and stylesheets never reach the sitemap. includeUrlPatterns and excludeUrlPatterns accept regular expressions, and a malformed pattern degrades to substring matching instead of killing the run.

Related Actor: this one generates a sitemap by crawling. If you already have a sitemap URL and want to parse, filter and audit its contents, use Sitemap URL Extractor instead.

---

Input

Field	Type	Default	Description
startUrls (required)	array	—	Where the crawl begins; the first URL sets the scope
maxPages	integer	200	Hard cap on pages visited
maxCrawlDepth	integer	3	Link hops to follow from a start URL
includeSubdomains	boolean	false	Follow links to subdomains
includeUrlPatterns	array	[]	Only crawl URLs matching these regexes
excludeUrlPatterns	array	[]	Skip URLs matching these regexes
respectRobotsTxt	boolean	true	Honour robots.txt disallow rules
generateSitemapFile	boolean	true	Write generated-sitemap.xml to the key-value store
sitemapIndexableOnly	boolean	true	Keep non-indexable pages out of the sitemap
concurrentRequests	integer	5	Pages fetched at once
requestDelayMs	integer	0	Pause after each page
maxAttempts	integer	2	Tries per page; transient failures only
requestTimeoutSec	integer	30	Per-request timeout
emitRunSummary	boolean	true	Append the site-audit summary record
proxyConfig	object	disabled	Optional Apify Proxy settings

Example input

{
  "startUrls": ["https://example.com"],
  "maxPages": 500,
  "maxCrawlDepth": 3,
  "includeSubdomains": false,
  "excludeUrlPatterns": ["/tag/", "\\?page="],
  "respectRobotsTxt": true,
  "generateSitemapFile": true,
  "sitemapIndexableOnly": true
}

---

Output

One page record per crawled URL, then an optional summary record. The sitemap itself is written to the key-value store as generated-sitemap.xml.

Page record

{
  "recordType": "page",
  "pageUrl": "https://example.com/about",
  "finalUrl": "https://example.com/about",
  "isRedirect": false,
  "statusCode": 200,
  "crawlDepth": 1,
  "pathDepth": 1,
  "pageTitle": "About us",
  "metaDescription": "Who we are and what we build.",
  "robotsDirectives": ["index", "follow"],
  "isIndexable": true,
  "indexabilityIssues": [],
  "isCanonical": true,
  "canonicalUrl": "https://example.com/about",
  "contentType": "text/html; charset=utf-8",
  "lastModified": "Mon, 01 Aug 2026 00:00:00 GMT",
  "responseTimeMs": 142.8,
  "pageSizeBytes": 48213,
  "wordCount": 612,
  "inboundLinkCount": 14,
  "outboundLinkCount": 37,
  "changeFrequency": "weekly",
  "sitemapPriority": 0.8,
  "crawlError": null,
  "crawledAt": "2026-08-11T12:00:00+00:00"
}

Summary record

{
  "recordType": "summary",
  "startUrls": ["https://example.com"],
  "urlsDiscovered": 842,
  "pagesCrawled": 200,
  "indexablePageCount": 171,
  "nonIndexablePageCount": 29,
  "indexabilityIssueBreakdown": { "non_canonical": 18, "redirect": 7, "http_404": 4 },
  "statusCodeBreakdown": { "200": 189, "301": 7, "404": 4 },
  "crawlDepthBreakdown": { "2": 96, "3": 71, "1": 32, "0": 1 },
  "redirectCount": 7,
  "nonCanonicalCount": 18,
  "pagesMissingTitle": 2,
  "pagesMissingMetaDescription": 41,
  "duplicateTitleCount": 6,
  "skippedOffSiteCount": 318,
  "skippedDisallowedCount": 44,
  "skippedNonPageCount": 129,
  "averageResponseTimeMs": 187.42,
  "generatedSitemapKey": "generated-sitemap.xml",
  "generatedSitemapUrlCount": 171,
  "finishedAt": "2026-08-11T12:04:22+00:00"
}

---

Common use cases

• Generate a missing sitemap — most small and custom-built sites have none; this produces a valid one from the site itself.
• Refresh a stale sitemap — compare generatedSitemapUrlCount against the published sitemap to find pages that were never listed.
• Technical SEO audit — indexabilityIssueBreakdown, duplicateTitleCount and pagesMissingMetaDescription surface the usual problems in one pass.
• Find broken internal links — filter pages by statusCode >= 400 and use inboundLinkCount to see how many pages point at them.
• Site migration checks — isRedirect and finalUrl confirm old URLs land where they should.
• Content inventory — a complete list of live pages with titles, descriptions and word counts.

---

Notes and limits

• The crawler reads server-rendered HTML. Pages whose links only appear after JavaScript runs will not be discovered, and a fully client-rendered site may yield just the start URL.
• Only pages reachable by following links from the start URLs are found. Orphan pages with no inbound links are, by definition, invisible to any crawler.
• maxPages is a hard stop. If it is reached with URLs still queued, the sitemap covers what was crawled — check urlsDiscovered against pagesCrawled to see whether you were truncated.
• <lastmod> reflects the Last-Modified header. Many servers, especially behind a CDN, do not send one, so the element is omitted for those pages rather than guessed.
• sitemapPriority and changeFrequency are conventional hints derived from crawl depth. Search engines largely ignore both; they are included because the sitemap format defines them.
• robots.txt support covers User-agent and Disallow grouping. Allow overrides, wildcards and Crawl-delay are not interpreted — use excludeUrlPatterns for finer control.
• Word counts come from visible body text with scripts, styles and markup removed. They are an indication of page substance, not an exact editorial count.
• Crawling consumes someone's bandwidth. Keep concurrentRequests modest and set requestDelayMs on small sites.

---

Local development

pip install -r requirements.txt
python -m tests.test_core      # 129 checks, including a live crawl and XML validation
apify run