Sitemap URL Intelligence

Turn any website's sitemap.xml into a clean, structured URL inventory — in seconds, without setting up a crawler. Point the Actor at a homepage or a sitemap URL and get back one tidy row per page, enriched with SEO and content metrics, ready to export as JSON, CSV or Excel.

Perfect for SEO audits, site migrations, competitor content research, and AI/RAG ingestion, where you need a complete, deduplicated list of a site's URLs fast.

Why use it

• No crawling required. Sitemaps are the site's own map — reading them is faster, cheaper and gentler than crawling every page.
• Handles real-world sitemaps. Follows sitemapindex files recursively, deduplicates URLs across overlapping sitemaps, and survives broken or missing sitemaps without crashing the run.
• Instant enrichment. Every URL is classified (homepage / html / image / document / file) with path depth, extension and query-param counts — the raw material for an SEO or content audit.
• Export anywhere. Results land in an Apify dataset you can download as CSV, JSON or Excel, or pull via the API into your own pipeline.

What it does

1. Accepts website homepages or direct sitemap URLs. For a homepage it automatically tries /sitemap.xml.
2. Detects sitemap index files and recurses into every child sitemap.
3. Extracts each <url> entry with its lastmod, changefreq and priority.
4. Optionally enriches each URL with derived metrics (see below).
5. Deduplicates URLs across all sitemaps and writes one row per page.
6. Records unreachable or malformed sitemaps as error rows so nothing is lost.

Use cases

Buyer	How they use it
SEO consultant	Export the full URL set to check indexation coverage, spot orphan sections, and compare lastmod freshness.
Migration engineer	Snapshot every live URL before a replatform, then diff old vs. new to build redirect maps.
Content marketer	Pull a competitor's blog/product URLs to size their content library and find gaps.
AI / RAG builder	Get a deduplicated URL list to seed a document ingestion or embedding pipeline.
Agency / analyst	Deliver a client-ready URL inventory spreadsheet without standing up a scraper.

Input

Field	Type	Default	Description
startUrls	array	—	Homepages or sitemap XML URLs (required).
maxSitemaps	integer	50	Cap on sitemap files fetched, incl. nested ones.
maxUrls	integer	10000	Stop after this many unique URLs.
includeUrlMetrics	boolean	true	Add SEO/content columns to each row.
requestTimeoutSecs	integer	30	Per-request timeout (2 retries on failure).
proxyConfiguration	object	disabled	Optional Apify Proxy for blocked targets.

Example input

{
  "startUrls": [{ "url": "https://www.apify.com/sitemap.xml" }],
  "maxUrls": 1000,
  "includeUrlMetrics": true
}

You can also pass a bare homepage and let the Actor find the sitemap:

{ "startUrls": [{ "url": "https://example.com" }] }

Output

One dataset item per URL. With includeUrlMetrics enabled a row looks like:

{
  "url": "https://example.com/blog/how-to-audit-seo",
  "sourceSitemap": "https://example.com/sitemap.xml",
  "lastmod": "2026-05-12",
  "changefreq": "weekly",
  "priority": 0.8,
  "hostname": "example.com",
  "path": "/blog/how-to-audit-seo",
  "pathDepth": 2,
  "extension": null,
  "urlType": "html",
  "queryParamCount": 0,
  "scrapedAt": "2026-07-04T10:00:00.000Z"
}

Output fields

Field	Description
url	The page URL from the sitemap (relative locs resolved to absolute).
sourceSitemap	The sitemap file this URL came from.
lastmod / changefreq / priority	Standard sitemap hints (null if absent).
hostname / path	Parsed host and path.
pathDepth	Number of path segments (a rough section-depth signal).
extension	File extension, or null for extension-less pages.
urlType	homepage, html, image, document, file, or invalid.
queryParamCount	Number of query-string parameters.
scrapedAt	ISO timestamp of when the row was produced.

Rows for sitemaps that could not be fetched or parsed contain sourceSitemap, error and scrapedAt instead — so failures are visible, not silent.

A run summary is also written to the default key-value store under the key SUMMARY:

{
  "sitemapsFetched": 4,
  "sitemapErrors": 0,
  "urlsFound": 1287,
  "urlTypes": { "html": 1180, "image": 90, "document": 17 },
  "generatedAt": "2026-07-04T10:00:05.000Z"
}

Limitations

• Reads what the sitemap declares — pages missing from the sitemap won't appear. This is not a crawler.
• Gzipped sitemaps (.xml.gz) are fetched but not decompressed; point the Actor at the plain .xml variant when available.
• lastmod, changefreq and priority are self-reported by the site and may be stale or absent.
• Very large sites are bounded by maxUrls / maxSitemaps; raise them for a full export.

Pricing positioning

Runs are cheap and fast because there is no page rendering or deep crawling — just a handful of XML fetches. A typical mid-size site (a few thousand URLs) completes in well under a minute on the smallest memory setting, making this one of the lowest-cost ways to get a complete URL inventory on the Apify platform.

Tips

• Feed the output straight into the Email & Social Lead Finder Actor to turn a URL inventory into contact data.
• Schedule the Actor and diff successive runs to monitor when a site publishes or removes pages.

---

Keywords: sitemap parser, sitemap.xml extractor, URL inventory, SEO audit, site migration, XML sitemap crawler, sitemap index, URL list export, content research, RAG ingestion, website URL scraper, SEO data.