SITEMAP INTELLIGENCE — every URL a site publishes, sorted

Turn a website's XML sitemaps into a structured URL inventory with content classification, last-modified metadata, and per-site totals. Sitemap Intelligence discovers sitemap files from robots.txt and conventional locations, follows sitemap indexes, reads compressed .xml.gz files, and returns a sortable Apify dataset.

Give it a homepage such as https://example.com. It does not crawl every page: it reads the site's own sitemap declarations, so large catalogs can be mapped quickly with lightweight HTTP requests.

What Sitemap Intelligence does

• Discovers sitemap files from robots.txt, /sitemap.xml, and related conventional paths.
• Follows sitemap indexes into child sitemap files.
• Supports regular XML and gzipped XML sitemaps.
• Classifies each URL as product, category, blog, page, or other.
• Preserves the sitemap's lastmod value and the source sitemap URL.
• Adds a free summary row for each site with totals by content type.
• Runs with limited permissions and respects the run's maximum charge.

Classification trusts sitemap context first. For example, a URL inside sitemap_products_1.xml is treated as a product even when its path does not contain /products/. URL-path patterns are only the fallback, and uncertain records remain other instead of being forced into a misleading category.

How to use

1. Add one or more site or store URLs in Sites. A homepage URL is enough.
2. Set Maximum URLs to cap the number of paid URL records.
3. Optionally choose content types such as Product or Category. Filtered-out URLs are not charged.
4. Adjust Sitemaps to read per site only when a large site splits its catalog across many files.
5. Click Start.
6. Open the Output tab to switch between the URL table and the per-site summary. Export the dataset as JSON, CSV, Excel, XML, RSS, or HTML.

Example input:

{
  "startUrls": [{ "url": "https://www.gymshark.com" }],
  "maxUrls": 5000,
  "includeTypes": ["product"],
  "maxSitemapsPerSite": 15
}

Input

Field	What it does
startUrls	Site or store URLs. The homepage is enough.
maxUrls	Hard cap across the entire run.
includeTypes	Optional filter for products, categories, blog posts, pages, or other URLs.
maxSitemapsPerSite	Maximum sitemap files to read for each site.
crawlerIdentity	Optional custom User-Agent for transparent, scheduled crawling.

The cheapest way to target one kind of content is to set includeTypes. Filtering happens while the sitemap is read, so the run spends its result budget only on records you requested.

Output

The default dataset contains two row types.

A URL row:

{
  "rowType": "url",
  "url": "https://www.gymshark.com/products/legacy-fitted-t-shirt",
  "type": "product",
  "lastmod": "2026-07-18",
  "domain": "www.gymshark.com",
  "fromSitemap": "https://www.gymshark.com/sitemap_products_1.xml"
}

A free per-site summary row:

{
  "rowType": "summary",
  "domain": "www.gymshark.com",
  "total": 8964,
  "product": 8402,
  "category": 221,
  "blog": 338,
  "page": 3,
  "other": 0,
  "sitemapsRead": 12,
  "sitemapsSkipped": 0,
  "source": "robots-sitemap"
}

The Output tab provides separate URLs and Per-site summary views. The same dataset is available through the Apify API and integrations, and can be scheduled for recurring sitemap inventories.

Pricing

Sitemap Intelligence uses transparent pay-per-event pricing with platform usage included:

Event	Price
Actor start at the default memory	$0.01
Each classified and delivered URL	$0.0015
Per-site summary row	Free

Typical costs at the default memory are about $0.16 for 100 URLs, $1.51 for 1,000 URLs, and $15.01 for 10,000 URLs. The exact total is controlled by your maximum charge and maxUrls. URLs that are filtered out, cannot be delivered, or are never produced are not charged.

Reliability and responsible use

Requests to one host are spaced out, response bodies are capped, timeouts are bounded, and an HTTP 429 receives one backed-off retry. Different sites can be processed concurrently, but a run does not fan out aggressively against one host.

This Actor reports what a site publishes in its sitemap; it is not a live-page availability audit. A listed URL may later return an error, and an unlisted page may still exist. If a site exposes no readable sitemap, the log says so instead of presenting an empty dataset as proof that the site has no pages.

Use the Actor only on public sitemap data and follow the target site's terms, robots policy, and applicable laws.

Automation and integrations

Use Apify schedules to take weekly or monthly URL inventories. Connect run results to webhooks and integrations, or retrieve them with the Apify API. Common workflows include SEO migration planning, catalog-size monitoring, content-type analysis, stale-content reviews, and feeding known URLs into downstream crawlers.

FAQ

Does it crawl every web page?

No. It reads sitemap files and does not download every listed page. This keeps runs fast and lightweight.

Why is a URL classified as other?

The sitemap context and URL shape did not provide enough evidence for a reliable category. Returning other is safer than guessing.

Are summary rows charged?

No. The per-site summary counts URL rows already delivered and is written without the paid url-extracted event.

Can I process several sites in one run?

Yes. Add multiple entries to startUrls; maxUrls remains the hard cap across the whole run.

Support

If a sitemap format is not handled correctly or you have a feature request, open an issue in the Actor's Issues tab and include the public sitemap URL plus the relevant run ID.