Sitemap URL Extractor

Get every URL from a website's sitemap.xml at $1 per 1,000 URLs. Paste a domain, a sitemap URL or a robots.txt URL and get one dataset row per URL. Filter by last modified date and pay only for the URLs you keep.

The run finishes even when some inputs have no sitemap or refuse the request. Those inputs are explained in the key-value record SITEMAPS_SUMMARY, which is not charged.

Who it is for

• SEO teams who need the full URL list of a site before an audit or a migration.
• Scraping pipelines that feed a crawler only the pages changed since the last run, using the lastmod filter.
• Competitor monitoring: see which pages a site published or updated this week.

What you get

• Discover sitemaps from robots.txt, then from common paths such as /sitemap.xml and /wp-sitemap.xml.
• Read sitemap indexes, gzipped files (including gzip that is not named .gz), and plain-text URL lists.
• Keep lastmod, change frequency, and priority. Images and hreflang links are optional.
• Stop at maxUrls.
• Pass lastmodSince to keep only URLs modified on or after that day. Older URLs are not returned and are not charged. URLs with no lastmod stay in the result unless you also enable "Drop URLs without lastmod".

Sample output

A real row from a run on oracle.com:

{
  "url": "https://www.oracle.com/ai-data-platform/",
  "lastmod": "2026-09-04",
  "changefreq": "monthly",
  "priority": 0.9,
  "sitemapUrl": "https://www.oracle.com/sitemap-core.xml",
  "sitemapDepth": 1,
  "discoveredVia": "robots-txt",
  "sourceDomain": "oracle.com",
  "inputRef": "https://www.oracle.com",
  "scrapedAt": "2026-09-23T20:04:37Z"
}

SITEMAPS_SUMMARY lists, per input, whether the result was ok, empty, no-sitemap, blocked, not-a-sitemap, or error.

How to use

1. Paste a domain, a sitemap.xml URL, or a robots.txt URL.
2. Leave Maximum URLs at 20 for a trial. The default cap is 1,000.
3. Run and export the dataset. Inputs with no sitemap are listed in SITEMAPS_SUMMARY and are not charged.

{
  "startUrls": [{ "url": "https://apify.com" }],
  "maxUrls": 20
}

Input

Field	Default	What it does
startUrls	https://apify.com	Domains, sitemap files, or robots.txt files.
maxUrls	1000 (prefill 20)	Stop after this many unique URLs. You pay per URL returned.
discoverViaRobotsTxt	true	Read Sitemap lines from robots.txt first.
tryCommonPaths	true	Also try /sitemap.xml and /wp-sitemap.xml.
includeUrlPatterns	empty	Keep URLs matching at least one regex.
excludeUrlPatterns	empty	Drop URLs matching any regex.
includeImages	false	Add image URLs from the image sitemap extension.
includeAlternates	false	Add hreflang alternates.
lastmodSince	empty	Drop URLs modified before this day. They are not charged.
dropUrlsWithoutLastmod	false	With a date filter, also drop URLs that have no lastmod.
maxSitemapDepth	5	How many sitemap-index levels to follow.

Pricing

You pay for URLs written to the dataset. Failed inputs and SITEMAPS_SUMMARY are free. The platform start event stays at $0.00005.

Plan	Per URL	Per 1,000 URLs
Free	$0.001	$1.00
Bronze	$0.0009	$0.90
Silver	$0.0008	$0.80
Gold, Platinum, Diamond	$0.0007	$0.70

Use with AI agents (MCP)

Call the Actor martinvarelaa/sitemap-url-extractor. Pass startUrls and, when you only need recent pages, lastmodSince. Read the dataset for URLs and SITEMAPS_SUMMARY for inputs that produced nothing.

Use via API

curl -X POST "https://api.apify.com/v2/acts/martinvarelaa~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com"}],"maxUrls":20}'

FAQ

What happens if a site has no sitemap?

The run still finishes. The summary says no-sitemap for that input and nothing is charged.

Can I get only the pages that changed recently?

Yes. Set lastmodSince to a date. URLs older than that are omitted and not charged. URLs with no lastmod stay unless you also enable "Drop URLs without lastmod".

Does it handle sitemap indexes and gzip?

Yes. It follows sitemap indexes up to maxSitemapDepth levels and reads gzipped files, even when the file name does not end in .gz.

Is the data legal to use?

This Actor extracts publicly available, non-personal data. You are responsible for using the data in compliance with applicable laws and with the target website's terms where applicable.

Other actors

• SEO Page Audit scores pages and checks sitemap URLs and AI-crawler rules.
• Wayback Machine Scraper lists Wayback Machine snapshots and archived pages.
• Clutch.co Scraper builds agency lead lists with rates, team size and published contacts.

Changelog

• 0.1 (2026-09-23): sitemap discovery from robots.txt and common paths, indexes and gzip, and the lastmodSince filter.