Sitemap URL Extractor - XML Sitemap to CSV with lastmod

Sitemap URL Extractor - XML Sitemap to CSV, lastmod, hreflang

Get every URL of a website in seconds — straight from its sitemap.xml, sitemap index files and robots.txt. No crawling, no browser, no guessing: the Actor reads what the site itself publishes for search engines and returns a clean, deduplicated list with lastmod, changefreq, priority, hreflang alternates and image counts.

Typical run: a 20,000-page site in under a minute for a few cents.

What you can do with it

• SEO audits and content inventories — list all indexable pages, spot stale sections by lastmod, compare against your crawl.
• Site migrations — export the old URL list, build redirects, verify nothing is lost.
• Change monitoring — run it daily with modifiedAfter to get only pages updated since yesterday.
• Feeding other scrapers and AI agents — produce the exact list of pages to visit instead of discovering links; call it from Claude, Cursor or any MCP client through the Apify MCP server.
• Competitor research — count pages per section (/blog/, /product/) with includePattern.
• International SEO — check hreflang coverage from the sitemap's xhtml:link alternates.

How it works

1. You give it one or more start URLs: a homepage, a sitemap.xml, a sitemap index or a sitemap.txt.
2. For a homepage it reads /robots.txt (Sitemap: lines) and falls back to common paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, …).
3. It opens sitemap index files recursively, decompresses .gz, parses XML, text and RSS/Atom sitemaps, applies your filters and writes each URL to the dataset.

It never fetches the pages themselves, so it is fast, cheap and gentle to the target site.

Input

Field	Type	Default	Description
startUrls	array	—	Homepage, sitemap or sitemap index URLs.
maxUrls	integer	50000	Stop after this many unique URLs.
includePattern	regex	—	Keep only matching URLs, e.g. ^https://example.com/blog/.
excludePattern	regex	—	Drop matching URLs, e.g. \?page=|/tag/.
modifiedAfter	date	—	Keep only URLs with lastmod on or after YYYY-MM-DD (URLs without lastmod are kept).
discoverFromRobots	boolean	true	Discover sitemaps via robots.txt and common paths.
followSitemapIndexes	boolean	true	Open child sitemaps listed in index files.
includeMetadata	boolean	true	Add lastmod, changefreq, priority, alternates, images.
maxSitemaps	integer	500	Safety cap on sitemap files downloaded.
requestTimeoutSecs	integer	30	Per-request timeout.
proxyConfiguration	object	off	Apify Proxy, only if the site blocks datacenter requests.

Example:

{
  "startUrls": [{ "url": "https://example.com" }],
  "includePattern": "^https://example.com/blog/",
  "modifiedAfter": "2026-01-01",
  "maxUrls": 10000
}

Output

One dataset item per URL:

{
  "url": "https://example.com/blog/hello-world",
  "lastmod": "2026-09-01",
  "changefreq": "weekly",
  "priority": 0.8,
  "alternates": [{ "hreflang": "pt", "href": "https://example.com/pt/blog/hello-world" }],
  "images": 2,
  "sitemapUrl": "https://example.com/sitemap-posts.xml",
  "host": "example.com",
  "depth": 1
}

Download it as JSON, CSV, Excel or XML from the run, or read it through the API. A run summary (sitemaps fetched, URLs found, discovery method, errors) is stored in the OUTPUT record of the key-value store.

Use cases

Ready-made examples you can run without configuring anything:

• Export an XML sitemap to CSV — ready-made task, run it as is
• Audit lastmod and changefreq across a site — ready-made task, run it as is

Export an XML sitemap to CSV

Point the Actor at a sitemap or a domain and download the full URL list as CSV or JSON — ready for a spreadsheet, a crawl budget review or a migration checklist.

{"startUrls": [{"url": "https://example.com"}], "discoverFromRobots": true}

Audit lastmod and changefreq across a site

Every row carries lastmod, changefreq and priority exactly as the site publishes them, so stale sections and wrong priorities show up with one sort.

{"startUrls": [{"url": "https://example.com/sitemap.xml"}]}

Collect hreflang alternates for international SEO

Sitemaps that declare xhtml:link alternates come back with every language variant attached to its page, which is the fastest way to spot missing or asymmetric hreflang.

{"startUrls": [{"url": "https://example.com/sitemap_index.xml"}]}

Pricing

Pay per event: you are charged per extracted URL, nothing per page visited and nothing for failed sitemaps. Set Max total charge on the run to cap spending; the Actor stops cleanly when the cap is reached.

Use it from code and from AI agents

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("<ACTOR_ID>").call(run_input={"startUrls": [{"url": "https://example.com"}]})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item.get("lastmod"))

Through the Apify MCP server any agent can call this Actor as a tool: ask it to "list all pages of example.com modified this month" and it will run the Actor and read the dataset.

Limits and notes

• Only URLs listed in sitemaps are returned. If a site has no sitemap the Actor reports sitemap not found in OUTPUT.errors.
• Sitemap indexes are followed up to five levels deep; maxSitemaps protects you from runaway sites.
• Relative <loc> values are resolved against the sitemap URL.
• Duplicates are removed across all sitemaps of a run.

FAQ

Does it crawl the website? No. It reads only sitemap files, which is why it is fast and costs a fraction of a crawl.

The site has millions of URLs — will it work? Yes; raise maxUrls and maxSitemaps, and cap spending with Max total charge.

Can I get only new pages? Use modifiedAfter. Pages without lastmod are kept so you never miss content silently.

Is this legal? Sitemaps are public files intended for machines. The Actor sends a descriptive user agent and downloads only sitemap files.

Related Actors

• Bulk URL Status Checker
• Tech Stack Detector
• Hacker News Search API

Cost control

Pay per event only works if you can see the bill before you pay it:

• Set dryRun to count every result and print what a full run would cost - nothing is charged, and only a few sample rows are stored.
• You are charged only for rows actually delivered to the dataset; a run that finds nothing costs nothing beyond the start event.
• maxItems is respected exactly, and the run stops before the platform's Max total charge for the run.
• Every run writes a COST record to the key-value store: chargeable events, charged events, the amount in USD and whether a limit was hit.