Sitemap URL Extractor

Get every URL of a website in seconds — straight from its sitemap.xml, sitemap index files and robots.txt. No crawling, no browser, no guessing: the Actor reads what the site itself publishes for search engines and returns a clean, deduplicated list with lastmod, changefreq, priority, hreflang alternates and image counts.

Typical run: a 20,000-page site in under a minute for a few cents.

What you can do with it

• SEO audits and content inventories — list all indexable pages, spot stale sections by lastmod, compare against your crawl.
• Site migrations — export the old URL list, build redirects, verify nothing is lost.
• Change monitoring — run it daily with modifiedAfter to get only pages updated since yesterday.
• Feeding other scrapers and AI agents — produce the exact list of pages to visit instead of discovering links; call it from Claude, Cursor or any MCP client through the Apify MCP server.
• Competitor research — count pages per section (/blog/, /product/) with includePattern.
• International SEO — check hreflang coverage from the sitemap's xhtml:link alternates.

How it works

1. You give it one or more start URLs: a homepage, a sitemap.xml, a sitemap index or a sitemap.txt.
2. For a homepage it reads /robots.txt (Sitemap: lines) and falls back to common paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, …).
3. It opens sitemap index files recursively, decompresses .gz, parses XML, text and RSS/Atom sitemaps, applies your filters and writes each URL to the dataset.

It never fetches the pages themselves, so it is fast, cheap and gentle to the target site.

Input

Field	Type	Default	Description
startUrls	array	—	Homepage, sitemap or sitemap index URLs.
maxUrls	integer	50000	Stop after this many unique URLs.
includePattern	regex	—	Keep only matching URLs, e.g. ^https://example.com/blog/.
excludePattern	regex	—	Drop matching URLs, e.g. \?page=|/tag/.
modifiedAfter	date	—	Keep only URLs with lastmod on or after YYYY-MM-DD (URLs without lastmod are kept).
discoverFromRobots	boolean	true	Discover sitemaps via robots.txt and common paths.
followSitemapIndexes	boolean	true	Open child sitemaps listed in index files.
includeMetadata	boolean	true	Add lastmod, changefreq, priority, alternates, images.
maxSitemaps	integer	500	Safety cap on sitemap files downloaded.
requestTimeoutSecs	integer	30	Per-request timeout.
proxyConfiguration	object	off	Apify Proxy, only if the site blocks datacenter requests.

Example:

{
  "startUrls": [{ "url": "https://example.com" }],
  "includePattern": "^https://example.com/blog/",
  "modifiedAfter": "2026-01-01",
  "maxUrls": 10000
}

Output

One dataset item per URL:

{
  "url": "https://example.com/blog/hello-world",
  "lastmod": "2026-09-01",
  "changefreq": "weekly",
  "priority": 0.8,
  "alternates": [{ "hreflang": "pt", "href": "https://example.com/pt/blog/hello-world" }],
  "images": 2,
  "sitemapUrl": "https://example.com/sitemap-posts.xml",
  "host": "example.com",
  "depth": 1
}

Download it as JSON, CSV, Excel or XML from the run, or read it through the API. A run summary (sitemaps fetched, URLs found, discovery method, errors) is stored in the OUTPUT record of the key-value store.

Pricing

Pay per event: you are charged per extracted URL, nothing per page visited and nothing for failed sitemaps. Set Max total charge on the run to cap spending; the Actor stops cleanly when the cap is reached.

Use it from code and from AI agents

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("<ACTOR_ID>").call(run_input={"startUrls": [{"url": "https://example.com"}]})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item.get("lastmod"))

Through the Apify MCP server any agent can call this Actor as a tool: ask it to "list all pages of example.com modified this month" and it will run the Actor and read the dataset.

Limits and notes

• Only URLs listed in sitemaps are returned. If a site has no sitemap the Actor reports sitemap not found in OUTPUT.errors.
• Sitemap indexes are followed up to five levels deep; maxSitemaps protects you from runaway sites.
• Relative <loc> values are resolved against the sitemap URL.
• Duplicates are removed across all sitemaps of a run.

FAQ

Does it crawl the website? No. It reads only sitemap files, which is why it is fast and costs a fraction of a crawl.

The site has millions of URLs — will it work? Yes; raise maxUrls and maxSitemaps, and cap spending with Max total charge.

Can I get only new pages? Use modifiedAfter. Pages without lastmod are kept so you never miss content silently.

Is this legal? Sitemaps are public files intended for machines. The Actor sends a descriptive user agent and downloads only sitemap files.