Sitemap URL Extractor: Every Page of a Website, with Lastmod

Give it a website and get back every URL its sitemaps list, with when each page last changed. It finds the sitemaps the way search engines do: the Sitemap lines in robots.txt, or /sitemap.xml and /sitemap_index.xml when there are none. It follows sitemap indexes down to the last file and reads gzipped and plain-text sitemaps. Filter by URL pattern or by date, or turn on the link crawl for a site with no sitemap. $0.0003 per URL, and failed sitemaps are free. It works for people and for AI agents through Apify's MCP server.

What it's for

• Get the list of pages to feed a scraper, a crawler or a RAG index, without crawling the whole site.
• Watch a competitor or a news site: list only the pages changed in the last day or week.
• Check your own site: which pages your sitemaps list, which have no lastmod, which sitemap files are broken.
• Give an AI agent a site's page list so it can pick what to read.

Input

Field	What to put
Websites or sitemaps	One per line. Required. A website like https://example.com is read from the sitemaps its robots.txt lists. A sitemap address like https://example.com/sitemap.xml is read directly, with any sitemaps it points to.
Max URLs	Stop after this many URLs. 100 if you leave it out, in the form, through the API or from an AI agent. Raise it for a big site.
Only URLs matching	Optional patterns, like https://example.com/blog/**. * matches within one part of the path, ** across parts. A pattern matches the whole URL, so write https:// and the host the way the sitemap does (with or without www), or start with ** to match any host, like **/blog/**.
Skip URLs matching	Optional patterns to leave out, like https://example.com/tag/**.
Only changed since	Optional. A date like 2026-10-01, or a span like 24 hours, 7 days, 2 weeks or 1 month. Only URLs with a lastmod that recent are kept.
Crawl links too	Off by default. On: after the sitemaps, it follows links on the site's pages to find pages the sitemaps miss, or to list a site that has no sitemap.
Max pages to crawl	With the crawl on: the most pages fetched, across all sites in the run. Default 100. Only fetched pages are listed, so this is also the most URLs the crawl adds. Raise it for a site with no sitemap.
Proxy	Off by default. Turn on Apify Proxy for a site that blocks cloud servers, or give your own proxy URLs. Residential proxies aren't available in this tool.

Example:

{
  "startUrls": ["https://techcrunch.com/news-sitemap.xml"],
  "newerThan": "24 hours"
}

Output

One row per URL. This one came from a test run on October 16, 2026, on TechCrunch's news sitemap:

{
  "type": "url",
  "url": "https://techcrunch.com/2026/10/15/tesla-reveals-spacex-powered-roadster-that-flies-and-falls-with-style/",
  "lastmod": "2026-10-16T01:17:43.000Z",
  "changefreq": null,
  "priority": null,
  "source": "sitemap",
  "sitemapUrl": "https://techcrunch.com/news-sitemap.xml",
  "listedAt": "2026-10-16T06:03:01.718Z"
}

• lastmod, changefreq and priority are what the sitemap says, or null when it doesn't say. Dates are ISO 8601 in UTC. In a news sitemap, lastmod is the article's publication date.
• source is sitemap, or crawl for a page found by following links.
• sitemapUrl is the sitemap file that listed the page, handy for finding a broken one.
• A sitemap that failed to load has a row with type: "sitemap" and an error. A website with no sitemap at all has a row with type: "site". Error rows are free.

The run's SUMMARY record says how many URLs were listed, how many sitemaps were read and failed, how many URLs the filters left out, and why the run stopped.

Price

• $0.0003 per URL listed, so $0.30 per 1,000 and $3 per 10,000.
• Free: duplicate URLs, URLs your filters leave out, and sitemaps that fail.
• You can set a maximum charge for a run. It stops when it's reached and keeps what it already listed.

Good to know

• Big sites. Some sites list millions of URLs. Max URLs is there so a run never costs more than you expect; set it to what you need.
• Only changed since needs a lastmod in the sitemap. Sites that don't give one have all their URLs left out when it's set; leave it empty for those. When a sitemap index dates its sitemaps, the ones older than your date aren't read at all, which keeps the run short. To watch a news site, give its news sitemap directly (often listed in robots.txt, like https://techcrunch.com/news-sitemap.xml): it holds only the last two days. On a big site whose index dates every sitemap today, the date filter still has to read them all, which can take minutes; use the news sitemap, or give the run a longer timeout.
• The link crawl reads robots.txt and skips what it disallows, stays on the same site, and skips links to files like images and PDFs. Crawled pages have no lastmod.
• Deep indexes. Sitemap indexes are followed 5 levels deep, which covers every site we've seen.
• Polite. It paces requests to the same site and reads at most 4 files at a time.

Use it from an AI agent

The Actor works through Apify's MCP server like any Store Actor. An agent can find it, read its input and call it. The MCP call waits 30 seconds by default (45 at most), and a site with a few hundred URLs usually finishes in that time. A bigger run takes longer, so the call returns the run's status and dataset ID, and the agent reads the rows with get-dataset-items.

Questions or problems

Open an issue on this Actor's Issues tab. I read them.