Sitemap URL Extractor: All Pages of Any Website

Enter a website and get the full list of pages it declares in its sitemaps, each with its last-modified date. No crawling, so even sites with hundreds of thousands of pages finish in seconds to minutes.

You pay only for page addresses returned, after your filters are applied. Websites without a sitemap cost nothing.

What it does

• Finds the sitemaps for you from the site's robots.txt, falling back to /sitemap.xml and /sitemap_index.xml. You can also give it a specific sitemap file.
• Follows sitemap indexes to any depth and reads gzipped sitemap files.
• Removes duplicates when the same page appears in several sitemaps.
• Filters by words in the address and by how recently the page changed.

What you get for each page

Field	What it is
url	The page address
lastModified	When the site says the page last changed, in standard date format
changeFrequency, priority	The sitemap's hints, where present
website, sitemap	The site and the sitemap file the entry came from

Input

{
  "websites": ["https://docs.apify.com"],
  "urlContains": ["/academy/"],
  "modifiedWithinDays": 30,
  "maxUrlsPerWebsite": 5000
}

Typical uses

• Feed a scraper or AI pipeline: get the exact list of pages to fetch, instead of crawling to discover them.
• Content monitoring: run daily with modifiedWithinDays set to 1 to see what a site published or changed.
• SEO audits and migrations: export every indexable address of a site, or compare two sites' structures.
• Competitor research: count a competitor's product, category or blog pages by filtering on the address.

Pricing

$0.30 per 1,000 page addresses ($0.0003 each). One charge per page address returned.

A site with 20,000 pages costs $6. A filtered run that returns 500 addresses costs $0.15.

Each run also has a start charge of $0.00005. There is no separate charge for computing time. Apify's paid plans get 10% to 30% off these prices; the Pricing tab shows yours. Apify's free plan includes $5 of credit a month, which covers about 16,000 page addresses.

Limits

• It lists what the website declares. Pages a site leaves out of its sitemaps are not found, and sites with no sitemap return nothing; those are listed in the run's NO_SITEMAP record.
• Last-modified dates are the site's own claims and are missing on some sites.

Questions

Can I run it on a schedule?

Yes. Save your input as a task and add an Apify schedule, for example every morning.

How do I get the results into Excel or Google Sheets?

Every run's results can be downloaded as Excel, CSV or JSON from the run's Output tab. Apify's integrations with Make, Zapier, n8n and Google Sheets can pass them on automatically.

Can I call it from my own code or from an AI agent?

Yes. The API tab has ready-made code for Python, JavaScript and cURL, and the tool can be connected to AI agents through Apify's MCP server.

Something looks wrong or is missing

Open a report under the Issues tab and include the input you used.