Sitemap URL Extractor (all pages of a website, with lastmod)

Get every URL a website publishes in its XML sitemaps: finds the sitemaps from robots.txt or common paths, follows sitemap indexes and gzipped files, and returns each URL with lastmod, changefreq, priority, image/video counts and the sitemap it came from. Filter by path keyword or modification date. The fastest way to list a site's pages or products for SEO audits, migrations, content inventories and as input for other scrapers.

What you can use it for

• SEO audits and migrations: the complete list of a site's pages with last-modified dates
• Content inventories and competitor research: how many products, posts or docs a site has and what changed since a date
• Input for other scrapers: feed thousands of product or article URLs straight into a crawler
• Monitoring: run weekly and diff the URL lists to catch new or removed pages

Why this one

• Finds the sitemaps itself: robots.txt entries, sitemap.xml, sitemap_index.xml, wp-sitemap.xml and other common paths; follows nested indexes and .gz files
• Keeps lastmod, changefreq, priority, image/video counts and the source sitemap for every URL
• Path keyword and modified-since filters, so a 2-million-URL site can be narrowed to this month's product pages
• Pay only for results: $0.30 per 1,000 URLs. No subscription.

Input

• Websites or sitemap URLs (websites): A domain (sitemaps are found via robots.txt and common paths) or a direct sitemap URL.
• URL must contain (any) (includeKeywords): e.g. /blog/, /product/
• URL must not contain (any) (excludeKeywords)
• Modified since (YYYY-MM-DD) (modifiedSince): Keep only URLs whose lastmod is on or after this date (URLs without lastmod are kept).
• Max URLs per site (maxUrlsPerSite) Default: 100000.
• Max URLs total (maxResults) Default: 100000.

Output

One row per url in the dataset (JSON, CSV, Excel, or through the API). Sample:

[
 {
  "site": "apify.com",
  "url": "https://apify.com/",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "images": null,
  "videos": null,
  "news": null,
  "alternates": null,
  "path": "/",
  "sitemap": "https://apify.com/sitemap/pages.xml"
 },
 {
  "site": "apify.com",
  "url": "https://apify.com/about",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "images": null,
  "videos": null,
  "news": null,
  "alternates": null,
  "path": "/about",
  "sitemap": "https://apify.com/sitemap/pages.xml"
 }
]

Tips

• Give a direct sitemap URL to skip discovery (e.g. https://example.com/sitemap-products.xml).
• Sites with hundreds of sitemap files can take a few minutes; set maxUrlsPerSite to stop early.

Integrations and API

Run it on a schedule, trigger it from the API or a webhook, and send results to Google Sheets, Zapier, Make, Slack, a database or your own code with the platform's standard integrations.

Data and fair use

This tool reads only public sources (official APIs, public feeds or pages the publisher makes available to everyone) and respects robots.txt and rate limits. You are responsible for using the data in line with the source's terms and applicable law.

How much it costs

You pay per result (pay-per-event), nothing per month and nothing when a run returns nothing. The price is $0.30 per 1,000 URLs on the Free plan, with automatic discounts on paid plans (Bronze −5%, Silver −10%, Gold −15%, Platinum −20%).

Worked examples:

• 100 URLs → $0.03
• 1,000 URLs → $0.30
• 10,000 URLs → $3.00
• Plus $0.001 per run start.

Platform usage (compute, storage) is included in the price: the actor runs at a small memory setting by default, so a typical run of 1,000 URLs finishes in well under a minute.

FAQ

What if a site has no sitemap?

The actor checks robots.txt and the usual sitemap paths. If none exists (or it is empty or broken), it falls back to a polite link crawl from the homepage: up to crawlMaxPages pages, same site only, robots.txt respected. Those rows are marked source: "crawl" and carry the HTTP status. Turn crawlIfNoSitemap off to get sitemap URLs only.

Why are some fields empty?

A field is null when the source does not publish it for that row. The output keeps every column so your spreadsheet or integration always sees the same shape.

Can I run it on a schedule?

Yes. Use Apify Schedules (daily, hourly, cron) and connect the run to Make, Zapier, n8n, Google Sheets, a webhook or the API. Each run writes a dataset you can download as JSON, CSV or Excel.

Is there a free trial?

The Apify Free plan includes $5 of monthly platform credit, enough to try this actor on a few hundred results at no cost. There is no subscription for the actor itself.

Can an AI agent use this?

Yes. The actor is available through Apify's MCP server and the API, so LLM agents (Claude, ChatGPT, LangChain, CrewAI) can call it as a tool with the same input.

What happens if something fails?

Runs log every skipped item with a reason. If the source is down, the actor stops cleanly and you are only charged for the results it returned.

Changelog

• 2026-10-21 — No sitemap? No problem: when a site has no usable sitemap, the actor now finds its pages by following links from the homepage (same site only, robots.txt and nofollow respected, crawlIfNoSitemap, crawlMaxPages). Those rows carry source: "crawl" and the HTTP status. URLs listed in several sitemaps are now returned once.
• 2026-10-18 — Lower price with Bronze/Silver/Gold discounts; smaller default memory (cheaper runs); clean table view of results; FAQ and cost examples; default limits lowered to 10,000 per site so a first run never writes 100,000 rows by surprise.

Support

Found a problem or need a field added? Open an issue on the Issues tab; it is read and answered.