Sitemap URL Extractor

Returns every page URL listed in the sitemaps of one or more websites, with lastmod, changefreq and priority for each URL. Give it websites or sitemap URLs. It finds the sitemaps through robots.txt and common paths, and follows sitemap indexes, .gz files and plain-text sitemaps. No proxy, login or API key needed.

What you get

One dataset item per page URL:

Field	Type	Description
url	string	Page URL listed in the sitemap
lastmod	string or null	Last modification time from the sitemap (ISO 8601), or null if not given
changefreq	string or null	Change frequency hint (always, hourly, daily, weekly, monthly, yearly, never), or null
priority	number or null	Priority hint from 0.0 to 1.0, or null
sitemapUrl	string	The sitemap file that listed this URL
inputUrl	string	The input entry this URL came from

Each URL appears once per input, even if several sitemaps list it.

Input

{
    "urls": ["https://wordpress.org", "https://crawlee.dev/sitemap.xml"],
    "maxUrlsPerSite": 1000,
    "includeUrlPatterns": ["/blog/"],
    "excludeUrlPatterns": ["/tag/"],
    "modifiedAfter": "2026-01-01"
}

• urls (required): websites (example.com) or direct sitemap URLs. For a website, the Actor reads the Sitemap: lines in robots.txt. If there are none, it tries /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml.
• maxUrlsPerSite: maximum URLs per input (default 1,000; 0 means no limit).
• includeUrlPatterns / excludeUrlPatterns: keep or drop URLs that contain any of these texts (case-insensitive).
• modifiedAfter: keep only URLs with a lastmod on or after this date. URLs without a lastmod are dropped when this is set.

Sample output

[
    {
        "url": "https://crawlee.dev/blog",
        "lastmod": null,
        "changefreq": "weekly",
        "priority": 0.5,
        "sitemapUrl": "https://crawlee.dev/sitemap.xml",
        "inputUrl": "https://crawlee.dev/sitemap.xml"
    },
    {
        "url": "https://www.sitemaps.org/protocol.html",
        "lastmod": "2022-12-15T00:00:00.000Z",
        "changefreq": null,
        "priority": null,
        "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
        "inputUrl": "https://www.sitemaps.org/sitemap.xml"
    }
]

Use cases

• RAG and LLM pipelines: get a clean list of a site's pages to feed a crawler or embedding job.
• SEO audits: see what a site tells search engines it has, and when each page last changed.
• Change monitoring: run on a schedule with modifiedAfter to get only new or updated pages.
• Crawl planning: scope a crawl to /blog/ or /docs/ before you spend money crawling.

How it handles problems

The run never fails because one site is broken. URLs from the other sites are still returned, and the run's status message lists every input that produced no URLs, with the reason:

• no sitemap found: robots.txt lists no sitemap and none of the common paths exist.
• site unreachable: DNS failure, refused connection or timeout.
• site blocked the request (HTTP 403 …) or sitemap unreadable: … HTTP 403: the site refuses automated clients.
• invalid URL: the input isn't a website or sitemap URL.

Limits

• No browser disguise. Requests identify themselves honestly as openquarry-sitemap-url-extractor. A few sites block unknown bots, and for those you'll get a "blocked" message instead of URLs. We don't get around such blocks.
• Sitemaps only. The Actor doesn't crawl pages or check whether listed URLs still work. If a site has no sitemap, use a crawler instead.
• Big sites take longer. A sitemap index with tens of thousands of URLs takes a minute or more with no limit. The default limit of 1,000 per site keeps runs to seconds.
• At most 2,000 sitemap files are read per input.

Pricing

Pay per result: $0.25 per 1,000 URLs returned (67,000 URLs from developer.mozilla.org cost about $17). Apify adds a small start fee of $0.00005 per run. Sites with no sitemap cost you nothing beyond that start fee. Use maxUrlsPerSite or the run's maximum cost setting to cap spending. The Actor stops cleanly when your limit is reached.

Changelog

• 0.1 (2026-10): First release.