Sitemap Extractor — All URLs, lastmod, Images & hreflang

Get every URL a website lists in its XML sitemaps. Give a website or a sitemap link: sitemaps are found through robots.txt and the usual locations, sitemap indexes, .xml.gz files and text sitemaps are followed, and each URL comes back with its lastmod, changefreq, priority, images, hreflang alternates and news data. Filter by date ("changed since") or by pattern, and optionally check the HTTP status of every URL to find broken links and redirects.

Use it to feed a crawler or Bulk HTTP Requests, to audit a site's SEO coverage, to watch a competitor's new pages, or to get a clean URL list for RAG ingestion.

$0.25 per 1,000 URLs. $1 per 1,000 status checks (optional). Sites without a readable sitemap are free. No browser and no proxy needed, so runs are fast and cheap.

Quick start

Press Start with the example input:

{
  "startUrls": ["https://docs.apify.com/sitemap.xml"],
  "maxUrlsPerSite": 200
}

One row per URL (trimmed):

{
  "url": "https://docs.apify.com/api",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "site": "https://docs.apify.com"
}

With Check HTTP status on, each row also gets status (for example 200 or 404) and finalUrl when the URL redirects.

What it does

• Finds the sitemaps: Sitemap: lines in robots.txt; if there are none, /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz and /sitemap-index.xml. Or give the sitemap link yourself.
• Follows everything: sitemap indexes (nested up to 6 levels, loops ignored), gzip files, text sitemaps, child sitemaps on another host (for example a CDN).
• Extras: images (image sitemap), alternates (hreflang links), newsTitle and newsDate (news sitemap), videoTitles.
• Filters: Changed since keeps URLs with a recent lastmod and skips whole child sitemaps whose lastmod is older, which saves time on big sites. Regular-expression include and exclude patterns.
• Status check (optional): HEAD request, and any error confirmed with a GET that does not download the page (many servers answer HEAD wrongly). Redirects followed, final URL reported. A page that does not answer within 15 seconds is reported with checkError.
• Summary per site in the key-value store record SUMMARY: sitemaps read and their errors, URLs found, filtered out and duplicates.
• Duplicates removed: a URL listed in several sitemaps is returned (and charged) once per site.

Polite and safe

• robots.txt is read once per site and honoured for every sitemap and every status check. A sitemap or page that robots.txt disallows is not requested.
• One request at a time per host, with a short pause between requests.
• XML is parsed safely (no entity expansion or external entities); gzip is unpacked with a size cap.
• Private and internal addresses (localhost, 10.x, 192.168.x, cloud metadata) are blocked, including every redirect.

Input

Field	Default	What it does
startUrls		Websites or sitemap links
maxUrlsPerSite	10,000	Cap per site
lastmodSince		Only URLs changed on or after this date: 2026-10-01, 7 days, 2 weeks, yesterday
keepUndated	false	With lastmodSince, keep URLs that have no lastmod
includePatterns, excludePatterns		Regular expressions on the URL
includeExtras	true	Images, hreflang, news, video titles
checkStatus	false	HTTP status of each URL
maxStatusChecksPerSite	1,000	Cap on status checks per site
maxSitemapsPerSite	1,000	Cap on sitemap files read per site

What it costs

Run	Price
A blog with 2,000 URLs	$0.50
20 sites, 500 URLs each	$2.50
Pages changed in the last week on a 1-million-URL news site (lastmodSince)	only the new URLs are charged
1,000 URLs with status checks	$1.25

You only pay for URLs you get. The run stops cleanly at your spending limit.

Good to know

• Only what the sitemaps list is returned. Pages a site leaves out of its sitemaps are not found; this is not a crawler.
• lastmod is whatever the site writes; some sites put today's date on every URL.
• Status checks are polite (about 4 requests a second per site), so checking 10,000 URLs of one site takes around 45 minutes. Checks on different sites run in parallel. The log shows progress, and a run that is about to hit its timeout stops checking and returns everything it has (checks not done are not charged).
• A site that answers its sitemap URL with an HTML page or JSON is reported as "Not a sitemap".

FAQ

Can I find every page that changed since my last run? Yes: schedule the Actor daily with Changed since = 1 day (or weekly with 7 days). The date moves with every run.

Can I use the URLs in another Actor? Yes. Pass the dataset to Bulk HTTP Requests, Website Screenshot + Change Detection or any crawler that takes a URL list.

Can an AI agent use it? Yes, through the Apify MCP server: the agent gives a website and gets its URL list back.

More from this developer

• Bulk HTTP Requests: call any API once for every row of a CSV, JSON list or Apify dataset.
• Website Screenshot + Change Detection: screenshots of many URLs on desktop and mobile, with visual change detection between runs.
• Document to Markdown: PDF, Word, PowerPoint, Excel and scans to clean Markdown for LLMs and RAG.
• Keyword Search Volume: Google search volume, CPC and trends for a keyword list, in up to 20 countries in one run.
• Audio to Text: audio, video and podcast feeds to timestamped text and subtitles with Whisper.
• Video to Scenes: scene cuts, keyframes, a contact sheet and the transcript of each scene.
• Career Site Jobs: open jobs from Greenhouse, Lever, Ashby, Workable, Recruitee and Workday career sites.
• Image to Text OCR: text from images and scanned PDFs in 40+ languages, with confidence and word boxes, $1.50 per 1,000 images.
• Tech Stack Detector: which CMS, ecommerce platform, analytics and hosting a website uses, $1.50 per 1,000 websites.

Code examples (curl, Python, Node.js, n8n) for this and the other Actors: github.com/msalihk/apify-actor-examples.

Questions or a missing feature: open an issue on the Actor page.