Sitemap Extractor: All URLs from sitemap.xml, robots.txt & .gz
Pricing
from $0.10 / 1,000 urls
Sitemap Extractor: All URLs from sitemap.xml, robots.txt & .gz
Sitemap extractor and URL finder: get every page URL of a website with lastmod, changefreq and priority. Finds sitemaps via robots.txt and common locations, reads nested indexes, .gz, text sitemaps and RSS/Atom feeds, images, videos, news and hreflang. Status check and new-URL monitoring.
Pricing
from $0.10 / 1,000 urls
Rating
0.0
(0)
Developer
Makerio
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
11 hours ago
Last modified
Categories
Features
Share
Sitemap Extractor
Get every URL from a website's sitemaps: give it a site or a sitemap URL and get one clean row per page with lastmod, changefreq, priority, images, videos, news and hreflang alternates. It finds the sitemaps for you, reads nested sitemap indexes, .gz files, plain-text sitemaps and RSS/Atom feeds, and keeps going when a sitemap is broken. $0.10 per 1,000 URLs.
What you can use it for
- Crawl seeding: a complete list of a site's pages to feed into your own scraper or crawler.
- SEO audits: check what a site tells search engines: missing
lastmod, URLs blocked by robots.txt, 404s and redirects in the sitemap, hreflang pairs. - Competitor monitoring: schedule it with Only new URLs and get the pages a competitor published since the last run.
- Content inventories and migrations: every URL, with dates, before you move a site.
- News and media: article titles and publication dates from news sitemaps, image and video URLs from media sitemaps.
How it works
- Finding the sitemaps. For a website, the Actor reads
robots.txtand follows itsSitemap:lines. If there are none, it tries the usual locations:/sitemap.xml,/sitemap_index.xml,/sitemap-index.xml,/wp-sitemap.xml,/sitemap.xml.gz,/sitemap.txtand a few more, and stops at the first one that works. If you give a sitemap URL that turns out to be wrong, it falls back to the same search. - Reading them. Sitemap indexes are followed to any depth (up to 10 levels), and a sitemap that links back to itself is read once. Files are streamed, so a 50 MB sitemap never has to fit in memory. Gzip is recognized by its content, not by the file name or headers, so
.gzfiles served as plain text and plain files named.gzboth work. - Forgiving parsing. Missing closing tags, namespace prefixes (
ns0:url), upper-case tags, CDATA and stray characters do not stop the run. An HTML page served at/sitemap.xml(a "soft 404") is reported as "not a sitemap" instead of being read as one. - One row per URL. A URL listed in several sitemaps is saved once. When a URL is in both a sitemap and a feed, the sitemap's data wins.
Measured on a site with about a million URLs in 23 sitemap files: all of them read in 15 seconds with about 100 MB of memory.
Input
| Field | What it does |
|---|---|
| Websites or sitemap URLs | A website, a sitemap or sitemap index (.xml, .xml.gz, .txt), a robots.txt file or an RSS/Atom feed. Bare domains such as example.com work too. |
| Max URLs | Stop after this many URLs in total. 0 means no limit. |
| Only URLs matching / Skip URLs matching | Patterns such as */blog/* or *.pdf, or a regular expression in slashes. |
| Only URLs modified since | A date (2026-10-01) or a period (7 days, 24 hours, 3 months). Child sitemaps whose own lastmod is older are skipped without being downloaded, which makes large news sites fast. |
| Only new URLs since the last run | Save only URLs that were not in the sitemaps last time. The summary also counts and lists removed URLs. |
| Check the HTTP status of every URL | Adds the status code, redirect target and response time of each URL. |
| Images, videos, news and hreflang | On by default. Turn off for leaner rows. |
Example input:
{"startUrls": [{ "url": "https://www.example.com" },{ "url": "https://shop.example.org/sitemap_products_1.xml" }],"excludeUrlPatterns": ["*/tag/*"],"modifiedSince": "30 days"}
Output
One row per URL:
{"url": "https://www.example.com/blog/how-we-ship","lastmod": "2026-11-12T09:30:00+00:00","changefreq": "weekly","priority": 0.8,"title": null,"images": [{ "url": "https://www.example.com/img/ship.jpg", "title": null, "caption": "Our warehouse" }],"videos": [],"news": null,"alternates": [{ "hreflang": "de", "url": "https://www.example.com/de/blog/how-we-ship" }],"sitemapUrl": "https://www.example.com/post-sitemap.xml","startUrl": "https://www.example.com","blockedByRobotsTxt": false}
lastmodis exactly as the sitemap gives it.titleis filled for feed items, andnewsholds the title, publication and date from news sitemaps.blockedByRobotsTxtistruewhen the site's robots.txt disallows the URL for Googlebot. A sitemap should not list such URLs.- With Check the HTTP status of every URL, each row also gets
statusCode,finalStatusCode,finalUrl(where redirects end),redirects,responseTimeMsandstatusError(for exampleTIMEOUT,DNS_FAILUREorBLOCKED_BY_ROBOTS_TXT).
The dataset has views for URLs, Status check, Images, videos, news and Hreflang. Export to JSON, CSV, Excel or HTML, or get the URLs alone with the fields=url option of the dataset export.
The OUTPUT record in the key-value store has a summary per start URL:
- how the sitemaps were found (robots.txt, a common location or the start URL) and what robots.txt lists;
- every sitemap file with its HTTP status, type (
urlset,sitemapindex,rss,atom,text), whether it was gzipped, its size, its URL count and any error (HTTP_404,NOT_A_SITEMAP,NO_URLS,TIMEOUT,GZIP_ERROR); - URLs found, saved, duplicated, filtered out and blocked by robots.txt;
- status code counts, and new and removed URLs when monitoring.
A start URL without any sitemap does not fail the run: the summary says what was tried and why nothing was found.
Pricing
| Event | Price |
|---|---|
| URL saved | $0.0001 ($0.10 per 1,000) |
| Status check, when turned on | $0.0002 per URL checked ($0.20 per 1,000) |
Finding and reading the sitemaps, the summary, and URLs that your filters skip are free. With Only new URLs, you pay only for the new ones.
| Run | URLs | Cost |
|---|---|---|
| A blog | 800 | $0.08 |
| An online store with status checks | 5,000 | $1.50 |
| A large site | 100,000 | $10 |
| A daily new-page check that finds 12 new URLs | 12 | $0.0012 |
You can set a maximum cost per run in the run options. The Actor stops at that limit and keeps what it has saved.
Monitoring new pages
- Turn on Only new URLs since the last run and save the input as a task.
- Add a daily or weekly schedule.
- Each run saves only the URLs that appeared since the run before. Send them to Slack, email, Google Sheets or a webhook with Apify integrations.
The first run saves every URL as a baseline. The lists of seen URLs are kept in the key-value store sitemap-extractor-monitor of your account, one record per start URL. Give runs different Monitor name values to keep separate lists for the same site.
Questions
The site has no sitemap. The summary lists what was tried. This Actor only reads sitemaps and does not crawl pages. To find pages by following links, use a crawler.
Why are there fewer URLs than the site has pages? A sitemap lists what the site owner put in it, which is often not every page. Check filteredOut and duplicates in the summary too.
Does it work with WordPress, Shopify, Wix and Squarespace? Yes. Their sitemaps are found through robots.txt or the standard locations (/wp-sitemap.xml, Yoast and Rank Math /sitemap_index.xml, Shopify's /sitemap.xml with product, collection and blog sitemaps).
Does it use proxies? No. Sitemaps are public files meant for bots, so the Actor requests them directly with its own user agent and retries on 429 and 5xx answers, honouring Retry-After. Sites that block all bots will show the error in the summary.
Is a big sitemap slow or expensive? Sitemaps are streamed and parsed as they download. You pay per URL saved; use Max URLs, patterns or Only URLs modified since to take only what you need.
Can I use it from my code or an AI agent? Yes, through the Apify API, the JavaScript and Python clients, or the Apify MCP server.