Sitemap URL Extractor Pro
Pricing
from $0.50 / 1,000 url extracteds
Sitemap URL Extractor Pro
Extract every URL from any website's sitemaps: auto-discovery via robots.txt, sitemap indexes, gzipped .xml.gz, TXT and RSS sitemaps, lastmod, hreflang, images, robots.txt conflicts and optional HTTP status codes.
Pricing
from $0.50 / 1,000 url extracteds
Rating
0.0
(0)
Developer
SteadyUtil
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
an hour ago
Last modified
Categories
Features
Share
Sitemap URL Extractor Pro: get every URL of any website from its sitemaps
Sitemap URL Extractor Pro turns any domain or sitemap link into a clean list of every page URL the site publishes, with lastmod, changefreq, priority, hreflang alternates, image URLs and (optionally) the live HTTP status of each page. Give it example.com and it finds the sitemaps by itself: robots.txt, sitemap indexes nested several levels deep, gzipped .xml.gz files, plain-text .txt sitemaps and RSS/Atom feeds.
It's built for SEO audits, content inventories, migrations, competitor research and as a fast pre-step for crawlers and AI/RAG pipelines. You get the URLs in seconds without crawling the whole site.
Why use it?
- Auto-discovery that actually finds sitemaps. It reads
Sitemap:lines in robots.txt, falls back to common locations (/sitemap.xml,/sitemap_index.xml,/wp-sitemap.xml,/sitemap.xml.gz,/sitemap.txtand more), and retrieshttp://sitemaps overhttps://, which is a common robots.txt mistake on big news sites. - Every format. Sitemap indexes (any depth), urlsets,
.xml.gz(with or without gzip headers), TXT sitemaps, RSS and Atom feeds, namespaced XML and CDATA. - Rich output.
lastmod,changefreq,priority, the sitemap each URL came from, image URLs, hreflang alternates, and Google News and video titles. - SEO checks built in. It flags sitemap URLs that robots.txt blocks (a classic SEO bug), and with one switch checks the HTTP status, redirects and final URL of every page.
- Filters that cut cost. Include/exclude regex patterns, "modified after" date, same-host only, and per-site and total URL limits. You only pay for URLs you keep.
- A bad site never breaks your run. Dead sitemaps, HTML soft-404 pages, DNS errors and timeouts are recorded in a per-site
SUMMARYand the run still succeeds. Retries use exponential backoff and honourRetry-After. - Fast and cheap. Plain HTTP only (no browser). Over 1,000,000 URLs from apify.com in about 2 minutes on 512 MB in our tests.
How to use it
- Add one or more websites or sitemap URLs (
apify.com,https://example.com/sitemap_index.xml,https://site.com/sitemap.xml.gz). - Optionally set Max URLs (your cost cap), filters, or turn on Check HTTP status.
- Click Start and download the results as JSON, CSV, Excel or HTML, or use the API.
Input example
{"startUrls": [{ "url": "https://apify.com" }, { "url": "https://www.nytimes.com" }],"maxUrls": 5000,"maxUrlsPerSite": 2500,"includeUrlPatterns": ["/blog/", "/2026/"],"excludeUrlPatterns": ["\\.pdf$"],"lastmodAfter": "2026-01-01","checkStatus": false,"respectRobotsTxt": true}
Output example
{"url": "https://www.ebay.com/itm/800940000815","lastmod": null,"changefreq": "daily","priority": 1,"sitemapUrl": "https://www.ebay.com/lst/GTC-0_10.xml.gz","sitemapType": "urlset","site": "https://www.ebay.com/lst/VIS-0-index.xml","host": "www.ebay.com","blockedByRobotsTxt": false,"imageCount": 1,"imageUrls": ["https://secureir.ebaystatic.com/pictures/aw/pics/stockimage1.jpg"],"alternates": [],"news": null,"videoTitle": null}
With Check HTTP status on, each item also has statusCode, finalUrl and redirected.
The run also stores a SUMMARY record in the key-value store. For each site it lists the sitemaps processed, the sitemaps that failed and why, whether robots.txt was found, the URL counts and the duration.
| Field | Description |
|---|---|
url | Page URL listed in the sitemap |
lastmod | Last modification date (ISO 8601) if the sitemap provides it |
changefreq, priority | Sitemap hints, if present |
sitemapUrl, sitemapType | The sitemap file the URL came from (urlset, feed, text) |
site, host | Which of your inputs it belongs to and its hostname |
blockedByRobotsTxt | true if robots.txt disallows the URL even though the sitemap lists it |
imageUrls, imageCount | Images attached to the URL in an image sitemap |
alternates | hreflang alternates { hreflang, url } |
news, videoTitle | Google News and video sitemap data |
statusCode, finalUrl, redirected | Only with Check HTTP status |
Run it from the API
Replace YOUR_API_TOKEN with your Apify API token. The call returns when the run finishes; then read the dataset.
cURL
curl "https://api.apify.com/v2/acts/steadyutil~sitemap-url-extractor-pro/runs?waitForFinish=300" \-X POST \-H "Authorization: Bearer YOUR_API_TOKEN" \-H "Content-Type: application/json" \-d '{"startUrls": [{ "url": "https://apify.com" }],"maxUrls": 1000,"checkStatus": false}'
Fetch results (use the defaultDatasetId from the run):
curl "https://api.apify.com/v2/datasets/DATASET_ID/items?format=json" \-H "Authorization: Bearer YOUR_API_TOKEN"
JavaScript
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });const run = await client.actor('steadyutil/sitemap-url-extractor-pro').call({startUrls: [{ url: 'https://apify.com' }],maxUrls: 1000,checkStatus: false,});const { items } = await client.dataset(run.defaultDatasetId).listItems();console.log(items.map((item) => item.url));
Python
from apify_client import ApifyClientclient = ApifyClient("YOUR_API_TOKEN")run = client.actor("steadyutil/sitemap-url-extractor-pro").call(run_input={"startUrls": [{"url": "https://apify.com"}],"maxUrls": 1000,"checkStatus": False,})items = client.dataset(run["defaultDatasetId"]).list_items().itemsprint([item["url"] for item in items])
The Actor is also available through the Apify MCP server, so AI agents can discover and call it from the Store page.
Related Actors
These SteadyUtil Actors combine well with this one:
- Website Screenshot Pro — after you have the URL list, capture desktop and mobile screenshots (or PDFs) of every page, or of only the new and changed ones.
- Website Tech Stack Detector & SEO Audit — take the unique domains from the sitemap and detect each site's CMS, ecommerce platform, analytics and on-page SEO score.
Automation recipes (Make, n8n, Zapier)
Each recipe starts the same way: in Apify Console, save this Actor as a Task with your input, then connect that Task from Make, n8n or Zapier (search for the Apify app). You can also call the API examples above from an HTTP module.
Weekly sitemap diff to Google Sheets
- Create a Task with your domain, a
lastmodAfterof seven days ago (or no date filter if you want the full list), and Max URLs as a cost cap. - In Apify Console, add a Schedule that runs the Task every Monday.
- In Make, n8n or Zapier, trigger on a finished Apify run (or poll the dataset), then upsert each
urlinto a Google Sheet. Keeplastmod,statusCodeandblockedByRobotsTxtas columns. - Optional: add a filter so the sheet only gets URLs with
statusCode404 orblockedByRobotsTxttrue, and email yourself the new rows.
New blog posts to Slack or email
Use the example Task Extract blog URLs from a sitemap (includeUrlPatterns such as /blog/), schedule it daily, and post any URL whose lastmod is after yesterday to Slack, email or a CMS draft queue.
Feed a crawler or screenshot run
Pass the dataset's url field into Website Screenshot Pro or your own crawler. In n8n, this is two Apify nodes in a row: extract URLs, then screenshot (or audit) them.
Pricing
This Actor uses pay-per-event pricing, so you pay only for results and never for compute time:
| Event | Price |
|---|---|
url-extracted: one URL in your dataset | $0.50 per 1,000 URLs |
url-status-checked: HTTP status check of one URL (optional) | $1.00 per 1,000 URLs |
residential-proxy-mb: only if you turn on Apify residential proxy, per started MB downloaded through it | $0.012 per MB |
Sites without a sitemap, failed sitemaps and filtered-out URLs cost nothing. Without a proxy or with the default datacenter proxy there are no other charges. Residential proxy traffic costs us about $8/GB, so it's passed on at $0.012/MB, and a typical gzipped sitemap is 0.2–2 MB. Use Max URLs to cap the bill, for example maxUrls: 10000 costs at most $5. Apify Store discounts for higher subscription plans apply automatically.
FAQ
How do I get all URLs of a website from its sitemap in 30 seconds?
Paste the domain (for example apify.com) into Websites or sitemap URLs, set Max URLs if you want a cap, and click Start. The Actor finds robots.txt and the sitemaps, then writes every URL to the dataset. Download JSON, CSV or Excel, or use the API examples above. You don't need the exact sitemap URL.
How do I find 404s and broken pages in a sitemap?
Turn on Check HTTP status and, if you like, set Max URLs. Each result gets statusCode, finalUrl and redirected. Filter the dataset (or the SEO issues view) for 404, 5xx or unexpected redirects. Try the public example Task Find 404s in your sitemap.
The site has no sitemap. What happens?
The site is listed in the SUMMARY record with status no-sitemap and the reason. You aren't charged and the run doesn't fail. Some sites, such as python.org and docs.github.com, genuinely publish no sitemap.
Does it crawl the website pages? No. It only downloads robots.txt and sitemap files, unless you enable Check HTTP status, which sends a lightweight HEAD request per URL.
Can it handle huge sites? Yes. Nested indexes and millions of URLs are processed as a stream with de-duplication. Use Max URLs or Max URLs per site when you only need a sample.
Do I need a proxy? Usually not, because sitemaps are meant for bots. If a site rate-limits datacenter IPs, enable Apify Proxy in the input.
Can I get only new or changed pages? Yes. Set Only URLs modified after to a date and schedule the Actor daily or weekly. Pair this with the Google Sheets recipe above.
Can I use it through the API or with AI agents? Yes. It works with the Apify API, the JS and Python clients, Make, Zapier, n8n and the Apify MCP server. Copy the examples in Run it from the API.
Support and feedback
Sitemap URL Extractor Pro is built and maintained by SteadyUtil, a small publisher of reliable, low-cost web utilities on Apify. Found a site that doesn't work, or need a field we don't output yet? Open an issue in the Issues tab of this Actor (fastest, usually answered within 1–2 business days) or email support@steadyutil.com.
Other SteadyUtil Actors: Website Screenshot Pro and Website Tech Stack Detector & SEO Audit.
Legal note
This Actor only reads robots.txt and sitemap files, which websites publish specifically so that automated agents can discover their public pages. It collects URLs and sitemap metadata only, never personal data. By default it respects robots.txt for the sitemap files it downloads. You are responsible for using the results in line with the target website's terms and applicable law.