Sitemap URL Extractor: All Pages of a Website
Pricing
$2.00 / 1,000 site mappeds
Sitemap URL Extractor: All Pages of a Website
List every URL a website publishes in its sitemaps: finds them via robots.txt, follows sitemap indexes and .gz files, returns lastmod dates, recently updated pages and counts by section.
Pricing
$2.00 / 1,000 site mappeds
Rating
0.0
(0)
Developer
Tudor Lane
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
12 hours ago
Last modified
Categories
Features
Share
Use when: an SEO, crawl or research agent needs the full list of pages a website publishes in its sitemaps, with lastmod dates and counts by section, without crawling the site.
Try this input (also the default, just click Start)
{"requests": [{"url": "https://www.gov.uk", "max_urls": 500, "max_sitemaps": 3}, {"url": "https://docs.apify.com", "max_urls": 500}]}
Sample output (real run, 16 Nov 2026, trimmed)
{"site": "https://www.gov.uk", "found_via": "robots.txt", "sitemaps_read": 2, "sitemaps_pending": 35, "url_count": 500, "truncated": true, "sections": {"/government": 303, "/employment-tribunal-decisions": 76, "/hmrc-internal-manuals": 52}, "newest_lastmod": "2026-11-09T09:27:06+00:00", "recently_updated": [{"url": "https://www.gov.uk/government/news/princes-trust-celebrates-40th-anniversary", "lastmod": "2026-11-09T09:27:06+00:00", "priority": "0.29"}], "error": null}{"site": "https://docs.apify.com", "found_via": "robots.txt", "sitemaps_read": 2, "url_count": 500, "truncated": true, "sections": {"/api": 295, "/academy": 202}, "newest_lastmod": null, "error": null}
Options
- Pass a site URL (robots.txt
Sitemap:lines, then common paths) or a sitemap / sitemap-index URL directly. includeis an optional regex to keep only matching URLs (e.g./blog/). Follows sitemap indexes and.gzsitemaps.max_urls(default 5000, cap 50,000) andmax_sitemaps(default 20, cap 200) keep large sites cheap.truncatedis true when a cap was hit.- Up to 200 sites per run in
requests. No API key needed.
Pricing: pay per event, $0.002 per site mapped. Errors are not charged. No subscription.
Limits & failure modes
- It only reads the site's public robots.txt and sitemap files; it does not crawl pages, and sites with no sitemap return an uncharged error.
- Caps apply: this default run stopped at 500 URLs on both sites (
truncated: true). Some sitemaps have nolastmod(docs.apify.com in the sample). - Empty or invalid URLs return an uncharged error (
url must be http(s)://...). Private/local addresses are blocked.
Last tested: 16 Nov 2026.
Use from AI agents (MCP)
- Apify MCP server: add
https://mcp.apify.com/?tools=tudorlane/sitemap-url-listerto Claude, Cursor, ChatGPT, VS Code or any MCP client. - REST:
POST https://api.apify.com/v2/acts/tudorlane~sitemap-url-lister/run-sync-get-dataset-items?token=YOUR_TOKENwith the input JSON.