Sitemap Extractor, Validator & Generator
Pricing
from $0.30 / 1,000 sitemap url listeds
Sitemap Extractor, Validator & Generator
All-in-one XML sitemap tool: extract every URL from a site's sitemaps (auto-discovered, indexes, .gz), validate against sitemaps.org and Google rules (lastmod, duplicates, 50k limit, robots-blocked, 404, noindex, canonical), or crawl a site politely to generate sitemap.xml. AI-ready summary.
Pricing
from $0.30 / 1,000 sitemap url listeds
Rating
0.0
(0)
Developer
Jerry Lin
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
6 days ago
Last modified
Categories
Features
Share
What does Sitemap Extractor, Validator & Generator do?
One Actor for everything XML sitemap: give it a website (e.g. https://example.com) and choose a mode.
| Mode | What you get |
|---|---|
| 📥 extract | Every URL from the site's sitemaps with lastmod, changefreq, priority. Sitemaps are auto-discovered from robots.txt and /sitemap.xml; sitemap indexes, .xml.gz and .txt sitemaps are followed. |
| ✅ validate | Extraction plus a full health check against the sitemaps.org protocol and Google's rules, and optional HTTP checks of the listed pages. Only URLs with issues are listed, with clear issue codes. |
| 🛠️ generate | Politely crawls the site and builds a fresh sitemap.xml (and sitemap.txt) with only indexable pages – no noindex, no non-canonical duplicates, no error pages, no login pages. |
Every run also writes a compact OUTPUT_SUMMARY with an llmSummary paragraph – ideal for AI agents and the Apify MCP server.
What does the validator check?
Sitemap files: HTTP errors, unreachable or robots-blocked sitemaps, HTML instead of XML, XML parse errors, wrong root element, missing sitemaps.org namespace, more than 50,000 URLs or 50 MB, empty files, nested sitemap indexes, redirecting sitemap URLs, sitemaps not listed in robots.txt, no sitemap at all.
Every URL: invalid or relative URLs, URLs longer than 2,048 characters, #fragments, whitespace, different host / www / http-vs-https than the sitemap, duplicate URLs, invalid or future lastmod, invalid changefreq and priority, and URLs listed in the sitemap but disallowed by robots.txt (a classic Search Console warning).
With HTTP checks (default on, capped by Max URLs to HTTP-check): 404/410, 5xx, unreachable pages, redirects (with final URL), noindex (meta robots or X-Robots-Tag) and canonical pointing elsewhere – all of which should not be in a sitemap.
Each row gets severity = error, warning or ok.
Polite and compliant by design
- ✅ robots.txt always honored (and its
Crawl-delay); disallowed URLs are never fetched. - ✅ Per-host rate limit (default 2 requests/s per host, never faster than 4/s) and max 5 parallel requests.
- ✅ Public pages only – no login, no cookies, no CAPTCHA solving. Clear User-Agent
SitemapValidatorBot. - ✅ Extraction downloads only the sitemap files – not the pages – unless you ask for HTTP checks.
How to use it
- Enter your website or sitemap URL(s).
- Pick mode:
extract,validateorgenerate. - Adjust limits (Max sitemap URLs, Max URLs to HTTP-check, Max pages to crawl).
- Click Start and open the Overview table,
OUTPUT_SUMMARY, or – in generate mode – downloadSITEMAP_XMLfrom the Storage tab.
Example input (validate)
{"urls": ["https://example.com"],"mode": "validate","maxUrls": 10000,"checkUrlStatus": true,"maxStatusChecks": 300}
Example output row
{"severity": "error","url": "https://example.com/old-landing","issues": ["not_found"],"httpStatus": 404,"sitemapUrl": "https://example.com/sitemap-pages.xml","lastmod": "2024-05-02"}
Example summary
{"totalUrls": 8421,"errorRows": 37,"warningRows": 112,"topIssues": [{ "issue": "redirects", "count": 64 }, { "issue": "not_found", "count": 21 }],"llmSummary": "Found 4 sitemap file(s) with 8421 URLs for https://example.com/. 37 URL errors, 112 warnings. Top issues: redirects×64, not_found×21, ..."}
How much does it cost?
Pay-per-event pricing – no subscription, no separate compute bill:
| Event | Price |
|---|---|
url-listed – one URL read from a sitemap (extract / validate) | $0.0003 |
page-checked – one page fetched over HTTP (validate HTTP checks, generate crawl) | $0.001 |
| Actor start (per GB of memory; default 512 MB = 1 event) | $0.002 |
Examples: extract 10,000 URLs ≈ $3.00; validate 1,000 URLs with 200 HTTP checks ≈ $0.50; generate a sitemap from 200 pages ≈ $0.20. URLs skipped because of robots.txt are free. Cap the cost with Max sitemap URLs, Max URLs to HTTP-check and Max pages to crawl, or set a max cost per run – the Actor stops cleanly at the limit.
Use it from AI agents (MCP)
https://mcp.apify.com?tools=rationalistic_tail/sitemap-validator-generator
An agent can ask "list every URL on example.com", "is this sitemap valid?" or "make me a sitemap.xml" and read OUTPUT_SUMMARY.llmSummary. Runs with limited permissions, so prepaid agent tokens (Apify AGI, x402) work too.
FAQ
Does generate mode render JavaScript? No, it follows links in the server-rendered HTML (up to 5,000 pages). For JS-only sites, provide your framework's route list instead.
Can I submit the generated sitemap to Google? Yes – upload SITEMAP_XML to your site root and add Sitemap: https://your-site/sitemap.xml to robots.txt.
Gzipped sitemaps? Yes, .xml.gz files are decompressed automatically.