Sitemap URL Extractor & SEO Health Checker

Give it a website. It reads the site's robots.txt and XML sitemaps (including sitemap indexes and .gz files), lists every URL, and checks each one: HTTP status, redirects, canonical tag, noindex, title, meta description and H1. You get a clean table plus a per-site issue summary. Pay per URL.

What does it do?

• Finds sitemaps automatically from robots.txt, or falls back to /sitemap.xml.
• Lists every URL with lastmod, changefreq and priority.
• Optionally fetches each URL and reports:
  • status code, final URL and the full redirect chain
  • response time
  • X-Robots-Tag and meta robots (noindex)
  • canonical URL and whether it points elsewhere
  • title (flagged if missing or longer than 60 characters), meta description, H1 count, hreflang count, language and word count
• Tags each problem with a simple issue code (e.g. HTTP_404, REDIRECT_CHAIN, META_NOINDEX, CANONICAL_POINTS_ELSEWHERE) and writes a per-site SUMMARY.

What data does it collect?

Only technical, page-level SEO data that the website publishes openly: URLs, HTTP headers, and standard HTML head tags. It does not collect personal data, emails or contact details. It does not log in, and it respects robots.txt by default. Disallowed URLs are listed but not fetched, and a site's Crawl-delay is honoured. Requests identify themselves with an honest user agent (SitemapUrlHealthBot), and a 429 Too Many Requests answer is respected before one polite retry.

How much does it cost?

Event	Price
URL listed from a sitemap (no fetch)	$0.0002
URL fetched and checked	$0.0008

Apify also adds its standard tiny start fee (about $0.00005 per run). For example, checking a 1,000-page site costs about $0.80, and only listing it costs $0.20. URLs skipped because of robots.txt are charged only as "listed". Use Max URLs per site and the run's "Maximum cost per run" to cap spending: the Actor never starts work it cannot bill within your limit.

Input example

{ "startUrls": [{ "url": "https://example.org" }], "maxUrlsPerSite": 5000, "checkUrls": true, "extractSeo": true }

Output example

Three dataset views: URL health, Problems only, and Sitemap inventory. A per-site SUMMARY record (issue counts, sitemaps read, crawl-delay) is saved in the key-value store.

{
  "url": "https://data.gov.my",
  "sitemap": "https://data.gov.my/sitemap-0.xml",
  "lastmod": "2026-10-25T05:48:07.104Z",
  "statusCode": 200,
  "finalUrl": "https://data.gov.my",
  "redirectChain": [],
  "responseMs": 120,
  "title": "data.gov.my",
  "metaDescription": "Your one-stop interface to browse Malaysia's wealth of open data!",
  "canonical": null,
  "h1Count": 0,
  "isIndexable": true,
  "issues": ["MISSING_H1", "MISSING_CANONICAL"]
}

Use cases

• A technical SEO health check before or after a migration.
• Finding broken (4xx/5xx) or redirected URLs that are still in your sitemap.
• Spotting noindexed or wrongly canonicalised pages that waste crawl budget.
• Feeding a site's full URL inventory into other Actors or AI agents (screenshots, content extraction, monitoring).

FAQ

My site has no sitemap. The run reports 0 URLs and lists the sitemap locations it tried. A link-crawl fallback is planned.

Can I run it on competitors' sites? Sitemaps are published for crawlers, and the Actor respects robots.txt. Still, follow each site's terms, and keep concurrency low on small sites.

Problems? Open an issue in the Issues tab. We respond within 2 business days.