Sitemap Crawler & Broken Link Checker

Find every URL a site publishes, confirm each one still responds, and catch broken links inside its pages. No API key, no account, no third-party service.

Mode	What you get
List sitemap URLs	Every URL from the site's sitemaps, with last-modified date, change frequency and priority
Check URL health	HTTP status, redirect chain, final URL, content type, size and response time for each URL
Find broken links on pages	Every link on each page, with its status and which page it was found on

Why use this one

• HEAD first, GET on rejection. Many servers answer a HEAD request with 405, 501 or 403 while serving the same URL fine over GET. Checkers that only send HEAD report those pages as broken. This one retries with GET and tells you which method produced the answer, so a clean report means a clean site. It is the single most common bug in this category, and it is covered by tests here.
• It finds the sitemap the way a search engine does. robots.txt first for its Sitemap: lines, then /sitemap.xml and /sitemap_index.xml. A sitemap index is followed into its children, and gzipped sitemaps are unpacked.
• Redirects are reported, not hidden. A URL that quietly 301s somewhere else is not broken, but you usually want to know. You get redirected, redirectCount and finalUrl on every row.
• Links are de-duplicated across pages. A link in your site-wide footer is checked once per run, not once per page, which keeps both the cost and the load on the target site down.
• robots.txt is honoured, using Python's own robots parser. Disallowed URLs come back marked skipped, never silently dropped and never counted as broken.

How to use it

1. Choose What to do.
2. Enter a Site (a domain is enough) or paste exact URLs to check.
3. Click Start, then download as JSON, CSV, Excel or HTML.

Example input: list everything a site publishes

{
  "mode": "sitemapUrls",
  "sites": ["example.com"],
  "maxUrlsPerSite": 5000
}

Example input: health-check every page in the sitemap

{
  "mode": "checkUrls",
  "sites": ["example.com"],
  "maxUrlsPerSite": 1000
}

Example input: find broken links on specific pages

{
  "mode": "brokenLinks",
  "urls": ["https://example.com/", "https://example.com/blog"],
  "maxLinksPerPage": 200,
  "internalLinksOnly": false
}

To find broken links across a whole site, give sites instead of urls and the Actor will scan the first maxPagesToScan pages from the sitemap (25 unless you raise it).

Output

Sitemap URL

{
  "type": "sitemapUrl",
  "site": "https://example.com",
  "url": "https://example.com/pricing",
  "lastModified": "2026-09-01T00:00:00Z",
  "changeFrequency": "weekly",
  "priority": 0.8,
  "foundIn": "https://example.com/sitemap.xml"
}

URL check

{
  "type": "urlCheck",
  "url": "https://example.com/old-page",
  "status": 301,
  "ok": true,
  "broken": false,
  "redirected": true,
  "redirectCount": 1,
  "finalUrl": "https://example.com/new-page",
  "contentType": "text/html",
  "responseMs": 184,
  "method": "HEAD",
  "error": null
}

Broken link adds foundOn and internal:

{
  "type": "link",
  "url": "https://partner.example/gone",
  "foundOn": "https://example.com/blog",
  "internal": false,
  "status": 404,
  "broken": true,
  "method": "GET"
}

Filter on broken: true to get just the problems.

Robots and identity

Requests carry a normal browser User-Agent, and robots.txt is evaluated against the * rules. If you run this against a site you do not own, check its rules first — the Actor honours Disallow but it does not announce itself as a bot.

Pricing

Result	Price
Sitemap URL listed, or a URL skipped under robots.txt	$0.50 per 1,000
URL checked	$2.00 per 1,000
Link on a page checked	$2.00 per 1,000

Checks cost more than listings because each one is a live request, sometimes two when a server rejects HEAD. Set a maximum spend per run in the run options and the Actor stops cleanly when it is reached.

Limits to know about

• This Actor sends real traffic to a real website. Default concurrency is 4 on purpose. Please raise it only against a site you own, and use the URL cap on large sites.
• Sitemap indexes are followed three levels deep and up to 50 sitemap files per run, which covers any normal site and stops a misconfigured loop.
• lastmod is only as good as the site makes it. Many content systems write today's date on every URL, so treat it as a hint.
• A 403 can mean a bot block, not a broken page. Sites behind an aggressive firewall may reject datacenter traffic; turn on Apify Proxy if a whole site returns 403.
• Links are read from the HTML as served. Links a page builds in JavaScript after loading are not visible to this Actor.
• Response times include network latency from Apify's servers and are useful for comparison between URLs, not as a benchmark of your server.

FAQ

Do I need to own the site? No for checking public URLs, but be considerate: this is live traffic. For broken-link scans across a large site, run it against your own property.

Why is a URL marked skipped? The site's robots.txt disallows it. Turn off Respect robots.txt only if the site is yours.

Why does method say GET on some rows? That URL's server rejected the HEAD request, so the Actor retried with GET to get a trustworthy answer.

Can I schedule this? Yes. Schedule it in Apify Console and connect the dataset to Slack or email, filtered to broken: true, for an alert when something breaks.

Is this legal? The Actor reads public pages and honours robots.txt by default. Keep concurrency reasonable and do not point it at sites you have no relationship with.

Feedback

Found a bug or need a field that's missing? Open an issue on the Actor's Issues tab.