Sitemap Crawler & Broken Link Checker
Pricing
from $0.50 / 1,000 sitemap url listed, or skipped under robots.txts
Discover every URL a site publishes through robots.txt and its sitemaps, check that each one still responds, and find broken links inside its pages. Handles sitemap indexes, gzipped sitemaps and servers that reject HEAD requests.
List the URLs a site publishes, check that those URLs respond, or scan pages for broken links inside them.
One per line, as a domain or any URL on it. The Actor reads robots.txt for Sitemap: lines, then falls back to /sitemap.xml and /sitemap_index.xml.
Check these exact URLs instead of discovering them. In broken-link mode these are the pages to scan. Leave empty to use the sitemaps from Sites above.
For List sitemap URLs and Check URL health. The main control on run size: large sites publish tens of thousands of URLs.
For Find broken links with a site: how many pages from its sitemap to open and scan. Each page can hold up to Max links per page results, so this sets the run's size.
For broken-link mode. A navigation-heavy page can hold hundreds of links; each unique link is checked once per run and reused across pages.
Ignore links pointing to other domains. Useful when you only care about your own broken pages.
Skip URLs the site's robots.txt disallows, and report them as skipped rather than broken. Leave this on unless you own the site and know what you are doing.
Stop the run after saving this many items. Leave empty for no limit.
This Actor sends traffic to a real website. Four at a time is polite; raise it only against a site you own.
