Broken Link Checker: Crawl a Site, Find 404s & Redirects
Pricing
from $0.40 / 1,000 link checkeds
`Broken link checker: crawl a site over plain HTTP, check every link once for 404s, redirect chains and timeouts. $0.40 per 1,000 links.`
Pages to start crawling from, one per line. Only pages on the same site (same host, www ignored) are crawled; links to other sites are only checked. Social-media and Google hosts are refused.
Optional sitemap.xml URLs (index files and .xml.gz work, up to 25 files). Their URLs are checked too, and the report lists sitemap URLs no crawled page links to and crawled pages missing from the sitemap. You can use a sitemap without start URLs.
How many same-site pages are fetched and read for links. Default 100, hard cap 10,000. Same-site links beyond this cap are still checked, just not read. Crawling is limited by the per-host rate, so 100 pages take a few minutes.
Upper limit of unique link targets per run (each one is a row). Default 5,000, hard cap 20,000. Further links are ignored and the report says so.
Also check links to other sites (checked once each, never crawled). Their robots.txt is obeyed too: disallowed links are not requested and come back as free blocked rows.
Also check img src, script src and link href targets, not just a href links.
Output and charge only rows that are not OK (broken, redirect, timeout; blocked rows are listed free). Working links are still counted in the report. On a healthy site this means a lot of unpaid crawling, so the run stops after 1,000 requests that produced no charged row.
Some sites ask crawlers, in robots.txt (Crawl-delay), to wait more than 2 seconds between requests. The Actor always waits the full delay and never shortens it. Off (default): links on such a site you are crawling are not requested; they come back as free rows with errorCode slow-host. On: delays of 2 to 2.8 seconds are honoured in full; over 2.8 seconds the site is always skipped (maximum this Actor honours). A slow site makes the run long, and compute is billed while the run waits, so turn this on only when you need that site checked. Applies to the sites you crawl; other sites linked from them are checked once per link and their robots.txt is not read.
Query parameter names (for example utm_source, ref, sort) removed from every link before de-duplication, so tracking or sorting variants of one page are one row. Session IDs in jsessionid and PHPSESSID are always ignored when de-duplicating; sid is not (many sites use it as a content ID), add it here if it is a session ID on your site. The first URL found is the one requested.
Start rate for each host, external hosts included. Default 1, maximum 4. A larger Crawl-delay in robots.txt on a site you crawl wins and is never shortened (see Allow slow hosts).
Global cap on requests in flight across all hosts (1 to 20).
The run stops cleanly after this time and saves what it has. Default 60, maximum 360.
