Broken Link Checker for Your Sitemap

Find broken links, missing images and dead scripts or stylesheets on your own website.

Give it your homepage or your sitemap URL. The Actor reads the sitemap (sitemap indexes and .xml.gz files work), fetches each page once, collects every <a href>, <img src>, <link href> and <script src>, and checks each unique target once. You get one result per broken or problematic link, with the page it was found on, plus a summary. Run it again later and each problem is marked as new or stillBroken, and you see how many were fixed.

This Actor is meant for checking sites you own or manage. It respects robots.txt and keeps the request rate low.

How it works

1. Pages. If startUrl looks like a sitemap (ends in .xml or .xml.gz, or has sitemap in the path), that sitemap is read. Otherwise the sitemaps listed in your robots.txt are used, or /sitemap.xml if robots.txt lists none. Only URLs on the same site are taken (www. is ignored, other subdomains are not the same site). Up to maxPages.
2. Crawl fallback. If no sitemap is found, the Actor starts at startUrl and follows <a href> links on the same site until maxPages pages have been queued.
3. Links. Each HTML page is downloaded with GET (up to 5 MB) and parsed. Links with mailto:, tel:, javascript:, data: and plain #anchors are ignored. Fragments are removed, so /page#a and /page#b are one target.
4. Checks. Each unique target is requested once with HEAD. If the server answers 403, 405 or 501 to HEAD, the Actor tries GET (without reading the body). Redirects are followed, up to 10. Pages already downloaded in step 3 are not requested again.

Requests identify themselves with the User-Agent Mozilla/5.0 (compatible; BrokenLinkChecker/1.0; +https://apify.com/isnard/broken-link-checker). Timeout is 15 seconds per request.

Being polite

• robots.txt. Pages and links on other websites are only requested when their robots.txt allows it, using the rules for BrokenLinkChecker or *. This cannot be turned off. On your own site it is also respected by default. Set respectRobots to false only if you own the site and want to check paths you hide from crawlers.
• Rate limit. At most 4 requests per second to your site, and at most 1 request per second to each other host. If your robots.txt has a Crawl-delay, that is used instead (up to 10 seconds).
• Concurrency. 4 requests in flight by default.

Input

{
    "startUrl": "https://example.com",
    "maxPages": 200,
    "checkExternal": true,
    "maxLinksPerPage": 500,
    "concurrency": 4,
    "respectRobots": true
}

• startUrl (required): your homepage or your sitemap URL. https:// is added when there is no scheme.
• maxPages (default 200): pages of your site to scan.
• checkExternal (default true): also check links to other websites. With false they are counted in the summary but not requested.
• maxLinksPerPage (default 500): links taken from each page, in document order.
• concurrency (default 4, max 20): requests at the same time.
• respectRobots (default true): see above. Only affects your own site.

Output

One dataset item per problem, that is, per pair of page and link. A missing logo linked from every page gives one item per page, so you see every place to fix.

Real example from a test run:

{
    "type": "link",
    "sourcePage": "https://crawlee.dev/blog/authors",
    "targetUrl": "https://crawlee.dev/blog/img/lukasp.webp",
    "tag": "link",
    "linkText": "",
    "isExternal": false,
    "issue": "broken",
    "status": 404,
    "error": null,
    "finalUrl": "https://crawlee.dev/blog/img/lukasp.webp",
    "redirectCount": 0,
    "redirectHops": null,
    "change": "new"
}

• issue:
  • broken: the final status is 400 or higher.
  • error: no HTTP answer (DNS failure, connection refused, TLS error, timeout) or more than 10 redirects. The reason is in error.
  • blocked: the server answered 401, 403, 429 or 999, which usually means it refuses automated requests. The page often works in a browser, so check it by hand before fixing anything.
  • longRedirectChain: the link works but needs more than 3 redirects. redirectHops lists every hop with its status.
• tag: a, img, link or script. sitemap when a URL listed in your sitemap is itself broken (then sourcePage is the sitemap). startUrl when the start URL itself fails (then sourcePage is null).
• linkText: anchor text of an <a> or alt of an <img>, first 100 characters.
• change: new or stillBroken, compared with the previous run for the same startUrl.

The last item is the summary:

{
    "type": "summary",
    "startUrl": "https://crawlee.dev/",
    "mode": "sitemap",
    "sitemaps": ["https://crawlee.dev/sitemap.xml"],
    "pagesScanned": 3,
    "linksChecked": 46,
    "skippedByRobots": 0,
    "externalSkipped": 0,
    "problems": 1,
    "broken": 1,
    "errors": 0,
    "longRedirectChains": 0,
    "new": 1,
    "stillBroken": 0,
    "fixed": 0,
    "firstRun": true,
    "stoppedByCostLimit": false,
    "runErrors": []
}

• pagesScanned: HTML pages downloaded and parsed (the billed count).
• linksChecked: unique URLs requested, including the pages.
• fixed: problems from the previous run that are gone. A problem only counts as fixed when its page was scanned again, so a run with a smaller maxPages does not look like a cleanup.
• runErrors: sitemaps that could not be read and similar notes. The Actor does not fail because of a bad page or link.

Comparing runs

Problems are stored in a named key-value store called broken-link-checker-state in your Apify account, one record per startUrl. Schedule the Actor weekly and filter the dataset by change = new to see only what broke since last time. Delete the store to start fresh.

Pricing

Pay per event: page-scanned, once for each HTML page of your site that was downloaded and parsed. Checking links is not billed, and pages that fail or are not HTML are not billed (they show up as problems instead). You can cap the cost of a run with the "Maximum cost per run" setting. When the cap is reached, no more pages are scanned, but the links already found are still checked and reported.

Limitations

• Only links in the HTML are seen. Links added by JavaScript, CSS url(...), srcset and iframes are not checked.
• Some sites answer 403, 429 or 999 to automated requests even when the page works (LinkedIn, some shops behind bot protection). These show up as broken with that status. Check them by hand.
• Each target is checked once per run with no retry. A timeout from a slow server is reported as error.
• Checks run from Apify's servers. Pages only reachable from your network or behind a login are reported as errors.
• The run takes about 1 second per 4 internal links at the default rate limit. A site with many unique links per page takes longer.

Run locally

python -m venv .venv
.venv/Scripts/pip install -r requirements.txt   # on Linux/macOS: .venv/bin/pip
# put input in storage/key_value_stores/default/INPUT.json
.venv/Scripts/python -m my_actor
python -m unittest discover -s tests -t .

Author: Matias Isnard.