Broken Link Checker: Crawl a Website, Find 404s & Redirects

Broken Link Checker crawls your website from one or more start URLs, collects every link on its pages (and, if you want, every image, script and stylesheet), checks each unique link once, and tells you which ones are broken (404, 410, 500, timeouts, DNS errors, redirect loops) and exactly where they are: the pages that link to them and the anchor text used. Redirects are reported with their full chain, so you can also clean up 301/302 hops after a migration.

It's a plain, fast HTML crawler: no browser, no proxies, no setup. Give it your home page and get a table you can export as CSV or Excel, send to Google Sheets, or get by email after a scheduled run.

What you get

• Each broken link with where it's linked from. For each link: up to 10 of the pages it was found on with the anchor text (or image alt text), plus the total number of pages linking to it (foundOnCount).
• Internal and external links. Links to your own site are crawled; links to other websites are checked once and never crawled.
• Redirect chains and loops. Each hop's URL and status code, the final URL, and redirect loops flagged as broken.
• Sitemap entries checked too. URLs listed in /sitemap.xml (and sitemap index files) are crawled as well, so pages nothing links to are covered, and a 404 in your sitemap shows the sitemap as where it was found.
• Images, scripts and stylesheets (optional): <img src> and srcset, <picture> sources, <script src> and <link rel="stylesheet">.
• Polite by default. At most 2 parallel requests per host, robots.txt respected for crawling, and a clear User-Agent (BrokenLinkChecker/1.0; +https://apify.com/charlesmarkette).

Use cases

• SEO audit: find the 404s and long redirect chains that waste crawl budget and leak link equity, with the pages to fix.
• Site migration: after moving to a new domain, CMS or URL structure, crawl the new site and list every link still pointing at an old URL (redirected) or a page that no longer exists (404).
• Pre-launch QA: crawl a staging or newly launched site and catch broken navigation, missing images and dead external references in the HTML before visitors do (links added by JavaScript aren't seen; see the FAQ).
• Scheduled monitoring with an email alert: schedule a weekly run with Output only broken links on, and add our Notify Step as an integration. By default it sends nothing when the dataset is empty, so you get an email (or a Slack, Teams, Discord or Telegram message) only when something is broken or couldn't be verified (add sites that always block bots, such as linkedin\.com, to Ignore URLs matching to keep those alerts quiet).
• Content and outbound-link hygiene: find external links to sites that went offline or now redirect somewhere unexpected.

How to use it

1. Add your Start URL (usually the home page). Several start URLs are fine; the crawl covers all of their sites.
2. Pick the Crawl scope: Same host (only www.example.com) or Same domain (also example.com, blog.example.com, shop.example.com).
3. Set Max pages to crawl (default 200) and, if you like, turn on Check images, scripts and stylesheets or Output only broken links.
4. Run it. When the crawl finishes, each unique link becomes one row. Use the Overview, Where each link appears and Redirects views, or export the dataset.

Input example

{
  "startUrls": ["https://www.example.com/"],
  "maxPages": 500,
  "crawlScope": "sameDomain",
  "checkResources": true,
  "ignoreUrlPatterns": ["/wp-admin/", "\\?replytocom="],
  "onlyBrokenInOutput": true
}

Output example (one row)

{
  "url": "https://www.example.com/pricing-2023",
  "statusCode": 404,
  "ok": false,
  "statusText": "Not Found",
  "isBroken": true,
  "isUnverified": false,
  "linkType": "internal",
  "resourceType": "link",
  "finalUrl": "https://www.example.com/pricing-2023",
  "redirectCount": 0,
  "redirectChain": [],
  "isRedirectLoop": false,
  "responseTimeMs": 87,
  "contentType": "text/html; charset=utf-8",
  "error": null,
  "foundOnCount": 14,
  "firstFoundOn": "https://www.example.com/",
  "foundOn": [
    {"page": "https://www.example.com/", "anchorText": "Pricing", "nofollow": false},
    {"page": "https://www.example.com/blog/launch", "anchorText": "see our plans", "nofollow": false}
  ],
  "nofollow": false,
  "depth": 1,
  "pageCrawled": false,
  "robotsDisallowed": false,
  "checkedAt": "2026-10-16T09:41:12Z"
}

All columns

• url: the link, normalised (lower-case scheme and host, default port removed, #fragment dropped, /./ and /../ resolved, percent-encoding normalised, query string kept).
• statusCode, ok (2xx), statusText: the final response, after redirects.
• isBroken: true for status 400 or higher (except the four below), any error (timeout, DNS failure, refused connection, TLS error, too many redirects, invalid URL…) and redirect loops.
• isUnverified: true for 401, 403, 429 and 999. These usually mean the site refused an automated check (login required, bot protection, rate limit, LinkedIn's 999), not that the page is gone, so they're not counted as broken. Open them in a browser to be sure.
• linkType: internal (inside the crawl scope) or external. resourceType: link, image, script or stylesheet.
• finalUrl, redirectCount, redirectChain ([{url, statusCode, location}]), isRedirectLoop.
• responseTimeMs, contentType, error, checkedAt.
• foundOn (first 10 {page, anchorText, nofollow}), foundOnCount (all pages), firstFoundOn. A start URL's foundOn lists only the pages that link to it; sitemap URLs show the sitemap file as the page.
• nofollow: at least one link to it had rel="nofollow". depth: clicks from a start URL. pageCrawled: the page was downloaded and read for links. robotsDisallowed: robots.txt doesn't allow crawling it (it was still checked, see FAQ).

A run summary is saved as OUTPUT in the key-value store: pagesCrawled, linksChecked, broken, internalBroken, externalBroken, unverified, redirects, byStatus (count per status code, error for no response), outputRows, notChecked, ignored, externalNotChecked, sitemapUrls, robotsTxtUnreachable and stoppedBy (null, max cost, max results, time limit, or charging error if the platform's charging API kept failing; links whose charge couldn't be confirmed are then left out).

How much does it cost?

Pay per event: $0.001 per checked link + $0.005 per run start, platform usage included.

Apify Store discounts: the price above is for Apify's Free plan. Paid plans pay less per 1,000 checked links: $0.90 on Starter, $0.80 on Scale, $0.70 on Business. The "from" price at the top of the Store page is the Business price.

A "checked link" is each unique URL that gets an HTTP check, and that includes the pages of your site that are crawled (each page is fetched once, and that fetch is its check). A dead link is a result, so links that return 404 or 500, or fail with a DNS error, timeout or refused connection, are charged. These are free: links that aren't valid http(s) URLs (reported as broken with Invalid URL), URLs refused by the private-address guard, mailto:/tel:/javascript: links (skipped), repeats of a link already checked in the run, links matching Ignore URLs matching, and external links when Check external links is off.

With Output only broken links on, every link is still checked and charged; the working ones are just left out of the dataset (broken and unverified ones are kept).

Worked examples:

Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default memory); Apify Proxy (where used) is billed separately.

• A 200-page site with 1,500 unique links (pages included): 1,500 × $0.001 + $0.005 = $1.505.
• A weekly check of a 50-page site with 400 unique links: $0.405 per run, about $1.76 per month.

If you set a maximum cost per run, the crawl stops cleanly at that limit and you get every row already paid for, plus the summary. A link is only checked when the remaining budget can pay for it.

FAQ and limits

Does it see links added by JavaScript?

No. It reads the HTML your server sends, without a browser, so links that only appear after scripts run (some single-page apps, menus built in JavaScript) aren't seen, and <meta http-equiv="refresh"> and JavaScript redirects aren't followed. Server-rendered sites (WordPress, Shopify, most CMSs and static sites) put their links in that HTML, so they're covered.

Why does a link show 403, 429 or 999 when it works in my browser?

Some sites (LinkedIn, Amazon, sites behind Cloudflare bot protection…) block automated requests or requests from data-centre IP addresses. The Actor doesn't use proxies or solve challenges, so it reports the status it actually got, but marks 401, 403, 429 and 999 as isUnverified: true instead of broken: the check couldn't tell whether the page works. This applies to your own site too, if its firewall blocks bots. Use Ignore URLs matching (e.g. linkedin\.com) to leave such domains out.

How are links checked?

External links and resources get a HEAD request first; if the server answers HEAD with any error (4xx/5xx) or drops the connection, the link is checked again with GET, so servers that mishandle HEAD aren't reported as broken. Pages of your site that are crawled are fetched with GET (their HTML is needed). Busy answers (429, 502, 503, 504) are retried twice with backoff, honouring Retry-After up to 10 seconds. Each request waits at most Timeout per request; a whole link, with its redirect hops and retries, gets at most 4× that (at least 60 seconds).

How does robots.txt work here?

With Respect robots.txt on (the default), pages that robots.txt disallows are not crawled: they're never downloaded and read for more links. A link pointing to such a page is still checked once with a single HEAD request (GET only if HEAD fails), the same as a visitor clicking it, and marked robotsDisallowed: true. We chose this because robots.txt governs crawling, and a broken link to a disallowed page (a members area, a checkout step) is still a broken link on your site. If a robots.txt can't be read because of a server error or timeout, crawling that host is treated as disallowed (RFC 9309), and you'll see it in robotsTxtUnreachable. Rules are matched for the BrokenLinkChecker user-agent, then *, with * and $ wildcards and longest-match precedence. Crawl-delay is not used; politeness comes from the per-host limit.

What about rel="nofollow"?

It's recorded (nofollow on the row and on each foundOn entry) and the link is still checked, and crawled if it's internal. A broken nofollow link is still broken for visitors.

Why do rows appear only at the end?

A link's foundOn list keeps growing while the crawl finds more pages that link to it, so rows are written once the crawl has finished. While it runs, the crawl state lives on disk, not in memory, and it's saved to the key-value store regularly: if the platform restarts the run (for example a server migration), it continues from the last save, without charging any link twice and without writing rows twice (links checked after the last save may be checked again, free of charge). Near the run's timeout the crawl stops early (stoppedBy: "time limit") to leave time to write the rows.

Other limits

• Up to 10,000 crawled pages per run (Max pages to crawl); all links on them are checked. At most 5,000 links are read from one page and at most 250,000 unique links are kept per run. The default 256 MB of memory suits typical sites; give the run 1 GB for crawls with around 100,000 unique links.
• At most 2 MB of HTML is read per page; links further down a larger page aren't seen. Pages are requested with gzip/deflate only.
• The sitemap: /sitemap.xml plus Sitemap: lines in robots.txt, following sitemap index files, .xml.gz and plain-text sitemaps, at most 25 sitemap files (20 MB each), and at most Max pages URLs taken from them. Sitemap URLs are crawled like start URLs (depth 0).
• URLs with different query strings are different links (?utm_source=… variants are checked separately); use Ignore URLs matching to leave tracking variants out. #fragments are dropped, so page#a and page#b are one link.
• <base href> is honoured. <iframe>, <video>, CSS url() and <link rel="canonical">/hreflang targets aren't collected.
• No login: cookies are kept only within one link's redirect chain, never between pages, so pages behind a login show up as whatever the server returns to a visitor who isn't logged in.
• Private, loopback, link-local and cloud-metadata addresses are never requested (free rows with Blocked: private address), including when a public URL redirects to one. That also means intranet sites can't be crawled.
• HTTP/1.1 only. Response times are measured from Apify's servers, not your location.
• Certificates are verified like Python does, not like a browser: a site whose server leaves out an intermediate certificate (which some browsers fill in themselves) is reported with a TLS error.

Related Actors

Actor	Use it to
Bulk URL Status Checker	Check a list of URLs you already have (no crawl), with SSL expiry and page metadata.
PageSpeed & Lighthouse Audit	Audit the site's key pages for Core Web Vitals, SEO and accessibility, no API key.
Sitemap URL Extractor & New Page Monitor	List every URL in the site's sitemaps, to compare with what the crawl found.
Notify Step	Email or Slack the broken links after a scheduled run, only when there are some.
Google Sheets Import & Export	Send the broken links to a Google Sheet your team works from.
Dataset Filter & Transform	Keep only internal 404s, pick and rename columns, or flatten foundOn for a spreadsheet.