Broken Link & Redirect Checker

What does this broken link checker do?

Broken Link & Redirect Checker crawls your website, collects every link it finds, and checks each one. For every link it reports the HTTP status, the full redirect chain (each hop, permanent or temporary), redirect loops, likely soft-404s, and the pages on your site that link to it, so you know exactly where to fix it. It can also seed pages from your XML sitemap, so pages with no inbound links get checked too.

Who is it for?

SEO teams cleaning up 404s and redirect chains, site owners after a migration or redesign, agencies running link audits, and AI agents that need a structured list of a site's link problems.

How it stays polite

• It only crawls the sites you list. Links to other sites are checked once each and never crawled.
• About 1 request per second per host (longer if robots.txt sets a Crawl-delay), and it backs off on HTTP 429 and Retry-After. A host that answers 429 three times in a row gets no further requests in that run.
• robots.txt is respected. Blocked links are reported as blocked-by-robots and aren't charged. If a host's robots.txt can't be reached or returns a server error, the whole host is treated as off-limits (RFC 9309), so its links are blocked-by-robots too. For your own sites only, ownSiteConfirmed lets you turn robots.txt off or go faster; external hosts always keep the 1-second limit and their robots.txt.
• It stores no page content, only the status codes and the redirect information. mailto: and tel: links are skipped, so no email addresses or phone numbers end up in the results.

Input

Field	Default	What it does
urls (required)	prefill crawlee.dev	Your site(s): domains or start pages.
seedFromSitemap	true	Also crawl pages listed in the site's XML sitemap.
maxPagesPerSite	50 (prefill 5)	Pages per site fetched and scanned for links.
maxLinks	1000 (prefill 100)	Cap on unique links checked. You pay per link, so this also caps cost.
checkExternalLinks	true	Check outbound links too (one request each).
checkAssets	false	Also check images, scripts, stylesheets and icons.
includeRegex	none	Report only links that match, e.g. /docs/. Pages are still crawled.
excludeRegex	none	Don't crawl or check URLs that match.
maxRedirects	10	Hops followed before a link counts as a loop.
requestDelayMs	1000	Delay per host. Under 1000 ms applies only to your own sites with ownSiteConfirmed.
maxConcurrency	5	Sites or hosts processed at once. Each host stays sequential.
respectRobotsTxt	true	Can be turned off only for your own sites, with ownSiteConfirmed.
ownSiteConfirmed	false	Tick only for sites you own.

Output

One item per unique link. category is one of ok, broken, redirect, redirect-chain, loop, soft-404, blocked-by-site, rate-limited, blocked-by-robots, blocked-private, timeout or dns-error.

• blocked-by-site: the link answered HTTP 401, 403 or 999, so the site refused the automated check (bot protection or a login wall). That isn't evidence the page is gone, so it's free and not counted as broken.
• rate-limited: the link answered HTTP 429 after retries, or wasn't requested because its host had answered 429 three times in a row. Also free and not counted as broken.
• blocked-private: the link points at, or redirects to, a localhost or private network address. It's never requested and never charged.

Real item from a local test run on the default prefill (1 Nov 2026), with the redirect chain and timestamp trimmed:

{
  "url": "https://stackoverflow.com/questions/tagged/crawlee",
  "category": "blocked-by-site",
  "ok": false,
  "status": 403,
  "hops": 0,
  "redirectType": null,
  "error": "HTTP 403: the site refused the automated check (bot protection or login required); not a dead link as far as we know, open it in a browser",
  "isExternal": true,
  "linkType": "a",
  "anchorText": "Stack Overflow",
  "sourcePages": ["https://crawlee.dev/", "https://crawlee.dev/blog"],
  "sourcePageCount": 5,
  "method": "GET",
  "chargeable": false
}

Each run also saves a Markdown summary (SUMMARY.md) of the problems found, with a "Check manually" list of the blocked-by-site and rate-limited links to open in a browser.

How long does it take?

On the default prefill (5 pages, 100 links on crawlee.dev), a cloud run took about 58 seconds and checked 97 links. Because each host gets about 1 request per second, a big single-site check takes longer: 1,000 links on one host is roughly 17 minutes. Raise the run timeout or lower maxLinks to match. The run stops cleanly before its timeout and keeps the results it has.

How much does it cost?

Pay per event: one link-checked event per unique link with a definitive result (a status, a redirect chain, a loop, a soft-404 or a DNS failure). These are free: timeouts, rate-limited links (429), links the site refused (401, 403 or 999, blocked-by-site), robots-blocked links and links to private addresses. Price: $0.80 per 1,000 links checked ($0.0008 per link-checked event). maxLinks and your maximum charge per run both cap the total.

Use with AI agents

Designed for limited permissions so it's callable through the Apify MCP server. Example: "Check crawlee.dev for broken links and tell me which pages link to them."

OUTPUT_SUMMARY stop flags

Each run also writes OUTPUT_SUMMARY to the default key-value store (alongside SUMMARY.md). Monitors should read it before treating unchecked pages' links as fixed. Boolean stop flags (real field names):

Flag	Meaning
capHit	maxLinks was reached with links still unchecked.
pageHit	maxPagesPerSite was reached with same-site URLs still queued.
budgetHit	The run's charge budget (maxTotalChargeUsd) was reached.
timeHit	The run stopped early because it was near the time limit.

linksNotChecked is the count of found links that were not checked because a stop flag fired. A missing flag (older builds) should be treated as unknown, not as "did not hit".

Dead internal links count as crawled pages. Same-site URLs (including ones that later answer 404) are queued as pages under maxPagesPerSite, so an internal 404 still consumes a page slot and can set pageHit.

Limits

• A 401, 403 or 999 doesn't mean the link is dead. Many sites block automated requests. These links are reported as blocked-by-site, aren't counted as broken and aren't charged. Check them in a browser before removing them. The checker never tries to get past the block.
• Rate-limited links aren't checked. A 429 after retries, or a host paused after three 429s in a row, gives rate-limited (free). Check those links in a browser or re-run later.
• Soft-404 detection is a heuristic. It flags 200 pages that say "not found" and deep URLs redirected to the homepage, and only for pages it fetched in full.
• Results cover the pages and links within your caps, not necessarily the whole site.
• It never logs in or gets past bot protection or CAPTCHAs.

FAQ

Does it check links to other websites? Yes, once each, unless you turn off checkExternalLinks. It never crawls other sites. What counts as a redirect chain? More than one redirect hop before the final page. redirectType shows whether the hops were permanent, temporary or mixed.

Legal

Checks public links only. Stores status codes and redirect information, not page content or personal data.

Changelog

See CHANGELOG.md.