Broken Link Checker & Internal Link Graph (Fast, No Browser)

Crawl any website and find broken links, broken images, redirect chains, insecure http:// links and links to noindex pages. You also get an internal link graph for every crawled page: inlinks, internal and external outlinks, title, canonical, noindex and orphan-page candidates from the sitemap.

It crawls over plain HTTP (async httpx, no headless browser), so it is fast and cheap. A 10-page crawl that checks 330+ unique links finishes in about 10 seconds.

What it does

1. Starts from one or more start URLs, reads robots.txt (respects Disallow and Crawl-delay) and the sitemap.
2. Crawls internal pages breadth-first up to Max pages and Max depth, with polite concurrency.
3. Parses every <a href>, <img src>, <link rel="canonical">, <meta name="robots"> and <base href>.
4. Checks every unique link target once: HEAD first, GET fallback when a server rejects HEAD. Redirects are followed by hand, so the full redirect chain is recorded. External hosts are rate-limited per host.
5. Writes three kinds of rows (rowType):
   • page: one row per crawled page, with url, statusCode, depth, title, h1, canonical, canonicalIsSelf, noindex, inlinks, outlinksInternal, outlinksExternal, imagesOnPage, brokenLinksOnPage, inSitemap, isOrphanCandidate.
   • link: problems only, one row per problem link on a page, with sourcePage, targetUrl, anchorText, linkType (link/image), statusCode, redirectChain, problem and problems.
   • summary: one final row with pages crawled, links checked, broken links by type and the top 10 pages with the most broken links.

Problems detected

problem	Meaning
broken_4xx / broken_5xx	Target answered 404, 410, 500, 503 and so on
timeout	No answer within the timeout
dns_error	Domain does not resolve (expired or mistyped domain)
connection_error / ssl_error / too_many_redirects	Connection refused, bad certificate, redirect loop
redirect_chain	2 or more redirect hops before the final URL
redirect_to_other_host	Link redirects to a different domain (www and non-www count as the same domain)
http_link_on_https_page	Insecure http:// link or image on an https:// page (mixed content)
links_to_noindex	Link points to a page marked noindex (meta robots or X-Robots-Tag)

Who it's for

• SEO specialists and agencies: regular broken-link audits, redirect cleanup and internal-linking analysis for clients.
• Webmasters and shop owners (Shopify, WooCommerce, WordPress) who want to find 404s before Google and customers do.
• Content teams after a migration or URL change.
• Developers: add a link check to CI/CD or a scheduled monitor through the API.
• AI agents: flat JSON rows are easy to read and act on ("fix the 12 broken links on /blog/").

Why use it instead of the alternatives

• Cheap: $0.001 per crawled page, and every link on that page is checked at no extra cost. Comparable audit actors charge up to $0.04 per page.
• Fast: no browser. Async HTTP, link targets deduplicated across the whole site, HEAD requests where possible.
• Problem rows only: you don't get thousands of "200 OK" rows to filter. Healthy links are counted in the summary.
• Link graph included: inlinks and outlinks per page, plus orphan candidates (in the sitemap but not linked internally). Most link checkers don't return these.
• Polite: respects robots.txt and Crawl-delay, rate-limits each external host, and sends an identifying User-Agent.

Input example

{
  "startUrls": ["https://books.toscrape.com/"],
  "maxPages": 10,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "checkExternalLinks": true,
  "checkImages": true,
  "respectRobotsTxt": true,
  "excludePatterns": ["\\?sort=", "/tag/"]
}

Other options: includePatterns (regex), includeSitemapUrls (also crawl sitemap URLs, for real orphan-page detection), checkUncrawledInternalLinks, maxConcurrency (default 5), maxRequestsPerExternalHost (default 2), maxLinkChecks, requestTimeoutSecs, userAgent and proxyConfiguration.

Output example

These values come from a real run on https://books.toscrape.com/ (10 pages, 333 unique URLs checked, 10.4 s):

{
  "rowType": "page",
  "url": "https://books.toscrape.com/",
  "statusCode": 200,
  "status": "ok",
  "depth": 0,
  "title": "All products | Books to Scrape - Sandbox",
  "h1": "All products",
  "canonical": null,
  "noindex": false,
  "inlinks": 0,
  "outlinksInternal": 73,
  "outlinksExternal": 0,
  "imagesOnPage": 20,
  "brokenLinksOnPage": 0,
  "inSitemap": null,
  "isOrphanCandidate": false
}

{
  "rowType": "summary",
  "pagesCrawled": 10,
  "pagesOk": 10,
  "uniqueLinksChecked": 333,
  "linkOccurrences": 1087,
  "brokenLinks": 0,
  "broken4xx": 0,
  "broken5xx": 0,
  "timeouts": 0,
  "dnsErrors": 0,
  "redirectChains": 0,
  "topPagesByBrokenLinks": [],
  "robotsTxtFound": false,
  "sitemapFound": false,
  "crawlComplete": false,
  "durationSecs": 5.5
}

The demo site has no broken links. On a site that has some, a link row looks like this:

{
  "rowType": "link",
  "problem": "redirect_chain",
  "problems": ["redirect_chain"],
  "sourcePage": "https://example.com/",
  "targetUrl": "https://example.com/r1",
  "linkType": "link",
  "isInternal": true,
  "anchorText": "Old pricing page",
  "statusCode": 200,
  "finalUrl": "https://example.com/pricing",
  "redirectHops": 2,
  "redirectChain": "https://example.com/r1 [301] -> https://example.com/r2 [301] -> https://example.com/pricing [200]",
  "error": null
}

The dataset has three views: Overview, Link problems and Pages & internal link graph. The summary is also stored as OUTPUT in the key-value store.

Pricing

Pay per event: $0.001 per crawled page ($1 per 1,000 pages). Checking links and images, the summary and pages that fail (network errors, 4xx and 5xx pages) are free. Platform usage is minimal: the 10-page demo run used about $0.0001 of compute at 512 MB.

Usage via API, n8n, Make, MCP

API (run synchronously and get the dataset items):

curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~broken-link-internal-link-checker/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://your-site.com/"],"maxPages":200}'

Python:

from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
run = client.actor("fast_data_asap/broken-link-internal-link-checker").call(run_input={"startUrls": ["https://your-site.com/"], "maxPages": 200})
problems = [i for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i["rowType"] == "link"]

n8n / Make / Zapier: use the Apify node or app, choose this actor, then filter rows by rowType = link and send them to Slack, Sheets or your ticket system. Add a schedule in Apify to monitor your site weekly.

MCP (Claude, Cursor and other AI agents): add https://mcp.apify.com?tools=fast_data_asap/broken-link-internal-link-checker as an MCP server.

Notes and limits

• The crawler only follows links in the HTML. Links that JavaScript adds after page load are not seen.
• Some sites block bots or rate-limit HEAD requests. The checker falls back to GET, but a few sites still answer 403 or 429 to any automated client. Treat such results as "check manually".
• Only public pages are crawled. No logins, and no captcha or bot-protection bypass.