Dead Link Finder (Site Crawl): 404 and Redirect Report

Give it your home page. It crawls your website page by page, checks every link it finds, and returns one row per broken or redirecting link: the page it is on, the anchor text, the status code, the error type and how many pages link to it. Plain HTTP, robots.txt honoured, no browser, no login, no proxy needed.

What data you get

• Broken links: 404, 410, other 4xx, 5xx, DNS failures, timeouts, SSL certificate errors, connection errors
• Redirect problems: loops, chains longer than 3 hops, every hop listed; simple redirects optional
• Source page and anchor text, internal or external, pages linking to it, check method, response time, checked-at time

Use cases

• Monthly SEO hygiene: schedule a crawl and fix the 404s that leak link equity, most-linked first.
• Site migrations: run before and after a redesign or domain move to catch broken internal links and redirect chains.
• Agencies: hand clients a CSV of broken links with the exact page and anchor text to edit.
• Content teams: find dead outbound links in old blog posts before readers do.
• Automation: send new broken links to Slack, a ticket queue or a sheet with n8n, Make or Zapier.

What this actor does

1. Reads robots.txt for your site and follows its rules (Allow, Disallow, wildcards, Crawl-delay).
2. Opens up to maxPages HTML pages of your site and extracts every link (optionally images, scripts and stylesheets too).
3. Checks each unique URL once: a HEAD request first, then GET when a server refuses or mishandles HEAD. Redirects are followed by hand so every hop is recorded and loops are caught.
4. Writes one row per problem link, sorted errors first and then by how many pages link to it.

Key features

• Same-site crawl with www and non-www treated as one site; subdomains optional.
• External links checked too (one status request each, never crawled), with a per-host rate limit so a page with 200 links to one site never floods it.
• Polite by default: 3 requests per second to your site, 30 per minute per external host, 8 in parallel, retries with backoff on 429 (Retry-After honoured) and on timeouts.
• Fails fast: an invalid start URL ends the run in a second with a clear message and nothing charged.
• Cost caps you control: maxLinksChecked and Apify's maximum cost per run. Skipped links are never charged.

Input

Field	What it does	Default
startUrls	Pages to start from (one site each)	crawler-test.com demo
maxPages	HTML pages to open and read links from	25
maxLinksChecked	Cap on unique URLs checked (the billed count)	1000
checkExternalLinks	Also check links to other websites	true
includeSubdomains	Treat subdomains as the same site	false
checkResources	Also check images, scripts, stylesheets, media	false
excludeUrlPatterns	Skip URLs containing these texts (* wildcard)	none
internalRequestsPerSecond	Crawl speed on your site	3
externalRequestsPerMinutePerHost	Rate limit per external host	30
maxConcurrency	Requests in flight at once	8
requestTimeoutSecs	Timeout per request (one retry)	15
reportRedirects	all, chains_only or none	all
includeSummaryRow	Append a totals row to the dataset	false
proxyConfiguration	Optional proxy; off by default	off

Example input

{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "maxPages": 500,
  "maxLinksChecked": 5000,
  "checkExternalLinks": true,
  "excludeUrlPatterns": ["/wp-admin/", "?replytocom="],
  "reportRedirects": "chains_only"
}

Output

One row per broken or redirecting link. Healthy links are counted but not listed. The crawl summary (pages crawled, links checked, counts by error type, robots.txt notes, why a run stopped early) is saved as the OUTPUT record and, if you ask for it, as a final rowType: summary row.

Field	Meaning
linkUrl	The broken or redirecting URL
sourcePage	First page where it was found
anchorText	Link text (or image alt) on that page
statusCode	Final HTTP status (empty for DNS, timeout, SSL)
errorType	404, 410, 4xx, 5xx, forbidden, rate_limited, blocked, dns, timeout, ssl, connection_error, redirect_loop, invalid_redirect, invalid_url, redirect_chain, redirect
severity	error, warning or notice
finalUrl, redirectChain, redirectHops	Where redirects lead, every hop
linkType	internal or external
foundOnPageCount, foundOnPages	How many crawled pages link to it (up to 20 listed)
checkMethod, responseTimeMs, checkedAt	How and when it was checked

Sample output

{
  "rowType": "link",
  "linkUrl": "https://crawler-test.com/redirects/redirect_to_404",
  "sourcePage": "https://crawler-test.com/",
  "anchorText": "Redirect to 404",
  "statusCode": 404,
  "errorType": "404",
  "severity": "error",
  "errorMessage": "HTTP 404",
  "finalUrl": "https://crawler-test.com/status_codes/status_404",
  "redirectChain": ["301 https://crawler-test.com/redirects/redirect_to_404"],
  "redirectHops": 1,
  "linkType": "internal",
  "resourceType": "link",
  "foundOnPageCount": 1,
  "foundOnPages": ["https://crawler-test.com/"],
  "checkMethod": "GET",
  "responseTimeMs": 412,
  "checkedAt": "2026-11-08T06:00:01.000Z"
}

Pricing

Pay per event: $0.20 per 1,000 links checked ($0.0002 per unique URL whose status was checked, crawled pages included), plus Apify's $0.00005 start fee per run. You pay for the work done, not for the number of problems found, so a clean site costs the same as a broken one of the same size. Links skipped by robots.txt, your exclusions, the external-links switch or the link limit are never charged. Example: a 300-page site with 1,800 unique links costs about $0.36.

Use with the API, n8n, Make, Zapier or AI agents

API (one call, returns the rows as JSON):

curl -X POST "https://api.apify.com/v2/acts/transparent_meteorite~dead-link-finder-site-crawl/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://crawler-test.com/"}],"maxPages":1,"maxLinksChecked":450,"checkExternalLinks":true}'

The call waits up to 5 minutes. For bigger inputs, start the run with /runs instead and read the dataset when it finishes.

Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("transparent_meteorite/dead-link-finder-site-crawl").call(run_input={"startUrls": [{"url": "https://crawler-test.com/"}], "maxPages": 1, "maxLinksChecked": 450, "checkExternalLinks": True})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)

• n8n, Make, Zapier: use the Apify app's "Run Actor" step with actor transparent_meteorite/dead-link-finder-site-crawl, paste the input JSON, then map the dataset items to your sheet or CRM.
• Clay: add an HTTP API column that calls the run-sync endpoint above.
• Schedules: in Apify Console, Schedules, run it daily or weekly with the same input for a steady feed.
• AI agents (MCP): connect the Apify MCP server at https://mcp.apify.com?tools=transparent_meteorite/dead-link-finder-site-crawl to Claude, Cursor or any MCP client, and your agent can run Dead Link Finder (Site Crawl) on request.

FAQ

Why HEAD and then GET? HEAD is fast and light. Some servers answer HEAD with 403, 404 or 405 while the page works fine, so any error on HEAD is confirmed with a GET before it is reported.

Does it run JavaScript? No. It reads the HTML your server sends, which is what search engines see first. Links injected only by JavaScript are not found.

Why does LinkedIn, X or a shop show as forbidden or blocked? Some sites refuse automated requests (403, 429 or 999). These are reported as warnings, not errors, because the link usually works in a browser.

Does it respect robots.txt? Yes, for every page on your own site. External links get one status request each and are never crawled.

Can I schedule it? Yes. Use Apify schedules and a webhook or integration to send new rows to Slack, email or a sheet.

Limits

• Pages are read as HTML only (no JavaScript rendering); single-page apps that build links in the browser will show few links.
• A redirect chain longer than 10 hops is reported as a redirect loop.
• A site whose robots.txt returns a server error is not crawled (only the start URL is checked), as robots.txt rules require.
• Very slow hosts can stop a run early to keep costs sane; the summary says so and names the reason.
• Up to 100 start URLs, 10,000 pages and 100,000 links per run.

Related actors

• Sitemap Link Inventory: every URL in a site's sitemaps, ready to compare with what the crawl finds.
• Technical SEO Page Audit: titles, meta tags, canonicals and headings per page.
• Wayback Page Change Tracker: find what a dead page used to say, to choose a redirect target.

Support

Found a bug or need a field? Open an issue on the Issues tab with the run URL and we will reply there. If the actor saved you time, a short honest review on the Reviews tab helps other people judge it.