Broken Link Checker Scraper

Run a sitewide broken link checker against one or more public websites. The Actor crawls bounded same-origin pages, tests each deduplicated internal and external link, follows redirects, and exports a typed audit dataset with HTTP status, redirect chain, source page, anchor text, response time, and bounded error details.

Use it after content changes, redesigns, migrations, or deployments to find link rot before visitors and search engines do.

What does Broken Link Checker Scraper do?

For every supplied website, the Actor:

1. validates that the start URL resolves to the public internet;
2. fetches server-rendered HTML pages on the same origin;
3. discovers and normalizes HTTP and HTTPS links;
4. deduplicates targets while preserving source-page evidence;
5. reuses crawl responses for internal links already fetched;
6. tests other targets with HEAD and falls back to GET on 405/501;
7. records every redirect hop;
8. classifies each result as ok, redirect, broken, or warning;
9. writes one dataset row per tested target.

Successful links are included alongside failures. Filter resultType = broken for actionable dead links, or retain all rows as a release-quality baseline.

Who is this website link audit for?

• SEO teams checking 404s, redirect chains, and external link rot.
• Web developers validating navigation after a deployment.
• Content teams finding stale references across documentation or blogs.
• QA engineers producing machine-readable link health evidence.
• Agencies running the same bounded audit for multiple client sites.
• Data teams loading recurring audit datasets into a warehouse or spreadsheet.

This Actor is designed for anonymously reachable public pages. It does not accept accounts, cookies, or private-network URLs.

Why use this broken link check?

A browser extension checks only the page you open. This Actor performs a bounded site crawl and produces reusable data through the Apify Console, API, schedules, webhooks, datasets, and integrations.

Useful details include:

• the first page and anchor text where a target appeared;
• up to 20 distinct source pages per target;
• total occurrence count from the bounded crawl;
• final status and URL after redirects;
• every observed 301, 302, 303, 307, or 308 hop;
• internal versus external classification;
• bounded timeout, DNS, TLS, connection, and URL-safety errors;
• response time and final request method.

What data does the Actor export?

Field	Meaning
targetUrl	Deduplicated URL discovered in an anchor element
finalUrl	Last URL reached after redirects, or null on a network error
sourcePage	First crawled page containing the target
sourcePages	Up to 20 crawled pages containing the target
anchorText	Normalized text from the first anchor occurrence
occurrenceCount	Number of occurrences found during the bounded crawl
linkType	internal or external relative to the website root
statusCode	Final HTTP status, or null if no response arrived
statusText	HTTP status text
redirectChain	Ordered redirect URL, status, and destination objects
redirectCount	Number of observed redirects
isRedirect	Whether the target redirected
isBroken	Whether the target has an HTTP or confirmed network failure
resultType	ok, redirect, broken, or warning
errorType	Bounded network/safety error category
errorMessage	Diagnostic message capped at 500 characters
responseTimeMs	Total request and redirect time
method	HEAD or GET used for the result
checkedAt	ISO 8601 audit timestamp

Fields that depend on an HTTP response are nullable because DNS, TLS, connection, and timeout failures may not produce a status.

How to run your first sitewide link audit

1. Open the Actor input page.
2. Add a public website URL under Websites to audit.
3. Keep maxPages and maxLinks small for the first run.
4. Leave external checking enabled if off-site references matter.
5. Click Start.
6. Open the Link audit dataset view.
7. Filter resultType to broken, redirect, or warning.
8. Export the result as CSV, JSON, Excel, XML, or RSS.

A useful smoke-test input is:

{
  "startUrls": [
    { "url": "https://crawler-test.com/links/broken_links_internal" }
  ],
  "maxPages": 6,
  "maxLinks": 40,
  "checkExternalLinks": false,
  "maxConcurrency": 6
}

Input parameters

startUrls

Required array of 1–20 public HTTP(S) website roots. The Actor follows same-origin links from each root. URL credentials, localhost, and private/reserved network destinations are rejected.

maxPages

Maximum total same-origin HTML pages crawled across all roots. Default: 5. Range: 1–1000.

maxLinks

Maximum unique targets tested and exported. Default: 20. Range: 1–5000.

checkExternalLinks

When true (default), test off-origin links too. When false, export only internal targets.

maxConcurrency

Concurrent crawl/check requests. Default: 10. Range: 1–30. Reduce it for small or rate-sensitive websites.

requestTimeoutSecs

Per-request timeout. Default: 15 seconds. Range: 3–60.

maxRedirects

Maximum redirect hops for each page or link. Default: 8. Range: 0–15.

Example broken-link output

This representative record comes from the current crawler-test workflow:

{
  "targetUrl": "https://crawler-test.com/links/not_found/foo1",
  "finalUrl": "https://crawler-test.com/links/not_found/foo1",
  "sourcePage": "https://crawler-test.com/links/broken_links_internal",
  "sourcePages": [
    "https://crawler-test.com/links/broken_links_internal"
  ],
  "anchorText": "Broken Internal Link 1",
  "occurrenceCount": 1,
  "linkType": "internal",
  "statusCode": 404,
  "statusText": "Not Found",
  "redirectChain": [],
  "redirectCount": 0,
  "isRedirect": false,
  "isBroken": true,
  "resultType": "broken",
  "errorType": null,
  "errorMessage": null,
  "responseTimeMs": 113,
  "method": "GET",
  "checkedAt": "2026-08-25T06:07:55.593Z"
}

Response times and timestamps vary by run.

How result classifications work

• ok — a final response below HTTP 400 with no redirect.
• redirect — one or more redirect hops ended in a non-broken response.
• broken — HTTP 400+ or a confirmed DNS, TLS, connection, or unsafe-destination failure.
• warning — timeout or unknown request failure that does not confirm the target is permanently broken.

A 401 or 403 is classified as broken from the link-audit perspective because anonymous visitors cannot reach it. Review authorization-sensitive targets before removing them.

Redirect and deduplication behavior

URL fragments are removed because they do not change the HTTP target. Query strings are preserved because they may produce different content or status. Each normalized target is tested once per run.

The Actor records the first anchor text, the first source page, up to 20 unique source pages, and a full occurrence count. This keeps records bounded without hiding how widely a link is used.

Redirect destinations are public-IP validated before each hop. Redirect loops or chains beyond maxRedirects become bounded warnings.

How much does it cost to check website links?

The Actor uses pay-per-event pricing:

• one Start event per run;
• one Item processed event per tested, exported link target.

At the BRONZE rate of $0.005 per start and $0.0008 per link, examples are:

Exported links	Estimated BRONZE calculation
10	0.005 + (10 × 0.0008) = 0.013 USD
100	0.005 + (100 × 0.0008) = 0.085 USD
500	0.005 + (500 × 0.0008) = 0.405 USD
1,000	0.005 + (1,000 × 0.0008) = 0.805 USD

Apify plan tier and platform usage treatment can affect the final invoice. The Console displays the applicable event prices before each run. Set maxLinks to create a hard output and event bound.

Recurring website QA workflow

Create an Apify Schedule that runs after a daily content publish or weekly maintenance window. Send the dataset to Google Sheets, Slack, a webhook, or a warehouse. Compare rows by targetUrl, statusCode, and resultType to detect new failures.

The Actor creates a fresh default dataset per run. It does not maintain historical diffs or send alerts itself; use Apify schedules, webhooks, or your downstream system for those steps.

Website migration workflow

Before migration, save a baseline audit. After cutover:

1. run the same input against the new site;
2. filter for broken internal links;
3. inspect redirect chains for legacy paths;
4. group by sourcePage to assign fixes;
5. compare the new dataset against the baseline.

This catches navigation errors and incomplete redirects without requiring a browser session.

cURL API example

Replace APIFY_TOKEN with your token:

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~sitewide-broken-link-checker/runs?token=APIFY_TOKEN&waitForFinish=300" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://docs.apify.com/"}],
    "maxPages": 10,
    "maxLinks": 100,
    "checkExternalLinks": true
  }'

Fetch dataset items from the defaultDatasetId returned by the run API.

JavaScript API example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/sitewide-broken-link-checker').call({
    startUrls: [{ url: 'https://docs.apify.com/' }],
    maxPages: 10,
    maxLinks: 100,
    checkExternalLinks: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const broken = items.filter((item) => item.resultType === 'broken');
console.log(broken);

Python API example

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/sitewide-broken-link-checker').call(run_input={
    'startUrls': [{'url': 'https://docs.apify.com/'}],
    'maxPages': 10,
    'maxLinks': 100,
    'checkExternalLinks': True,
})

items = client.dataset(run['defaultDatasetId']).list_items().items
broken = [item for item in items if item['resultType'] == 'broken']
print(broken)

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/sitewide-broken-link-checker"

Claude Desktop setup

Add this server entry to the Claude Desktop MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/sitewide-broken-link-checker"
    }
  }
}

Cursor setup

Open Cursor MCP settings, add the same remote apify server URL, and enable the server for Agent mode.

VS Code setup

Add the same apify URL to the VS Code MCP configuration, start it from the MCP view, and confirm this Actor appears in the available tools.

Example prompts:

• “Audit the first 20 pages of this public documentation site and list broken internal links.”
• “Check external references, then group failures by source page.”
• “Show redirects with more than one hop and their final status.”

Performance and cost tips

• Begin with 5–25 pages and 50–250 links.
• Disable external checking when only site navigation matters.
• Lower concurrency when a server starts returning 429 or 503.
• Increase timeout for slow but legitimate sites rather than treating a timeout as broken.
• Audit a focused section URL instead of the homepage when only one documentation area changed.
• Use repeated inputs and downstream comparisons for consistent regression checks.

Limitations

• Only links present in server-rendered <a href> elements are discovered.
• JavaScript-generated links and browser-only navigation are not rendered.
• Authenticated pages, accounts, cookies, and login flows are unsupported.
• Private-network, localhost, credential-bearing, and non-HTTP(S) targets are blocked.
• robots.txt is not an access-control mechanism; users remain responsible for crawl permission and rate choices.
• Some servers reject automated HEAD and GET requests or return bot-specific statuses.
• Fragment-only failures inside a page are not checked because URL fragments are removed.
• The first 20 unique source pages are retained per target; occurrenceCount may be higher.
• A timeout is a warning, not proof that a link is permanently dead.

Troubleshooting

The dataset is empty

Confirm the start page returns server-rendered HTML with ordinary anchor elements. An empty dataset means no supported HTTP(S) links were discovered under the selected scope. Check the run log for fetch or public-IP validation errors.

Many links return 403 or 429

Lower maxConcurrency and rerun a small sample. The Actor intentionally has no automatic residential-proxy fallback, so it cannot bypass a site that blocks anonymous direct requests.

An internal link appears external

Classification uses exact URL origin (scheme, hostname, and port). A link from www to a bare domain, or from HTTP to HTTPS, is external unless the page redirect already changed the audited root origin.

A working link is marked broken

Check whether it requires login, geography, browser JavaScript, cookies, or a specific user agent. Review 401/403 results manually. Increase the timeout only for timeout warnings.

Responsible use and legality

Use the Actor only on public pages you are authorized to request. Respect site terms, applicable robots guidance, crawl-rate expectations, copyright, privacy, and computer-access laws. Do not use it to probe internal infrastructure or evade access controls. The public-IP validation is a safety boundary, not legal permission.

Dataset URLs and anchor text may contain personal or confidential information published by the target site. Apply appropriate retention and access controls to exports.

Related Automation Lab Actors

• Redirect Chain Analyzer for a focused list of known URLs.
• Robots & Sitemap Analyzer for crawl directives and sitemap structure.
• Lighthouse Website Audit for performance, accessibility, best-practice, and SEO audits.

Choose this Actor when you need source-page and anchor context from a bounded site crawl. Choose Redirect Chain Analyzer when you already have the exact URL list and do not need link discovery.

FAQ

Does it check both internal and external links?

Yes. External checking is enabled by default and can be disabled with checkExternalLinks.

Does it report healthy links?

Yes. One row is exported for every tested target, making the dataset suitable as a complete bounded audit baseline.

Can I audit several websites in one run?

Yes, up to 20 roots. maxPages and maxLinks are total run limits shared across them.

Does it follow redirects?

Yes, up to maxRedirects, with each hop stored in redirectChain.

Does it use a browser or proxy?

No. It uses direct HTTP and server-rendered HTML to keep runs lightweight and predictable.

Can it monitor changes automatically?

Schedule recurring Actor runs and compare datasets downstream. The Actor does not maintain history or send alerts by itself.

Is a timeout definitely a broken link?

No. Timeouts and unknown failures are warning results. Retry them separately before making a content decision.