Broken Link Checker: Crawl a Site, Find 404s & Redirects

Crawl a website over plain HTTP and check every unique link once: broken links (404, 410, 5xx, DNS errors), redirect chains and loops, timeouts, and links the server refuses. One dataset row per checked link, with the pages it was found on and the anchor text, plus a REPORT summary. No browser, no proxy, no key.

Price: $0.40 per 1,000 links checked ($0.0004 per row), plus Apify's default Actor-start event. Crawled pages are not charged on their own: you pay for output rows only.

• Whole-site crawl: same-site pages are read for a href links (optionally img src, script src, link href); each unique target is checked once, however many pages link to it.
• Redirects spelled out: up to 5 redirects followed, every hop recorded (redirectChain), loops flagged.
• Where to fix it: up to 20 foundOn pages per link, with anchor text.
• External links too (on by default): checked, never crawled. Rate-limited per host, external hosts included.
• Sitemap coverage (optional): give a sitemap and the report lists sitemap URLs no page links to, and crawled pages missing from the sitemap.
• Quick start: press Start. The prefilled input crawls one page of https://quotes.toscrape.com/tag/simile/ (a public scraping sandbox) and checks its links: about 23 rows, under 30 seconds, about $0.009.

Plain HTTP only. Pages are fetched without running JavaScript, so links created by scripts (single-page apps) are not seen. robots.txt is obeyed, requests are rate-limited, and a block (403, 429, challenge page) is reported as a free blocked row, never worked around.

What it does

1. Starts from your start URLs and/or the URLs in your sitemaps.
2. Fetches same-site pages (same host, www. ignored; subdomains count as other sites) up to Max pages, and reads their links.
3. Checks every unique link target once: HEAD first, GET if the server answers 405, 403 or 501 (or the HEAD request fails). Each request (every redirect hop too) has its own 15 s limit, counted after its rate-limit wait.
4. Writes rows as checks finish, in batches (every 50 rows or 30 s), and saves a checkpoint after each batch. foundOn in a row lists up to 20 pages known when the row was written; foundOnCount is the true number. The LINK_SOURCES checkpoint record keeps the first 5 pages per link (it is the resume state, not a full list). A REPORT is saved at every checkpoint (marked partial: true) and finally.

Links with mailto:, tel:, javascript:, data: and fragment-only links are skipped, fragments are dropped (/a#x and /a are one link). Social-media and Google hosts (YouTube, Facebook, Instagram, X, TikTok, LinkedIn, Reddit, Google domains and similar) are never contacted: such links get a free blocked row with errorCode: "platform".

Input

Field	What it does
Start URLs (startUrls)	Pages to start from. Only the same site is crawled.
Sitemap URLs (sitemapUrls)	Optional. Index files and .xml.gz work (up to 25 files). Their URLs are checked and coverage is reported. Works without start URLs.
Max pages to crawl (maxPages)	Pages fetched and read for links. Default 100, hard cap 10,000. Further same-site links are still checked, just not read.
Max unique links (maxLinks)	Default 5,000, hard cap 20,000. Extra links are ignored (the report says so).
Check external links (checkExternal)	Default on. Off: links to other sites are ignored and not output.
Include images, scripts and stylesheets (includeResources)	Default off. On: also img src, script src, link href.
Only output problems (onlyProblems)	Default off. On: only non-ok rows are output and charged. See the note below.
Allow slow hosts (allowSlowHosts)	Default off. A site you crawl can ask in robots.txt for a Crawl-delay between requests. The Actor always waits the full delay and never shortens it. Off: when the delay is over 2 seconds the site's links are not requested and come back as free slow-host rows. On: delays of 2 to 2.8 seconds are honoured in full (the run gets long and compute is billed while it waits, see Pricing); over 2.8 seconds the site is always skipped (free slow-host rows, "over the 2.8 s maximum this Actor honours").
Requests per second per host (requestsPerSecondPerHost)	Default 1, maximum 4. A larger Crawl-delay in robots.txt on a site you crawl wins and is never shortened (see Allow slow hosts).
Max parallel requests (maxConcurrency)	Global cap, default 10, maximum 20.
Ignore query parameters (ignoreQueryParams)	Parameter names (for example utm_source, ref) that are ignored when de-duplicating: variants that differ only in these count as one link. Session IDs in jsessionid (path or query) and PHPSESSID are always ignored the same way. sid is not ignored by default (many sites use it as a content ID); add it here if it is a session ID on your site. The URL that is requested and output is always the first one found, with its parameters as they were.
Max run time (minutes) (maxDurationMinutes)	Default 60, maximum 360. The run stops cleanly and saves what it has. Set the run timeout in the run options to this plus 5 minutes.

robots.txt and external links. robots.txt (rules and Crawl-delay) is read for every site the Actor contacts: your start URLs, sitemap hosts, every external link target and every redirect hop to a new site, once per run per site (user-agent token broken-link-checker-actor, then *; one retry; a 4xx answer means no rules). A link that robots.txt disallows is not requested and comes back as a free blocked row (errorCode: "robots"); a robots.txt that cannot be read after one retry (server error, 429 or unreachable) gives a free blocked row with errorCode: "robots-unreachable". A Crawl-delay on an external site is treated exactly as on your own site: up to 2 s is honoured, over 2 s makes the link a free slow-host row unless Allow slow hosts is on (then honoured in full up to 2.8 s), over 2.8 s is always skipped. Reading robots.txt of other sites is unpaid work: it counts toward the unpaid-work cap, and a run reads at most 500 different other sites (REPORT limits.externalRobotsHostsRead, externalRobotsHostCap, externalHostCapReached); links to further new sites are free robots-host-cap rows. Nothing is crawled on external sites: one HEAD (or GET) per unique URL at no more than requestsPerSecondPerHost per host.

Speed is set by the rate limit. A single host gets at most requestsPerSecondPerHost requests per second, so 100 pages with their own-site links take several minutes at the default of 1.

Only output problems: the Actor still has to request every link, so on a healthy site it does a lot of unpaid work. To protect against that, the run stops (stoppedBy: "unpaid-work-cap") after 1,000 requests that produced no charged row (blocked links count here too). Everything found up to then is saved.

Example inputs

1. Quick check of one page and what it links to:

{ "startUrls": ["https://quotes.toscrape.com/tag/simile/"], "maxPages": 1 }

2. Whole small site, problems only, gentle on the server:

{
  "startUrls": ["https://www.example.org/"],
  "maxPages": 300,
  "onlyProblems": true,
  "requestsPerSecondPerHost": 0.5
}

3. Sitemap audit, internal links only, including images and scripts:

{
  "sitemapUrls": ["https://www.example.org/sitemap.xml"],
  "maxPages": 500,
  "checkExternal": false,
  "includeResources": true
}

Output

One dataset row per unique checked link:

{
  "url": "https://www.example.org/old-page",
  "status": "redirect",
  "statusCode": 200,
  "finalUrl": "https://www.example.org/new-page",
  "redirectCount": 2,
  "redirectChain": [
    { "url": "https://www.example.org/old-page", "statusCode": 301 },
    { "url": "https://www.example.org/interim", "statusCode": 302 },
    { "url": "https://www.example.org/new-page", "statusCode": 200 }
  ],
  "redirectLoop": false,
  "contentType": "text/html",
  "responseTimeMs": 212,
  "error": null,
  "errorCode": null,
  "external": false,
  "startUrl": false,
  "foundOnCount": 14,
  "foundOn": [
    { "pageUrl": "https://www.example.org/", "text": "Our services", "tag": "a" }
  ],
  "checkedAt": "2026-10-20T08:00:00.000Z"
}

inSitemap is added when you gave sitemap URLs. Start URLs and crawled pages are rows too (a broken start URL shows up as a broken row).

Status values

status	Meaning
ok	Final response 2xx, no redirect.
redirect	Reached a 2xx after one or more redirects (http to https, trailing slash, moved pages).
broken	4xx (404, 410 and so on), 5xx, DNS failure, connection refused or reset, TLS error, redirect loop, more than 5 redirects.
timeout	No answer within 15 s.
blocked	The server refused us (401, 403, 429, 999; not retried and not worked around), or the link was not requested: disallowed by robots.txt (your site or any other site), robots.txt unreadable (robots-unreachable), the per-run limit of other sites read (robots-host-cap), a slow host (slow-host: robots.txt asks for a Crawl-delay over 2 s and Allow slow hosts is off, or over the 2.8 s maximum this Actor honours), a platform host, or a private address. Free: blocked rows are never charged.

errorCode: "internal" (status broken) is an unexpected error in this Actor, not a problem with your link. Free: such rows are never charged, and each one counts at least 1 toward the unpaid-work cap.

errorCode values: http-404 (and any other http-NNN), timeout, dns, connection-refused, connection-reset, tls, redirect-loop, too-many-redirects, bad-redirect, invalid-url, robots, robots-unreachable, robots-host-cap, slow-host, platform, ssrf, network, internal.

REPORT (key-value store record)

Counts by status, by HTTP status code and by error code; topBroken (the 20 broken or timed-out links found on most pages); redirectLoops; pagesCrawled; rows output and charged; why the run stopped; and, with sitemaps, sitemapCoverage: URLs in the sitemap that no crawled page links to, crawled pages that are not in the sitemap, and sitemap URLs that are broken. Those two lists are only complete when complete is true (the crawl covered every reachable page); the report says when they are not. RUN_SUMMARY has charged events, requests, bytes read and failures by code.

Pricing

Pay per event: $0.0004 per link row output (event link-checked), plus Apify's standard Actor-start event. No other fee from this Actor.

Link rows	Cost
100	$0.04
1,000	$0.40
5,000 (default cap)	$2.00
20,000 (hard cap)	$8.00

blocked rows are free: links the server refused (401, 403, 429, 999) and links that were not requested (robots.txt of any site, unreadable robots.txt, slow hosts, platform hosts, private addresses) are listed but not charged. slow-host, robots, robots-unreachable and robots-host-cap rows, and every robots.txt request to another site, count toward the unpaid-work cap (1,000). With Only output problems you pay only for the problem rows. Set Max total charge in the run options to cap spending: the run stops requesting and never writes a row it cannot charge. A row for an internal error (errorCode: "internal", a bug in this Actor, not in your site) is free too. Apify subscription discounts apply on top.

Slow hosts (opt-in): with Allow slow hosts on, a site whose robots.txt asks for a Crawl-delay of D seconds gets one request every D seconds, so it takes about D seconds per link (100 links at 2.8 s: about 5 minutes; delays over 2.8 s are not accepted). The price per row does not change.

Use it via API, n8n or Make

Get an API token in Apify Console under Settings, API & Integrations, and keep it in an environment variable (APIFY_TOKEN below). Never paste it into shared code.

cURL (starts a run, waits and returns the rows; the synchronous endpoint waits up to 5 minutes, so use it for small runs and the clients below for big ones):

curl -X POST "https://api.apify.com/v2/acts/ataism~broken-link-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://quotes.toscrape.com/tag/simile/"], "maxPages": 1}'

Python (pip install "apify-client>=3"):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("ataism/broken-link-checker").call(run_input={
    "startUrls": ["https://www.example.org/"],
    "maxPages": 200,
    "onlyProblems": True,
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["status"], row["statusCode"], row["url"], "<-", [f["pageUrl"] for f in row["foundOn"][:3]])

(With apify-client versions before 3.0, call() returns a dict: use run["defaultDatasetId"].)

JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('ataism/broken-link-checker').call({
    startUrls: ['https://www.example.org/'],
    maxPages: 200,
    onlyProblems: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const link of items) console.log(link.status, link.statusCode, link.url, link.foundOnCount);

n8n:

1. Add a Schedule trigger (weekly) or any trigger that gives you a site URL.
2. Add the Apify node, operation Run an Actor and get dataset, Actor ataism/broken-link-checker, input {"startUrls": ["{{ $json.siteUrl }}"], "maxPages": 200, "onlyProblems": true}.
3. Send the rows to a sheet, Slack or email; filter on status or foundOnCount.

Make:

1. Add the Apify module Run an Actor, choose ataism/broken-link-checker, put your site URL in the input JSON and set Run synchronously to Yes.
2. Add the Apify module Get Dataset Items and map the run's default dataset ID (one bundle per link).
3. Send the rows to Airtable, Google Sheets, a ticket tool or a webhook.

Privacy and terms

• The Actor reads only the pages you point it at (and checks the targets they link to). It collects URLs, HTTP status data and short anchor texts. mailto: and tel: links are skipped, and email addresses and phone numbers are removed from anchor text. The url column is the first form of each link the crawl found (fragment removed, everything else unchanged, including any session ID or tracking parameter), so a page that puts personal data inside a URL puts it in your dataset too.
• Nothing is sent to any AI or third-party service. No logins, no proxies, no CAPTCHA solving.
• Your dataset belongs to your Apify account and follows your retention settings.
• You are responsible for having the right to check the sites you enter (site terms, permission, rate expectations). The Actor obeys robots.txt and is polite by default, but it does not know your agreement with the site owner. If robots.txt cannot be read after one retry (server error or unreachable), the Actor treats the site as disallowed for that run, as RFC 9309 prescribes.
• User-Agent: broken-link-checker-actor/0.1 (https://apify.com/ataism/broken-link-checker). In robots.txt, User-agent: broken-link-checker-actor targets this Actor.

Safety

Only http and https to public addresses. Private, loopback, link-local and internal addresses and other schemes are blocked, including after every redirect (at most 5) and at connection time. Page bodies are capped at 2 MB (a longer page is read up to the cap), responses are decompressed with a size limit, and every request has a 15 s limit.

Limits and honest caveats

• No JavaScript. Links that only exist after scripts run are not found, and pages that need a browser to render may look empty.
• Limits: 10,000 pages, 20,000 unique links, 20 foundOn pages per link, 100 start and sitemap URLs, 25 sitemap files, 360 minutes per run.
• Some servers answer HEAD or automated requests differently from browsers: a link shown as broken may work for a person, and a blocked (403) link usually does. Treat blocked as "could not check", not as "broken".
• Same-site means the same host; blog.example.org is another site (checked, not crawled) when you start from example.org. Add it as a second start URL to crawl it.
• Soft 404 pages that answer 200 are ok. Anchor text is limited to 100 characters.
• Rows are written during the run in batches (every 50 rows or 30 s). If the run is aborted or the server migrates, buffered rows are written and the run saves its progress; a migrated run resumes without writing or charging a link twice. A hard crash can lose at most the checks of the last few seconds (they are redone on resume). Memory grows with the number of links (measured, 20,000 links with 20 sources each: about 215 to 225 MB when anchor texts repeat, up to about 470 MB when every anchor text is unique and nothing has been written yet). The 256 MB default is fine for up to about 5,000 links; above 500 pages or 5,000 links start the run with 512 MB or 1,024 MB. The Actor logs a warning at start when it has less than 512 MB and your limits allow a large crawl. After a resume the LINK_SOURCES checkpoint keeps the first 5 pages per link; foundOnCount stays exact.

FAQ

Why is a page I know exists blocked? The server (or a bot-protection layer) answered 401, 403 or 429. The Actor does not retry or bypass this. Lower the rate, or check those links another way.

Does it check each link once? Yes. A link used on 200 pages is one row with foundOnCount: 200 and up to 20 pages listed.

Is every crawled page charged? A crawled page is also a link, so it is one row, like any other link. There is no extra per-page fee.

Why does a run take minutes for 100 pages? The per-host rate limit (default 1 request per second). Raise it up to 4 only for sites you own or have permission to load.

Does the default run work without setup? Yes. It crawls one page of the public scraping sandbox quotes.toscrape.com and checks its links (about 23 rows, under 30 seconds).

Open-source components

The Actor uses the Apify SDK (apify 3.7.2, Apache-2.0). Link extraction, robots.txt and sitemap handling are plain code in this Actor.