Broken Link & Redirect Chain Checker

Crawl a site you own. For every unique internal link and every outbound link, the Actor reports the HTTP status, the redirect chain, and the page and anchor text that pointed at it. Schedule it after a migration and the next run marks links that were not broken last time.

No browser. No login. No cookies. No form submissions. The crawler does not run JavaScript and does not solve CAPTCHAs.

What you get

One dataset row per unique link target (the same URL found on many pages is one row):

Field	Description
url	Normalized target. The host is lowercased. The fragment and the query string are always removed before anything is stored.
linkType	internal (the start host, plus the www twin only when Include www is on) or outbound.
status	Final HTTP status, or null when the link was not requested.
category	ok, redirect, broken, error, not checked (robots.txt), blocked (private address), or skipped (cap).
finalUrl	URL of the last response. Same normalization as url.
redirectChain	Each hop actually requested: { url, status }. At most 5 redirects (the first request plus 5 hops).
redirectCount	Number of redirects that were followed.
isRedirectLoop	true when the chain repeats a URL.
foundOn	Up to 20 { pageUrl, anchorText } pairs from your pages. Anchor text is trimmed to 200 characters.
foundOnCount	How many times the link was seen, including ones past the 20 stored pairs.
firstSeenDepth	Clicks from the start page. The start page is 0.
checkedAt	ISO 8601 time.
error	Short reason, or null.
isNewlyBroken	true only in monitor mode when the link is broken now and was not broken on the previous run. Otherwise false.

ok is a final 2xx with no redirect. redirect is a final 2xx after one or more redirects. broken is a final status of 400 or higher, except 408 and 429, which are error. A redirect loop is broken with error redirect loop. Hitting the 5-redirect cap is error with redirect limit.

Example row

{
  "url": "https://example.com/pricing",
  "linkType": "internal",
  "status": 200,
  "category": "redirect",
  "finalUrl": "https://example.com/plans",
  "redirectChain": [
    { "url": "https://example.com/pricing", "status": 301 },
    { "url": "https://example.com/plans", "status": 200 }
  ],
  "redirectCount": 1,
  "isRedirectLoop": false,
  "foundOn": [{ "pageUrl": "https://example.com/", "anchorText": "Pricing" }],
  "foundOnCount": 1,
  "firstSeenDepth": 1,
  "checkedAt": "2026-10-27T12:00:00.000Z",
  "error": null,
  "isNewlyBroken": false
}

The default key-value store also has:

• OUTPUT, the run summary (counts, caps, the user-agent, and why the run stopped).
• BROKEN_LINKS.csv, every broken link classified in this run. With Only new breaks, the dataset omits old breaks and the CSV still lists them.

Input

Field	Default	What it does
startUrl	console prefill only	First page. Required. The crawl stays on this exact host.
ownershipConfirmed	false (required)	You own or manage the site, or you have permission to scan it. The run stops immediately when this is not true.
includeWww	false	Also crawl the www / non-www twin. Off means example.com and www.example.com are different sites.
maxPages	100 (prefill 10, hard cap 50,000)	HTML pages to download. Further internal links still get one status check.
maxLinks	2,000 (hard cap 50,000)	Unique targets to keep. The rest are counted and dropped.
maxDepth	2 (hard cap 20)	How deep to parse HTML. The start page is depth 0.
checkOutboundLinks	true	One status check per external URL. Those URLs are never crawled.
includeQueryString	false	When off, a link's query is removed before the request. When on, that query is sent. Links that differ only by query collapse into one stored row. It is never stored. A redirect's Location query is always requested. Logs never keep a query, a fragment, or a username.
monitorMode	false	Compare broken links with the previous run. Requires the verification file on the default port.
onlyNewBreaks	false	Requires monitor mode. The run rejects this flag when monitor mode is off, before any request. With monitor mode, save only newly broken rows. Requested URLs are still billed.
verificationToken	empty	16–64 characters of letters, digits, _ or -. The start URL must use the default port.
ignoreRobotsOnOwnSite	false	Skip robots.txt on the exact start host only. The www twin is still obeyed. Requires the verification file.

A one-off run needs ownershipConfirmed and nothing else. Monitor mode, Ignore robots.txt on my site, and any scheduled run also need the verification file, and the start URL must use the default port (443 for https, 80 for http). A custom port is rejected before any request. A scheduled run is one Apify starts with APIFY_META_ORIGIN=SCHEDULER (the same value as meta.origin on the run). Without that file, the run is one-off only, and only after you check the ownership box. A one-off run may still use a custom port.

Place the token alone in:

https://YOUR-HOST/.well-known/broken-link-checker-verify-TOKEN.txt

The Actor requests that exact URL over HTTPS on port 443: https://<start host>/.well-known/broken-link-checker-verify-<token>.txt. It accepts only HTTP 200 from that origin and path, with a body that is exactly the token. A redirect is a failure, including a redirect to the www twin or to another host. The token is not written to the log or to OUTPUT.

The console prefill for startUrl lives only in .actor/input_schema.json. The daily test uses our own fixture site, https://hardwoodlogic.github.io/broken-link-checker-fixture/. ownershipConfirmed is prefilled true and maxPages is prefilled 10. Crawlers read the host root: https://hardwoodlogic.github.io/robots.txt returns 404, which allows the host. The fixture file https://hardwoodlogic.github.io/broken-link-checker-fixture/robots.txt returns 200 and allows every agent (User-agent: SiteLinkChecker / Allow: /, and User-agent: * / Allow: /).

Pricing

Pay per event: $1.00 per 1,000 checked links ($0.001 each). The event id is link-checked and the title is Link checked. There is no start fee. apify-actor-start and apify-default-dataset-item are not used.

A charge is one unique stored URL (origin and path) that was requested and came back billable: ok, redirect, broken, or error other than dns. linksChecked and linksCharged both count those successful link-checked events, so they match. A HEAD that is not allowed is followed by a single Range GET, and that pair is still one check. These are free and are not link-checked events:

• the same URL seen again
• a URL robots.txt disallows, including a redirect hop whose path or host is disallowed (an earlier hop in that chain is not billed either)
• a private or otherwise blocked address
• a DNS failure (ENOTFOUND, EAI_AGAIN, and the same family). The row is error with error dns, because no request was made. A robots.txt lookup that fails DNS does not hide this as not checked (robots.txt)
• mailto:, tel:, javascript: and data: links
• anything past the page, link, depth, or charge cap

Monitor mode uses the same rule. A broken link that is not new is still billed when it was requested. Rows that are not billed are saved with no charge event. The run stops at your max cost and leaves the remaining links as skipped (cap) without requesting them. The previous monitor snapshot is kept when the charge limit stops the run, so a partial run does not hide older breaks. The default dataset is the charge ledger: a resumed run rebuilds the charged set from pushed rows, and a row already in the dataset is not charged again. Crawl progress is appended as deltas about every 30 seconds and when the run is migrating, aborting, or persisting state. A hard kill can re-charge up to 4 links, the pushes still in flight (global concurrency is 4). An onlyNewBreaks charge is written into one ledger record for that in-flight batch before the slot is released, so it is inside the same cap.

The price is set in .actor/pay_per_event.json and applied in Apify Console (Publication → Monetization) when the Actor is published. It is not applied automatically from this repo. On the pay-per-event model, platform usage is included the same way as our other Actors: the event price is the price of a checked link.

Default memory is 512 MB (minimum 256, maximum 1024) and the default timeout is 30 minutes. There is no browser. The per-host gap is 550 ms (under 2 requests per second) so a sliding one-second window does not see three starts. One thousand links on a single host take about 550 seconds. Customers pay only the link-checked event, $1.00 per 1,000 links. Platform usage is included.

How a run behaves

• At most 2 requests per second per host (550 ms between starts), counted separately for your host and for each outbound host. Global concurrency is 4. Each request waits at most 15 seconds, and that timer starts after the host slot is acquired. A 429 or 503 backs off and honours Retry-After.
• HTML is downloaded only when the URL being requested, after any earlier redirect, is already on your host (www only when Include www is on). The decision is made before the body is read. An off-site URL, including the page an internal URL redirects to, is one status check: the body is discarded at the headers, nothing is extracted, and nothing new is enqueued. Other content types and obvious binary files get a status check only.
• HTML pages on your host are parsed for links (a, area, canonical link, img, source, video, audio, including srcset and poster). A <base href> is used only when it is an http(s) URL. A mailto:, javascript:, data:, file:, tel: or sms: base is ignored and the fetched page URL is used instead.
• Outbound links get one HEAD. If the server answers 405 or 501, the Actor sends one GET with Range: bytes=0-0 and stops at the headers. Error pages are cut off the same way. Page bodies are not stored.
• Redirects are followed at most 5 times. The request uses the full Location URL, query included. Loop detection uses that full URL. Stored rows, crawl state, and the monitor snapshot keep origin and path only. Every hop is checked again for a private or link-local address, including 198.18.0.0/15, and for that hop's robots.txt, including a same-host path change. A blocked hop is blocked (private address) and is not billed. A disallowed hop stops the chain and is not billed.
• robots.txt is fetched once per host per run and cached. Rules are matched against the path and query of the URL that would be requested. A complete file of at most 500 KiB is parsed. A file larger than that is truncated and treated as unreachable, so the host is disallowed. A 4xx robots.txt allows the host. A 5xx, a network failure, or a file that cannot be read does not. A DNS failure is an error / dns row instead. The product token in robots.txt is SiteLinkChecker. Ignore robots.txt on my site applies only to the verified start host. With Include www on, the www twin is still obeyed.
• The user-agent on every request is exactly SiteLinkChecker/0.1 (+https://apify.com/hardwoodlogic/broken-link-checker).

Monitor state for a verified host lives in a named key-value store blc- plus the first 40 hex characters of the SHA-256 of the hostname. The record key is broken-set. The value is { "v": 1, "urls": ["https://…"] } of broken URLs with queries and fragments removed. The store is opened only after the ownership file checks out. The first run has an empty previous set, so every current broken link is newly broken.

FAQ

How do I find redirect chains after a site migration?

Run the Actor on the new site. In the dataset, filter category to redirect, or keep rows with redirectCount greater than 0. redirectChain lists each hop and its status, and finalUrl is where the chain landed. Chains stop after 5 redirects. A loop is category broken with error redirect loop.

What do I need for a scheduled run?

A schedule is not a one-off run. Set verificationToken and publish /.well-known/broken-link-checker-verify-TOKEN.txt on the host before the schedule fires. The ownership checkbox is still required. Monitor mode and the robots override need the same file even when you start the run by hand.

Why were some links not checked?

not checked (robots.txt) means that host's robots.txt disallows the URL, or the file could not be used. An oversized robots.txt (over 500 KiB) disallows the whole host. A 5xx, a network error that is not DNS, or a truncated file does too. A 4xx, including a 404, allows the host. error with dns means the name did not resolve; that row is free. blocked (private address) means the URL, or a redirect hop, is not a public address. skipped (cap) means a page, link, or charge cap was already hit. None of those are billed. linksChecked counts only successful link-checked events, the same number as linksCharged. mailto:, tel:, javascript: and data: links are counted in OUTPUT.skippedSchemes and are not stored.

Safeguards

The crawl is limited to the start host, at 2 requests per second, with a real product token in the user-agent and in robots.txt. It does not pretend to be a browser. It sends no cookies and does not log in. A query on a link you discovered is sent only when you opt in. A query on a redirect Location is always requested. Queries are never stored, and logs drop the query, the fragment, and any username. Anchor text is scrubbed for email addresses and phone numbers. The Actor does not collect personal data and does not extract addresses or phone numbers from pages.

Opt out and abuse

If a site should not be checked, disallow SiteLinkChecker in robots.txt. The group name is matched case-insensitively. To report a run or ask a question, use the Actor's public Issues tab. The user-agent info URL is https://apify.com/hardwoodlogic/broken-link-checker. There is no personal email address on this Actor.

Privacy

Rows contain the URL, the status, the redirect chain (URL and status only), and the source page URL and anchor text from your own pages. Page bodies and response headers other than the status and Location are not stored. mailto: and tel: values are not stored. Query strings are never stored. With Keep query strings off, a query on a link you discovered is not sent either. A query on a redirect Location is sent and then dropped before it is saved.

Run summary (OUTPUT)

status is SUCCEEDED or FAILED. error is a short message or null. stoppedReason is null or chargeLimit. resumed is true when the run continued from the dataset or from saved crawl progress. Counts include every segment of the run, so linksChecked stays equal to linksCharged. The record also includes the normalized startUrl (origin and path only), host, the flags you set (includeWww, includeQueryString, checkOutboundLinks, monitorMode, onlyNewBreaks, scheduled, ignoreRobotsOnOwnSite), verification (not required or passed), maxRedirects (5), maxPages (default 100), the caps, pagesCrawled, pagesParsed, linksDiscovered, linksChecked (successful link-checked events), linksCharged (the same count), rowsPushed, robotsSkipped, blocked, linksOverCap, outboundNotChecked, capsHit, skippedSchemes, countsByCategory, newlyBroken, chargeEvent (link-checked), userAgent, brokenCsvKey (BROKEN_LINKS.csv), and monitorStore (the named store, or null). The verification token is not part of this record.

Limits

• Default run options: 512 MB and 1,800 seconds (30 minutes). Raise memory up to 1024 MB and the timeout in the run options for a larger crawl. maxPages defaults to 100 so a large site is opt-in. The hard cap is 50,000 pages and 50,000 links.
• HTML downloads are capped at 2 MiB. A complete robots.txt of at most 500 KiB is parsed. A larger robots.txt disallows the host.
• Only http and https are requested.

En français

Ce programme parcourt un site dont vous êtes propriétaire ou que vous êtes autorisé à analyser. Pour chaque lien interne et chaque lien sortant, il indique le code HTTP, la chaîne de redirection (5 redirections au plus) et la page d'origine. Le mode surveillance signale les liens nouvellement cassés. Il n'utilise pas de navigateur, n'envoie pas de cookies et ne se connecte pas à un compte.

Usage accepté : un contrôle ponctuel après avoir confirmé que vous gérez le site. Le mode surveillance, l'ignorance de robots.txt sur votre propre hôte, et toute exécution planifiée exigent un fichier de vérification à /.well-known/broken-link-checker-verify-<jeton>.txt, servi sur le port par défaut : 443 en HTTPS, ou 80 en HTTP. Un port personnalisé est refusé avant toute requête. Sans ce fichier, l'exécution reste ponctuelle.

Le jumeau www est un autre site. Il n'est parcouru que si l'option www est activée, et son robots.txt reste appliqué même lorsque vous ignorez robots.txt sur l'hôte exact de départ.

Un robots.txt de plus de 500 Kio est tronqué et interdit tout l'hôte. Un fichier 4xx, y compris un 404, autorise l'hôte. Seuls un 5xx, un fichier injoignable ou un fichier tronqué l'interdisent.

Une interruption brutale peut refacturer au plus 4 liens : les envois encore en cours, car la concurrence globale est de 4. Un lien déjà présent dans le dataset n'est pas refacturé. Une charge onlyNewBreaks est écrite dans un seul registre pour ce lot en cours, avant que son créneau ne soit libéré, donc elle reste dans ce plafond de 4. Le test quotidien utilise notre propre site de démonstration : https://hardwoodlogic.github.io/broken-link-checker-fixture/ . Les robots lisent la racine de l'hôte, https://hardwoodlogic.github.io/robots.txt, qui répond 404. Le fichier du fixture, https://hardwoodlogic.github.io/broken-link-checker-fixture/robots.txt, répond 200 avec Allow: /.

Vie privée : aucune donnée personnelle n'est collectée. Les adresses mailto: et tel: ne sont pas enregistrées. Les query strings ne sont jamais enregistrées. Avec l'option query activée, des liens qui ne diffèrent que par la query sont fusionnés en une seule ligne. Une page qui redirige hors du site n'est pas analysée. Pour vous opposer à un contrôle ou signaler un abus, interdisez SiteLinkChecker dans robots.txt et écrivez dans l'onglet Issues public de l'Actor : https://apify.com/hardwoodlogic/broken-link-checker/issues

Run locally

npm install
cd actors/broken-link-checker
npm test

Tests use a fake fetch. They do not call the network.

$apify run --purge -i '{"startUrl":"https://hardwoodlogic.github.io/broken-link-checker-fixture/","ownershipConfirmed":true,"maxPages":10}'

The daily test uses our own fixture site. That command is the console prefill: the same URL, in .actor/input_schema.json only, with ownershipConfirmed true and maxPages 10. The site has a few pages and one deliberate 404. Crawlers read the host root: https://hardwoodlogic.github.io/robots.txt returns 404, which allows the host. The fixture file https://hardwoodlogic.github.io/broken-link-checker-fixture/robots.txt returns 200 with Allow: /. Point startUrl at another host you control when you run it for real.

Deploy to Apify

This Actor is part of a monorepo. The image is built from the repository root (dockerContextDir is ../../..). Releases are upload builds. Package a clean checkout, then upload dist/broken-link-checker. The steps are in the ../../README.md#releasing-an-actor. Publish only after QA, with approval.