SEO Audit & Broken Link Checker – Site Crawler

Crawls a website over plain HTTP and returns one record per page: every link and resource on it verified, the redirect chains, the on-page SEO problems, a 0-100 score with a letter grade — and, if you schedule it, what changed since the last run.

Area	What is checked
Links	every <a> link, internal and external, requested once per run: broken (404/410, 5xx, DNS, refused, TLS, redirect loop), redirecting (with the final URL and number of hops), or unverified (see below)
Resources	every image, script, stylesheet and iframe URL; http:// resources on https:// pages (mixed content)
Redirects	the full chain of each page, hop by hop, with status codes; loops and chains of 2+ hops
On-page	title (missing, multiple, too long/short), meta description, H1, canonical, lang, viewport, images without alt, word count
Indexing	noindex (meta and X-Robots-Tag), canonical pointing elsewhere, pages missing from the sitemap, sitemap pages nobody links to (orphans), noindex pages listed in the sitemap
Site level	duplicate titles and meta descriptions, robots.txt status, sitemap files read, security headers of the home page

Three verdicts, not two. A link is broken only when that is certain — and a 5xx answer is asked a second time before it counts. A target that answers 401, 403, 429, 503 or times out is reported as unverified: the server exists but will not show the page to a bot (LinkedIn, Stack Overflow and most bot-protected sites do this). Unverified links are listed separately and never lower the score as broken links.

Polite by design. It reads and obeys robots.txt (RFC 9309, including Crawl-delay), identifies itself as BrokenLinkAuditBot, sends at most 2 requests per second to the audited site (configurable 1-5) and never more than 1 per second to any other host. No browser, no cookies, no login.

Typical uses: find and fix broken links before they cost rankings · pre-launch and post-migration checks (did every old URL survive the redesign?) · a weekly scheduled audit that reports only new problems · agency health checks across client sites · checking a list of URLs from a spreadsheet.

Quick start

1. Put your site in Sites or URLs (example.com is enough).
2. Set Max pages per site — in crawl mode it is also your cost cap.
3. Run. Open the Overview table, or the BROKEN_LINKS record for a fix list grouped by broken target.

{
  "startUrls": ["https://example.com/"],
  "maxPagesPerSite": 200
}

Input

Field	Default	Meaning
startUrls	—	Crawl mode: one entry per site (a path such as example.com/blog/ starts there). List mode: the exact URLs to audit
mode	crawl	crawl follows internal links and the sitemap · list audits only the URLs given, one record each, whatever their status
maxPagesPerSite	200	Crawl mode: stop after this many pages per site. List mode audits every URL you give
checkExternalLinks	true	Verify links that leave the site
checkResources	true	Verify images, scripts, stylesheets, iframes
useSitemap	true	Read the sitemaps declared in robots.txt (or /sitemap.xml): index files, .gz and text sitemaps are supported
trackChanges	false	Compare each page with the previous run (see Monitoring)
stateKey	default	Name of the monitor: runs with the same name share one memory
maxDepth	20	Clicks from the start URL; 0 audits only the start URL and verifies its links (sitemap pages are not audited either)
maxLinksPerPage	500	Links beyond this number on one page are counted as skipped
stayInStartFolder	true	A start URL inside a folder (example.com/blog/) audits only pages under that folder; links elsewhere are checked but not audited or charged. A start URL at the root audits the whole site
includeSubdomains	false	Treat blog.example.com as part of example.com (www always is)
excludeUrlPatterns	—	URLs containing these texts are neither crawled nor checked; * is a wildcard (*/tag/*, ?replytocom=)
maxRequestsPerSecond	2	1-5, to the audited site only
timeoutSecs	15	Per request; link checks use at most 10 s

Output

Dataset — one record per page

57 typed fields, all documented in the dataset schema (Storage → Dataset → Schema). A real record (2026-10-06, the first web page ever published, which has one dead link):

{
  "url": "http://info.cern.ch/hypertext/WWW/TheProject.html",
  "finalUrl": "http://info.cern.ch/hypertext/WWW/TheProject.html",
  "site": "info.cern.ch",
  "discoveredVia": "start",
  "depth": 0,
  "statusCode": 200,
  "redirectHops": 0,
  "responseTimeMs": 133,
  "score": 72,
  "grade": "C",
  "issueCodes": [
    "BROKEN_INTERNAL_LINK",
    "META_DESCRIPTION_MISSING",
    "VIEWPORT_MISSING",
    "LANG_MISSING",
    "CANONICAL_MISSING"
  ],
  "title": "The World Wide Web project",
  "titleLength": 26,
  "h1": "World Wide Web",
  "h1Count": 1,
  "wordCount": 169,
  "canonical": null,
  "indexable": true,
  "linksTotal": 23,
  "internalLinks": 23,
  "externalLinks": 0,
  "resourceLinks": 0,
  "linksChecked": 23,
  "brokenLinksCount": 1,
  "brokenLinks": [
    {
      "url": "http://info.cern.ch/hypertext/WWW/Administration/Mailing/Overview.html",
      "type": "link",
      "internal": true,
      "status": 404,
      "reason": "HTTP 404",
      "anchorText": "Mailing lists",
      "occurrences": 1,
      "finalUrl": null
    }
  ],
  "redirectedLinksCount": 0,
  "unverifiedLinksCount": 0,
  "skippedLinksCount": 0,
  "changeStatus": null,
  "coverageComplete": true,
  "issues": [
    {
      "code": "BROKEN_INTERNAL_LINK",
      "severity": "high",
      "area": "links",
      "message": "1 link to this site is broken.",
      "count": 1
    }
  ]
}

(shortened: the full record also carries redirectChain, redirectedLinks, unverifiedLinks, mixedContent, metaDescription, metaRobots, xRobotsTag, inSitemap and the change fields.)

Key-value store

• BROKEN_LINKS — the fix list: one entry per broken target with its status, the pages that link to it (foundOn), the anchor texts and how many pages carry it.
• SUMMARY — per site: pages audited, average score, grades, count of every issue code, duplicate titles and meta descriptions, robots.txt and sitemap report (files read, URLs listed, URLs not reached), security headers of the home page, and the change report.

Score and grade

score = 100 − Σ penalties, one penalty per issue code present on the page: critical 25 · high 12 · medium 6 · low 2 · info 0, floored at 0. Grade: A ≥ 90 · B ≥ 75 · C ≥ 60 · D ≥ 40 · F below 40 or any critical issue. Only analysed HTML pages get a computed score. A start or input URL that is certainly dead gets 0 / F; one that was not analysed (unreachable, refused the bot, robots.txt, not HTML) gets score: null and is left out of the site average.

Issue codes

Code	Severity	Meaning
HTTP_4XX (404/410) · HTTP_5XX · REDIRECT_LOOP	critical	the URL itself is certainly dead: score 0
FETCH_FAILED	critical	the URL could not be reached at all (DNS, timeout, refused): no score, not charged
NOT_VERIFIABLE	info	the URL answered 401/403/429/503 or another 4xx to the bot: it may work in a browser. No score, not charged
BROKEN_INTERNAL_LINK · BROKEN_RESOURCE	high	links to your own site, or images/scripts/stylesheets/iframes, that do not load
BROKEN_EXTERNAL_LINK	medium	links to other sites that are certainly dead
MIXED_CONTENT	high	http:// resources on an https:// page
REDIRECT_CHAIN / REDIRECTED	medium / info	the URL needs 2+ redirects / one redirect to reach its page
INTERNAL_LINK_REDIRECTS	low	internal links that point to a redirecting URL: link to the final address
LINKS_UNVERIFIED	info	targets that block bots, ask for a login or timed out
TITLE_MISSING	high	no <title>
META_DESCRIPTION_MISSING · H1_MISSING · VIEWPORT_MISSING	medium
TITLE_TOO_LONG (> 60) · TITLE_TOO_SHORT (< 10) · TITLE_MULTIPLE · META_DESCRIPTION_TOO_LONG (> 160) · H1_MULTIPLE · IMG_ALT_MISSING · LANG_MISSING · CANONICAL_MISSING	low
NOINDEX_IN_SITEMAP	medium	the page says noindex but the sitemap lists it
NOT_LINKED_INTERNALLY	low	in the sitemap, but no crawled page links to it
NOINDEX · CANONICAL_POINTS_ELSEWHERE · NOT_IN_SITEMAP	info	facts worth knowing, no penalty
NOT_HTML · BLOCKED_BY_ROBOTS	info	a start/input URL that is a file (PDF, image…) or is forbidden by robots.txt: no score
SLOW_RESPONSE (> 3 s) · HTML_TOO_LARGE (> 3 MB)	low

Codes are stable: filter on issueCodes in your integration.

Monitoring: what changed since the last run

Turn on trackChanges and schedule the Actor (daily, weekly). The first run is the baseline; from the second run every record carries:

• changeStatus: new · changed · unchanged
• newIssueCodes / resolvedIssueCodes
• newBrokenLinks / fixedBrokenLinks

and SUMMARY.sites[].changes lists the new and fixed broken links of the whole site. Send only the news to Slack or email: filter the dataset on changeStatus = changed or new. The memory lives in the named key-value store broken-link-seo-site-audit-state in your account; use a different stateKey for independent monitors of the same site. Every page is audited (and charged) on every run — the comparison needs the fresh data.

Pricing

Pay per event, one event: page-audit — one page analysed with all its links verified. No start fee, no per-link fee: a page with 300 links costs the same as a page with 3.

Not charged (coverageComplete: false): URLs that could not be reached, URLs that refused the bot (401/403/429/503…), URLs forbidden by robots.txt, pages left unfinished because the run was stopped. Invalid input entries produce no record at all.

What counts as one event besides an analysed page: a start or input URL whose status is established — a 404/410, a 5xx confirmed twice, a redirect loop, or a file that is not HTML. In crawl mode that can only be one of your start URLs; in list mode it is every URL you list, so list mode doubles as a status checker at the page price. Pages found by the crawl are charged only when they are HTML pages that were fully analysed.

Example: a 150-page site audited every Monday is 150 events per run, about 650 a month — multiply by the price shown in the Pricing tab. maxPagesPerSite (crawl mode) and the run's maximum cost both cap the bill: the Actor stops when either is reached. In list mode the cap is the length of your list, and the run's maximum cost.

How long does it take

Speed is bounded by politeness, not by compute: each unique URL of the audited site is one request, at 2 per second by default. Roughly: 100 pages with 400 unique internal URLs and 150 external links ≈ 4-5 minutes. A Crawl-delay in robots.txt overrides the setting (capped at 10 s per request): a site that asks for Crawl-delay: 10 takes about 5 minutes per 30 URLs, and the log says so. When the run timeout approaches, the Actor stops starting new pages and still writes SUMMARY.

Limits and honesty

• No JavaScript rendering. Links and content injected by client-side scripts are not seen. Server-rendered sites, static sites and most CMS sites (WordPress, Shopify, Webflow, Hugo, Docusaurus …) are fully covered; a single-page app that renders everything in the browser is not a good fit.
• Links inside CSS files, srcset, <video>/<audio> sources and inline styles are not checked. Fragments (#section) are not verified against the target page.
• robots.txt is obeyed for the audited site: disallowed URLs are neither crawled nor checked (they count as skipped). If robots.txt answers 5xx the site is treated as fully disallowed, as RFC 9309 requires. External link targets are checked with a single HEAD/GET, as any link checker does.
• "Broken" for an external link means the answer at the time of the run. A server that is down for a minute looks the same as a dead one: with trackChanges you see whether it stays broken.
• Soft 404s (a "not found" page served with status 200) cannot be told from real pages.
• Two addresses that serve the same content (/ and /index.html, or the same page with different query strings) are two pages: two records, two events. The second one shows CANONICAL_POINTS_ELSEWHERE when the site declares a canonical. Use excludeUrlPatterns for tracking or filter parameters (?utm_, ?sort=), and for calendars or faceted navigation that generate endless URLs.
• rel="nofollow" links are followed and checked like any other link.
• A page served without a text/html (or XHTML) Content-Type is treated as a file, not analysed.
• An incomplete TLS chain on a link target is reported as unverified, not broken: most browsers repair it on their own. An expired, self-signed or wrong-name certificate is broken.
• If the site starts refusing the bot (many 403/429 answers on its own pages), SUMMARY says so in warnings and internalPagesNotReadable, and the site status becomes partial when that is a fifth of the site or more — a quiet "all clean" on a blocked crawl would be a lie.
• Size: built for sites up to a few tens of thousands of pages per run (about 35 KB of memory per page). For larger sites split the audit by section (start URLs example.com/blog/ and example.com/shop/ in separate runs: with stayInStartFolder on, each run audits and charges only its own folder) or raise the run memory.
• With trackChanges, a page first reached in a later run (because an earlier run stopped at maxPagesPerSite) is reported as new; the memory holds the 20,000 most recently audited pages per site.
• Duplicate titles and descriptions are compared among indexable pages, exact match ignoring case.
• Title and description limits are character counts (60 / 160), an approximation of what search results show.
• Private and loopback addresses (localhost, 10.x, 192.168.x …) are never requested.
• This is a technical screening, not a ranking guarantee.

API & integrations

curl -X POST "https://api.apify.com/v2/acts/far003~broken-link-seo-site-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://example.com/"],"maxPagesPerSite":50}'

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('far003/broken-link-seo-site-audit').call({ startUrls: ['https://example.com/'], maxPagesPerSite: 50, trackChanges: true });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const fixList = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('BROKEN_LINKS');

Works with Apify schedules, webhooks, Make, Zapier, n8n and the Apify MCP server: every field is typed and described, so an AI agent can read the result without guessing.