SEO Site Audit: Broken Links, Sitemap, Robots.txt

What this Actor does

SEO Site Audit crawls a website from one start URL and lists its technical SEO problems, page by page, with a fix-first checklist at the end.

For each page it records the HTTP status, redirects, response time and size, title, meta description, H1s, canonical tag, noindex and nofollow (meta tag and X-Robots-Tag header), hreflang, lang, word count, images without alt text, structured data types (JSON-LD and microdata) and Open Graph tags.

It also checks:

• Links: every internal link it crawls and every external link (up to a limit) for 404s, server errors, timeouts, redirect chains and redirect loops, with the pages that link to each one.
• robots.txt: whether it exists, lines crawlers do not understand, rules that block Googlebot from the whole site or the start page, and which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) it allows.
• Sitemaps: found through robots.txt or /sitemap.xml, including sitemap indexes and .gz files. Sitemap URLs that return errors, redirect, are noindex, point their canonical elsewhere or are blocked by robots.txt are flagged, and so are crawled pages that are missing from the sitemap.
• llms.txt and llms-full.txt: whether they exist and whether llms.txt follows the llms.txt format.
• HTTPS and www: whether http:// redirects to https://, and whether the www and non-www homepages end at the same URL.

Use it to check your own site before a launch or migration, to find broken links on a schedule, or to audit a client's site.

Input

Only Start URL is required. All settings:

Field	Default	What it does
Start URL (startUrl)	none	Where the crawl starts, usually the homepage. Links on the same host are followed; www and non-www count as the same site.
Maximum pages to audit (maxPages)	100	The crawl stops after this many HTML pages (1 to 5,000).
Check external links (checkExternalLinks)	true	Checks links to other sites with a HEAD request, then a GET if HEAD fails.
Maximum external links to check (maxExternalLinks)	500	Unique external URLs checked per run (0 to 10,000). At most 100 per external site.
Include subdomains (includeSubdomains)	false	Treats subdomains of the start host (such as blog.example.com) as part of the site.
Also audit pages listed in the sitemap (crawlSitemapUrls)	true	After the pages found through links, audits sitemap URLs too. This finds pages nothing links to.
Respect robots.txt (respectRobotsTxt)	true	Skips URLs the site's robots.txt does not allow for this crawler and follows its Crawl-delay. Turn off only for a site you own or may audit.
Requests at once (maxConcurrency)	5	Parallel requests (1 to 20). Requests to one host still start at least 1 second apart.

Example:

{
    "startUrl": "https://www.example.com/",
    "maxPages": 300,
    "checkExternalLinks": true,
    "maxExternalLinks": 1000
}

Output

The dataset has three kinds of rows, told apart by rowType. The dataset has a view for each: Pages, Links and Site checks.

• site (one row): robotsTxt, sitemap, llmsTxt, llmsFullTxt, homepageRedirects, aiCrawlers and the site-level issues.
• page (one row per crawled URL): the page facts listed above, brokenLinks and redirectedLinks found on the page, inlinks (how many crawled pages link to it), inSitemap, indexable, depth (clicks from the start URL) and issues. status is audited, error (4xx, 5xx, timeout or redirect loop), not_allowed (redirects to a private address) or skipped (the run hit your cost limit).
• link (one row per linked URL with a problem): linkType (internal or external), status (broken, unreachable, timeout, redirect, redirect_loop, blocked or not_checked), statusCode, finalUrl, redirectChain, linkedFromCount, linkedFrom (up to 20 pages) and anchorText. Links that work are counted in the summary but get no row.

Every row has issues: a list of { severity, code, message }, worst first. Severity is error (fix first), warning (should fix) or notice (worth a look).

This is a real page row from a test run on jvns.ca (both broken links and the second H1 were checked by hand):

{
    "rowType": "page",
    "url": "https://jvns.ca/blogroll",
    "status": "audited",
    "statusCode": 200,
    "finalUrl": "https://jvns.ca/blogroll/",
    "redirectChain": [
        {
            "url": "https://jvns.ca/blogroll",
            "statusCode": 301
        }
    ],
    "contentType": "text/html; charset=UTF-8",
    "responseTimeMs": 63,
    "sizeBytes": 14286,
    "depth": 1,
    "foundVia": "link",
    "title": "Blogs I like",
    "titleLength": 12,
    "metaDescription": "Blogs I like",
    "metaDescriptionLength": 12,
    "h1Count": 2,
    "h1": "Julia Evans",
    "canonicalUrl": "https://jvns.ca/blogroll/",
    "robotsMeta": null,
    "xRobotsTag": null,
    "noindex": false,
    "nofollow": false,
    "indexable": true,
    "hreflang": [],
    "lang": "en",
    "wordCount": 499,
    "imagesTotal": 0,
    "imagesMissingAlt": 0,
    "imagesMissingAltExamples": [],
    "structuredDataTypes": [],
    "openGraph": {
        "title": "Blogs I like",
        "description": null,
        "image": null,
        "url": "https://jvns.ca/blogroll/",
        "type": "website"
    },
    "twitterCard": null,
    "internalLinks": 11,
    "externalLinks": 67,
    "brokenLinks": [
        "https://mickens.seas.harvard.edu/wisdom-james-mickens",
        "https://blog.safia.rocks/post/171448973555/getting-into-git-init"
    ],
    "redirectedLinks": [
        "https://jvns.ca/about",
        "https://jvns.ca/talks",
        "https://jvns.ca/blogroll",
        "https://jvns.ca/bookshelf",
        "http://kamalmarhubi.com/blog/",
        "http://kamalmarhubi.com/blog/2016/04/25/segfaults-are-our-friends-and-teachers/",
        "http://decomposition.al/",
        "http://decomposition.al/blog/2019/04/26/an-example-run-of-the-chandy-lamport-snapshot-algorithm/",
        "https://www.harihareswara.net/ces.shtml",
        "https://www.harihareswara.net/sumana/2018/07/11/0",
        "https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8"
    ],
    "inlinks": 97,
    "inSitemap": true,
    "issueCount": 5,
    "issues": [
        {
            "severity": "error",
            "code": "broken-links-on-page",
            "message": "2 broken links."
        },
        {
            "severity": "notice",
            "code": "description-too-short",
            "message": "Meta description is 12 characters."
        },
        {
            "severity": "notice",
            "code": "h1-multiple",
            "message": "2 H1 headings."
        },
        {
            "severity": "notice",
            "code": "og-missing",
            "message": "Missing og:image."
        },
        {
            "severity": "notice",
            "code": "redirected-links-on-page",
            "message": "11 links to redirected URLs."
        }
    ],
    "error": null
}

Summary and checklist

The OUTPUT record in the run's key-value store has:

• pagesAudited, pagesWithErrors, pagesSkipped, urlsBlockedByRobotsTxt, internalUrlsNotCrawled
• externalLinks: found, checked, broken, redirected, blocked, timed out, not checked
• issueCounts by severity, topIssues (the 10 worst issue types) and issuesByType (every issue type with up to 5 example URLs and what to do)
• checklist: one plain-language line per issue type, worst first, for example [error] Broken link. 3 links: Fix or remove the link on the pages listed in linkedFrom.
• site: the same site checks as the site row
• stoppedEarly: why the crawl stopped before the page limit, if it did

Issues it reports

Page issues:

Code	Severity	Meaning
page-4xx	error	Page returns a 4xx error
page-5xx	error	Page returns a server error
page-blocked	warning	Page returns 401, 403, 429 or 503 to the crawler
page-unreachable	error	Page timed out or the connection failed
redirect-loop	error	Redirect loop or more than 10 redirects
redirect-chain	warning	2 or more redirects before the page
title-missing	error	Missing or empty title
title-duplicate	warning	Same title as another indexable page
title-too-long / title-too-short	notice	Over 60 or under 10 characters
title-multiple	warning	More than one <title> tag
description-missing	warning	Missing or empty meta description
description-duplicate	warning	Same meta description as another indexable page
description-too-long / description-too-short	notice	Over 160 or under 50 characters
h1-missing	warning	No H1
h1-multiple	notice	More than one H1
canonical-missing	notice	No canonical tag
canonical-multiple	warning	More than one canonical tag
canonical-other	notice	Canonical points to another URL
canonical-broken	error	Canonical points to a URL that is broken or redirects
canonical-chain	warning	Canonical points to a crawled page whose own canonical points elsewhere, including two pages that point to each other
noindex	warning	noindex in meta robots or X-Robots-Tag
nofollow-page	notice	nofollow for the whole page
blocked-for-google	warning	robots.txt blocks Googlebot from the page
hreflang-invalid	warning	hreflang value is not a valid language code
lang-missing / lang-invalid	notice	Missing or malformed <html lang>
viewport-missing	warning	No viewport meta tag
low-word-count	notice	Fewer than 200 words of visible text
images-missing-alt	warning	<img> tags without an alt attribute (alt="" counts as present)
structured-data-invalid	warning	A JSON-LD block that does not parse
og-missing	notice	No og:title or og:image
slow-response	notice	Over 3 seconds to download
large-html	notice	HTML over 2 MB
page-truncated	warning	HTML over 10 MB; only the first 10 MB were audited
duplicate-content	warning	Exactly the same visible text as another indexable page
broken-links-on-page	error	The page links to broken URLs
redirected-links-on-page	notice	The page links to URLs that redirect
http-links-on-https-page	warning	Internal links that use http:// on an HTTPS site
not-in-sitemap	notice	Indexable page missing from the sitemap
orphan-page	notice	Found only in the sitemap; no crawled page links to it
non-html-start	error	The start URL is a file, not an HTML page

Link issues: link-broken (error), link-redirect-loop (error), link-unreachable (warning: timeout, refused connection or private address), link-redirect-chain (warning), link-redirect (notice) and link-blocked (notice: the site answered 401, 403, 429 or 503, or its robots.txt does not allow checks). A link whose domain does not exist is reported as link-broken.

Site issues: robots-missing, robots-server-error, robots-html, robots-invalid-lines, robots-rules-before-user-agent, robots-blocks-site, robots-blocks-start, robots-no-sitemap, robots-too-large, robots-disallows-auditor, sitemap-missing, sitemap-invalid, sitemap-url-broken, sitemap-url-redirect, sitemap-url-noindex, sitemap-url-non-canonical, sitemap-url-blocked, sitemap-url-other-host, sitemap-invalid-lastmod, llms-txt-invalid, http-not-redirected, https-unavailable, homepage-variants-differ and homepage-redirect-chain. A missing llms.txt is shown in the llmsTxt field but is not counted as an issue, since the file is optional.

Pricing

You pay per event, not for compute time:

Event	Price
Run start	$0.002 per run
Page audited	$0.002 per HTML page
External link checked	$0.0005 per unique external URL

Included at no extra charge: the robots.txt, sitemap, llms.txt and HTTPS checks, and internal link checks. Not charged: pages that return errors, are not HTML (PDFs, images) or are blocked by robots.txt, and external links that time out, refuse the connection, block automated checks or are not allowed by robots.txt.

Example: a 200-page site with 300 unique external links costs 0.002 + 200 × 0.002 + 300 × 0.0005 = $0.552.

If a run reaches the maximum cost you set, the crawl stops, pages it did not get to are listed with status skipped, and only work that was done is charged. Charges are made at the end of the run, so an aborted run costs only the start event.

Limits and good manners

• Speed: requests to one host start at least 1 second apart, or further apart if robots.txt sets a Crawl-delay (up to 10 seconds). A 100-page audit takes about 2 to 3 minutes; 1,000 pages take about 20 minutes. Give long audits a run timeout to match. If a run gets close to its timeout, the crawl stops early and the results so far are saved.
• Robots.txt is respected by default, for page fetches and external link checks. The crawler identifies itself as PlainsourceSEOAudit.
• No JavaScript rendering. Pages are read as the server sends them. A site that builds its content in the browser shows little text, no links and missing tags; Google renders JavaScript, so results for such sites can differ from what Google sees.
• Status codes are reported as sent. Some single-page apps send 404 for pages that still render in a browser, and some sites send 403 to all bots. Those show up as errors or as blocked.
• What is checked: <a> and <area> links only. Image, script and stylesheet URLs are not checked.
• One site per run, up to 5,000 pages and 10,000 external links. Internal links beyond the page limit are counted in internalUrlsNotCrawled but not checked. Sitemaps: up to 50 files, 50 MB each, and 50,000 URLs.
• No proxies, logins or bot-check workarounds. LinkedIn, Facebook, Instagram, X and TikTok links are not checked, because their terms forbid automated access.
• Private networks are refused. Private and local addresses are not fetched, including through redirects.

FAQ

Why does a page show blocked or page-blocked? The server answered 401, 403, 429 or 503. Usually that is bot protection (Cloudflare and similar), which may also affect search engines. Open the URL in a browser to check.

Why are some external links "not checked"? They are over the limit you set, over 100 links to one site, on a site whose robots.txt does not allow checks, or on LinkedIn, Facebook, Instagram, X or TikTok.

Does it follow nofollow links? Links on pages marked nofollow for the whole page are not followed while robots.txt is respected. Individual rel="nofollow" links are still crawled and checked.

Can I audit a staging site that blocks all crawlers? Yes, if you own it: turn off Respect robots.txt. The staging site must be reachable from the internet.

Can I call it from code or an AI agent? Yes. Use the Apify API or the Apify MCP server with the same input. With the API client: client.actor("fernandosord/seo-site-audit").call({ startUrl: "https://www.example.com/" }).

Found a bug or need a check that is missing? Open an issue on this Actor's Issues tab.