Website SEO Audit Crawler & Broken Link Checker

What does Website SEO Audit Crawler do?

Website SEO Audit Crawler is a fast, low-cost technical SEO audit tool and site crawler that runs in the cloud. Give it a website and it crawls the pages the way a search engine bot does, then returns one clean row per page with everything an SEO audit needs: HTTP status codes, redirect chains, broken links (internal and external), title and meta description length, H1 headings, canonical tags, noindex and X-Robots-Tag directives, hreflang, Open Graph tags, JSON-LD structured data types, word count, images missing alt text, mixed content, sitemap coverage and duplicate titles and descriptions. Each page also gets a list of machine-readable issue codes and a 0–100 SEO score.

On top of the per-page rows, every run writes a site summary (the SUMMARY record) with every issue type, its severity, the number of affected pages, five example URLs and a plain-English how to fix note, plus groups of duplicate titles and meta descriptions, all broken link and image targets, orphan sitemap URLs and pages blocked by robots.txt. That is your audit report in one JSON file.

You can also use it as a plain broken link checker: internal and external links found during the crawl (up to Max link checks, 500 by default) and the images on each page are checked, and 404s, 5xx errors, dead domains and redirect chains are listed per page and in the summary. Links that only fail because the target blocks bots (Facebook, LinkedIn, Wikipedia, Cloudflare challenges) are listed separately as unverified, so they don't flood the report with false alarms.

It needs no browser, no login and no API key. It reads robots.txt and the XML sitemaps itself, so a 500-page audit usually finishes in a minute or two and costs about $4.

Typical uses:

• SEO agencies: run a technical audit for a new client in minutes, or schedule a weekly crawl of every client site and catch new 404s, noindex mistakes and missing titles before rankings drop.
• In-house SEO and content teams: find thin pages, duplicate titles, pages missing from the sitemap and images without alt text.
• Developers and QA: check a site after a migration or release for broken internal links, redirect chains and pages that suddenly return 5xx.
• Lead generation: audit prospects' websites and open the sales conversation with concrete issues ("your sitemap lists 40 URLs that return 404").

Why use Website SEO Audit Crawler?

• One row per page, ready for a spreadsheet. Flat columns such as statusCode, titleLength, metaDescriptionLength, h1Count, canonicalUrl, indexable, inSitemap, brokenLinksCount and seoScore work directly in Google Sheets, Excel or Looker Studio.
• Issues are already diagnosed. Instead of raw HTML you get issue codes such as missing_title, duplicate_meta_description, broken_internal_links, redirect_chain, canonical_points_elsewhere, noindex_in_sitemap and not_in_sitemap, each with a severity (error, warning, notice).
• Broken link checker included. Every internal link target is verified. External links are checked too (HEAD request with a GET fallback), and link checks are free: you only pay per audited page.
• Sitemap coverage and orphan pages. The crawler reads the sitemaps listed in robots.txt (including sitemap indexes and gzipped sitemaps), audits orphan URLs that are in the sitemap but not linked from the site, and flags indexable pages that are missing from the sitemap.
• Three modes. Crawl a whole site, audit only the sitemap URLs, or audit an exact list of URLs (a bulk URL status and metadata checker).
• Polite and safe. robots.txt rules and crawl-delay are always respected, concurrency is limited, and the crawl stays on the site you give it.
• Schedulable and API-ready. Run it daily or weekly with Apify schedules, connect it to Make, Zapier or n8n, or let an AI agent call it through the Apify API or MCP server.
• Never loses results. The crawler stops itself before the run timeout and always saves the pages it has audited plus the summary.

What data can you extract?

Field	Example	Description
url	https://www.example.com/blog/	The audited URL
statusCode	200	First HTTP status (e.g. 301 for a redirect)
finalUrl / finalStatusCode	https://www.example.com/blog/, 200	Where redirects end and the status there
redirectChain	[{"url": "http://example.com/", "statusCode": 301}]	Each hop of a redirect
responseTimeMs / pageSizeKb	412, 88.4	Server response time and HTML size
depth / foundOn	2, https://www.example.com/	Clicks from the start URL and the page that linked to it (or sitemap)
title / titleLength	Blog - Example, 14	<title> and its length
metaDescription / metaDescriptionLength	News and guides..., 132	Meta description and its length
h1 / h1Count / h2Count	["Our blog"], 1, 6	Headings
canonicalUrl	https://www.example.com/blog/	<link rel="canonical">
metaRobots / xRobotsTag	noindex, follow	Robots directives from HTML and HTTP headers
indexable	true	2xx, not redirected, no noindex, canonical is self or missing
inSitemap	true	Whether the URL is listed in the site's sitemap
lang / viewport	en, true	<html lang> and mobile viewport tag
hreflang	[{"lang": "sv", "url": "https://www.example.com/sv/"}]	Alternate language versions
openGraph / twitterCard	{"title": "...", "image": "..."}	Social sharing tags
jsonLdTypes / microdataTypes	["Organization", "BreadcrumbList"]	Structured data (schema.org) types found
jsonLdInvalid	0	JSON-LD blocks that are not valid JSON
wordCount	812	Visible words (scripts and styles removed)
imagesCount / imagesMissingAlt / imagesMissingAltSample	14, 3, [...]	Image alt text check
mixedContent	["http://cdn.example.com/a.js"]	Insecure resources on HTTPS pages
internalLinksCount / externalLinksCount	48, 7	Outgoing links
brokenLinks / brokenLinksCount	[{"url": ".../old", "status": 404, "anchorText": "Pricing", "internal": true}]	Broken link targets on this page (status is the HTTP code, or dns_error, connection_error, ssl_error, redirect_loop)
brokenImages / brokenImagesCount	[{"url": ".../hero.jpg", "status": 404}]	Images on this page that return an error
unverifiedLinks / unverifiedLinksCount	[{"url": "https://www.facebook.com/acme", "status": 400}]	Links that could not be verified because the target blocks bots or timed out; not counted as broken
jsRendered	false	true when the server HTML has almost no text and the page is built by JavaScript
issues	["title_too_long", "missing_h1"]	Issue codes (see the FAQ for the full list)
errorsCount / warningsCount	0, 2	Issue counts by severity
seoScore	84	0–100 page score: −25 per error, −8 per warning, −2 per notice (null for pages that blocked the audit)
error	null	Fetch error (timeouts, DNS, redirect loops)
checkedAt	2026-10-06T12:00:00+00:00	Timestamp

The SUMMARY record (key-value store) contains: pagesAudited, averageSeoScore, pagesBlocked, statusCodes, indexablePages, issues (each issue with severity, number of pages, up to five examplePages and a howToFix note), duplicateTitles, duplicateMetaDescriptions, brokenLinks, brokenImages, unverifiedLinks, sitemap (URLs in sitemaps, sitemap URLs not audited), blockedByRobotsTxt, linkTargetsChecked and linkTargetsNotChecked (over the Max link checks budget).

How to use Website SEO Audit Crawler (step by step)

1. Click Try for free (you need a free Apify account; new accounts get monthly free credits).
2. Enter the homepage of the site in Website or page URLs, e.g. https://www.example.com/.
3. Set Max pages. Start with 50–200 to get a feel for the site, then raise it for a full audit.
4. Pick a Crawl mode: crawl the site (default), sitemap URLs only, or only the URLs you entered.
5. Click Start. Watch the log: it reports sitemap discovery, crawl progress and link checks.
6. Open the SEO audit view in the Output tab, sort by seoScore or filter by issue, and download as JSON, CSV, Excel or HTML. Open the Site summary output for the report.

Input example

{
  "startUrls": ["https://www.example.com/"],
  "maxPages": 500,
  "crawlMode": "crawl",
  "checkExternalLinks": true,
  "excludeUrlPatterns": ["/tag/", "\\?replytocom="],
  "onlyPagesWithIssues": false
}

Audit an exact list of landing pages instead:

{
  "startUrls": ["https://www.example.com/pricing", "https://www.example.com/features", "http://example.com/old-page"],
  "crawlMode": "list"
}

Output example

[
  {
    "url": "https://www.python.org/about/",
    "statusCode": 200,
    "finalUrl": "https://www.python.org/about/",
    "redirectChain": [],
    "contentType": "text/html",
    "responseTimeMs": 21,
    "depth": 0,
    "foundOn": null,
    "inSitemap": null,
    "title": "About Python™ | Python.org",
    "titleLength": 26,
    "metaDescription": "The official home of the Python Programming Language",
    "metaDescriptionLength": 52,
    "h1": ["Python is powerful... and fast; plays well with others; runs everywhere; is friendly & easy to learn; is Open."],
    "h1Count": 1,
    "canonicalUrl": null,
    "indexable": true,
    "lang": "en",
    "jsonLdTypes": ["WebSite"],
    "wordCount": 698,
    "imagesMissingAlt": 0,
    "brokenLinksCount": 0,
    "issues": ["title_too_short", "meta_description_too_short", "duplicate_meta_description", "missing_canonical"],
    "errorsCount": 0,
    "warningsCount": 1,
    "seoScore": 86
  },
  {
    "url": "https://www.python.org/psf/",
    "statusCode": 301,
    "finalUrl": "https://www.python.org/psf-landing/",
    "finalStatusCode": 200,
    "redirectChain": [{"url": "https://www.python.org/psf/", "statusCode": 301}],
    "issues": ["redirect"],
    "seoScore": 98
  }
]

Fields are shortened here. Every run also saves the site report as the SUMMARY record. Results can be exported to CSV, Excel, JSON, XML or HTML, or read via the API.

How much does it cost?

The actor uses pay-per-event pricing, so you only pay for pages that were actually audited.

Event	Price
Actor start	$0.0005 per run
Page audited	$0.008 per page ($8 per 1,000 pages)
Link checks, sitemap parsing, SUMMARY report	free
Pages that could not be reached at all (DNS errors, timeouts) or that blocked the audit (401, 403, 429)	free

Higher Apify plans get automatic volume discounts on the per-page price (Bronze −15%, Silver −30%, Gold and above −45%).

Price change: the per-page price is $0.008 from 26 October 2026 (it was $0.002 until then). The start fee and the free items are unchanged. Full-site audit tools in the Store charge up to $40 per 1,000 pages.

Worked examples:

• A 300-page company website: 300 × $0.008 ≈ $2.40 per full audit.
• Weekly audits of 10 client sites with 500 pages each: 10 × 500 × 4 ≈ 20,000 pages ≈ $160 per month (about $88 on Gold-tier plans).
• Daily check of 50 key landing pages in list mode: 50 × 30 = 1,500 pages ≈ $12 per month.

Platform compute is small because there is no browser: a 500-page crawl typically uses a few cents of compute or less on top. Use Only output pages with issues to skip (and not pay for) clean pages, and Max pages as a hard cost cap.

Use cases

• SEO agency monthly reporting. Schedule a crawl per client, push the SUMMARY into Google Sheets or Looker Studio, and track averageSeoScore and error counts over time.
• Site migrations. Before launch, audit the old URLs in list mode and check every one redirects in one hop to a 200 page. After launch, crawl the new site for broken internal links and redirect chains.
• Content cleanup. Filter on thin_content, duplicate_title and duplicate_meta_description to find pages to merge, rewrite or noindex.
• Technical health monitoring. A daily run with onlyPagesWithIssues and a webhook alerts you when pages start returning 4xx/5xx or a release accidentally adds noindex.
• Structured data inventory. Use jsonLdTypes to see which templates carry Product, Article, FAQPage or BreadcrumbList markup and which are missing it.
• Sales prospecting for web agencies. Audit a list of prospect homepages in list mode and mention concrete issues in outreach.
• AI agents. An agent can call the actor with a URL and reason over the issue codes to suggest fixes.

Use with AI agents, MCP and the API

Run it from the command line (replace <YOUR_TOKEN>):

curl -X POST "https://api.apify.com/v2/acts/teddythebuilder~website-seo-audit/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://www.example.com/"], "maxPages": 100, "onlyPagesWithIssues": true}'

Or from Python with apify-client:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_TOKEN>")
run = client.actor("teddythebuilder/website-seo-audit").call(run_input={
    "startUrls": ["https://www.example.com/"],
    "maxPages": 200,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page["errorsCount"]:
        print(page["url"], page["issues"])
summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("SUMMARY")["value"]
print(summary["averageSeoScore"], summary["issues"][:5])

AI agents (Claude, ChatGPT, Cursor and others) can discover and run this actor through the Apify MCP server (https://mcp.apify.com). Ask your agent to "audit example.com for technical SEO issues with the Website SEO Audit Crawler".

Integrations

Connect the actor to Make, Zapier, n8n, Google Sheets, Slack, Airbyte or your own backend with webhooks. A common setup: schedule a weekly run, send the SUMMARY to Slack, and append the pages with errors to a Google Sheet for the content team.

Data source and legality

The actor fetches public web pages exactly like a search engine crawler, identifies itself with a clear user agent (ApifyWebsiteSeoAudit), always respects robots.txt and crawl-delay, and keeps concurrency low. It collects page metadata and technical SEO signals only. It does not log in, does not bypass paywalls or bot protection, and does not extract personal data. Audit sites you own or are allowed to audit, and keep the crawl size reasonable for small sites.

Limitations

• No JavaScript rendering. The audit sees the HTML the server returns, as most crawlers do on the first pass. Sites that build all content with client-side JavaScript will show low word counts and missing tags. This is also a useful signal, because search engines index the raw HTML first.
• Bot protection. Pages behind Cloudflare challenges or similar return 403/429. The actor does not try to get around bot protection. Such pages are reported with the access_blocked issue and are free; when the site answers 429 the crawler pauses and retries twice first. Many Shopify stores rate-limit automated audits this way, so a Shopify audit may come back mostly blocked. For links to other sites, bot-blocking answers are listed as unverifiedLinks, not as broken.
• Sitemaps are read up to 30 files and 100,000 URLs per run, with a 15-second discovery budget.
• Very large sites: crawl in sections with includeUrlPatterns, or raise the run timeout and memory.

FAQ

Which issues are detected? Errors: http_4xx, http_5xx, fetch_failed, redirect_loop, missing_title, broken_internal_links. Warnings: access_blocked (401/403/429/999), broken_images, title_too_long (>60 characters), missing_meta_description, meta_description_too_long (>160), missing_h1, broken_external_links, images_missing_alt, slow_response, mixed_content, duplicate_title, duplicate_meta_description, invalid_json_ld, noindex_in_sitemap, canonical_to_non_200, redirect_chain. Notices: title_too_short (<30), meta_description_too_short (<70), multiple_h1, missing_canonical, canonical_points_elsewhere, noindex, thin_content (<200 words), missing_lang, missing_viewport, missing_open_graph, not_in_sitemap, redirect, url_too_long, non_html, js_rendered_content (almost no text without JavaScript). The SUMMARY explains each one with a how-to-fix note.

Does it work on the free Apify plan? Yes. A 20-page test run costs about $0.16, well inside the monthly free credits.

How do I stop it from crawling filters, tags or search pages? Add regular expressions to Exclude URLs matching, e.g. /tag/, sort= or \?page=.

Why is a 403 link not reported as broken? 401, 403, 429 and 999 usually mean the target blocks bots (LinkedIn, Wikipedia, some CDNs), not that the page is gone, so they are not counted as broken links. For links to other sites the same goes for 400 and 406 (Facebook and some news sites answer bots that way), timeouts and servers that hang up without answering. These are listed in unverifiedLinks so you can check them by hand. Dead domains (dns_error), refused connections, TLS errors, 404/410 and 5xx are counted as broken.

Can I audit several sites in one run? Yes. Add several start URLs; each one's host is in scope. For clean per-site reports, run one site per run (for example with an Apify task per client).

How fresh is the data? Every run fetches the pages live, so results reflect the site at run time.

Can you add a check I need? Yes. Open an issue with the check you want. If the actor saves you time, a rating on the Store page helps others find it.

Related actors

• Website to Markdown: the main content of every page as clean Markdown for LLMs, RAG and content analysis.
• Domain WHOIS & DNS API: bulk WHOIS/RDAP registrar and expiry dates, domain availability, DNS records, SPF/DMARC email security grade and SSL certificate expiry in one row per domain.
• Tech Stack Detector: find the CMS, ecommerce platform, analytics and marketing tools a website uses. Pair it with this audit for a full website report.
• Career Site Jobs API: jobs straight from company career sites on Greenhouse, Lever, Ashby, Workable, Recruitee and Teamtailor.
• Nordic Company Registry API: companies from Sweden, Norway and Finland with industry codes, for building lead lists to audit.
• Swedish Jobs Scraper and Nordic Tenders Scraper.

Changelog and support

• 1.1.3 (2026-10-24): links to social networks (Twitter/X, Facebook, LinkedIn, Instagram...) and 503 and Cloudflare 520-526 answers from external sites are now listed as unverifiedLinks, not broken (X share buttons answer bots with HTTP 520). Pages that could not be fetched at all (DNS, TLS, connection) no longer get an SEO score, so they don't drag down the average.
• 1.1 (2026-10-18): quality update after hand-checking audits of six real sites. Bot-blocked and timed-out links are now unverifiedLinks instead of broken (Facebook 400, BuzzFeed 406, Tumblr hang-ups, share buttons that redirect off-site were false alarms). Broken links say why (dns_error, connection_error, ssl_error, redirect_loop). New checks: broken_images, access_blocked (free, with automatic slow-down and retry on 429) and js_rendered_content. HEAD answers of 400+ are confirmed with a GET. The SUMMARY now has a how-to-fix note and example pages per issue, pagesBlocked and linkTargetsNotChecked. Default Max link checks raised from 100 to 500.
• 1.0 (2026-10): first release. Crawl, sitemap and list modes; 34 issue checks with severities and a 0–100 score; free internal and external link checking; sitemap coverage and orphan pages; duplicate title and description detection; site SUMMARY report.

Questions or bugs? Open an issue. We reply within 24 hours.