SEO Audit Crawler - Find What Broke Since Last Crawl

Most SEO crawlers give you a 400-row report of everything that is wrong with a site. After a release, a migration or a CMS change, the useful question is different: what broke since the last crawl?

This Actor crawls your site, remembers the result, and on every re-run tells you what got worse: a page that was 200 and now 404s, a noindex that appeared, a canonical that now points elsewhere, a removed H1 or title, structured data that vanished, new redirect chains and broken internal links. Findings are ranked critical / high / medium / low and summarised in plain English at the top, with an optional Slack or Discord alert. (Formerly "SEO Regression Auditor"; it also absorbs the page-level diff of the Site Diff Engine.)

What it catches

Category	What is compared with the last crawl	Severity
Status codes	A URL that was OK and now returns 4xx/5xx or is unreachable	critical
Indexability	New noindex (meta or X-Robots-Tag), new robots.txt disallow	critical
Canonicals	Canonical removed, or changed to another URL	high
Titles and H1	Title or H1 removed (high), reworded (medium)	high / medium
Structured data	A JSON-LD type that disappeared, JSON-LD blocks that stopped parsing	high
Broken links	New broken internal links, with the pages that link to them	high
Redirects	New redirect chains (2+ hops) and loops	medium
Sitemap	Pages that left the sitemap	medium
Meta description	Removed (medium), reworded (low)	medium / low
Page text	Body text mostly rewritten (low) or shrunk by half (high)	low / high
Hygiene	Duplicate titles, length limits, heading order, orphan pages	medium / low

Also reported: fixed issues (what got better) and a 0-100 health score with its change since last time.

Sample output

{
  "type": "summary", "site": "https://example.com", "isBaseline": false,
  "headline": "2 critical, 0 high, 1 medium, 0 low changes since the last crawl (score 83 -> 78).",
  "whatBroke": [
    "CRITICAL: https://example.com/usage was HTTP 200, now HTTP 404",
    "CRITICAL: NEW noindex on https://example.com/install",
    "MEDIUM: https://example.com/usage was in the sitemap last run and is gone now"
  ],
  "severityCounts": { "critical": 2, "high": 0, "medium": 1, "low": 0, "info": 0 },
  "score": 78, "previousScore": 83, "scoreDelta": -5
}

{ "type": "change", "kind": "title-changed", "severity": "medium",
  "url": "https://example.com/changelog",
  "summary": "Title changed on https://example.com/changelog: \"Changelog\" -> \"Release notes\"",
  "before": "Changelog", "after": "Release notes" }

{ "type": "issue", "severity": "high", "regression": "NEW", "category": "canonical",
  "key": "canonical-elsewhere:https://example.com/changelog", "detail": { "canonicalTargets": ["https://example.com/index"] } }

Row types: summary (one per site), change (page-level before/after), issue (current problems, marked NEW / EXISTING), fixed, page-error, error.

How it works

1. Crawl up to maxPages pages (sitemap first, topped up from internal links) and status-check up to maxLinksChecked more internal links, following redirects hop by hop.
2. Audit every page and store a compact snapshot: status, title, meta description, H1, canonical, robots directives, JSON-LD types, a fingerprint of the body text.
3. Compare with the snapshot saved by the previous run (named key-value store, one record per site).
4. Output ranked findings, update the stored baseline, optionally send the alert.

First run = baseline. It outputs everything it finds but reports no regressions. Schedule the Actor (daily, or after every deploy via an Apify webhook/API call) and the second run onwards shows what changed.

Input

Field	Default	What it does
startUrls	https://example.com	Sites to audit. Each has its own baseline
maxPages	40	Pages fetched and audited in full
maxLinksChecked	150	Extra internal links that only get a status check
useSitemap	true	Discover pages from sitemap.xml and detect orphan pages
includePatterns / excludePatterns	empty	Substring filters for internal URLs
detectContentChanges, contentChangePercent	true, 25	Body text change detection and its threshold
onlyRegressions	false	After the baseline, output only changes and new/fixed issues
resetBaseline	false	Forget the stored crawl and start over
webhookUrl, alertMinSeverity	empty, high	Slack/Discord/any webhook; fires only on re-runs that find regressions
titleMin/MaxLength, descMin/MaxLength	15/60, 50/160	Length thresholds
stateStoreName	seo-regression-auditor-state	Name of the baseline store; use one per client

Run with {} to try it on example.com.

Pricing

Pay per event:

Event	Price	When
Site audited	$0.01	per site per run
Regression found	$0.01	per critical or high regression that is new this run, capped at 20 per site per run ($0.20 max)

Baseline runs, clean re-runs, and medium/low/info findings only cost the $0.01 site fee. A clean re-run of a 40-page site is $0.01 total. Findings that are two views of the same fault (for example "H1 removed" and "missing H1" on one page) are billed once.

Use cases

• After every release: schedule or trigger a run and get a short list instead of a spreadsheet.
• Agencies: one run, many client sites, one baseline store name per client.
• Migrations and redesigns: compare before and after, catch lost canonicals, redirects and noindex.
• Watch someone else's site: note when a competitor changes titles, H1s or structured data.

FAQ

Does it run JavaScript? No. It audits the HTML a crawler receives without rendering. Content added by JavaScript is not seen (that is also what most AI and many search crawlers see first).

How many pages can it audit? Up to 2000 audited pages and 5000 status-checked links per site per run. Larger sites: use includePatterns per section.

Is the score comparable with Lighthouse or other tools? No. It is this Actor's own 0-100 health score (penalties per issue type) and is meant for comparing a site with its own past, not with other sites.

Why was nothing reported on the first run? The first run is a baseline. Run it again later.

Can it miss a regression? Yes, anything outside the crawled sample (maxPages, maxLinksChecked) and anything that needs JavaScript or a login. Body text changes are estimated with a similarity sketch, not an exact diff. Orphan-page checks are only as good as the sitemap and the pages crawled.

Does it respect robots.txt? The crawl is an owner-style audit and fetches the pages you point it at. Disallowed pages are reported as an indexability finding rather than skipped. Only use it on sites you are allowed to audit.

Integrations

Works with Make, Zapier, n8n, Slack/Discord webhooks, the Apify API and scheduler, and Apify's MCP server for agents. Trigger it from your CI/CD after a deploy with the Apify API, then read summary.whatBroke.

If it caught something for you, a review on the Store page helps others find it. Issues and ideas are welcome on the Actor page.