# SEO Audit Crawler - Find What Broke Since Last Crawl (`that_red_bird/seo-audit-regression-crawler`) Actor

Find what broke on your site since the last crawl: pages that now 404, new noindex, changed canonicals, titles, H1s and schema, redirect chains and broken internal links, ranked by severity with a plain-English summary and Slack/Discord alert. Re-run after every release.

- **URL**: https://apify.com/that_red_bird/seo-audit-regression-crawler.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** SEO tools, Marketing, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SEO Audit Crawler - Find What Broke Since Last Crawl

Most SEO crawlers give you a 400-row report of everything that is wrong with a site. After a release,
a migration or a CMS change, the useful question is different: **what broke since the last crawl?**

This Actor crawls your site, remembers the result, and on every re-run tells you what got worse: a page
that was 200 and now 404s, a `noindex` that appeared, a canonical that now points elsewhere, a removed H1 or
title, structured data that vanished, new redirect chains and broken internal links. Findings are ranked
**critical / high / medium / low** and summarised in plain English at the top, with an optional Slack or
Discord alert. (Formerly "SEO Regression Auditor"; it also absorbs the page-level diff of the Site Diff Engine.)

### What it catches

| Category | What is compared with the last crawl | Severity |
|---|---|---|
| Status codes | A URL that was OK and now returns 4xx/5xx or is unreachable | critical |
| Indexability | New `noindex` (meta or `X-Robots-Tag`), new `robots.txt` disallow | critical |
| Canonicals | Canonical removed, or changed to another URL | high |
| Titles and H1 | Title or H1 removed (high), reworded (medium) | high / medium |
| Structured data | A JSON-LD type that disappeared, JSON-LD blocks that stopped parsing | high |
| Broken links | New broken internal links, with the pages that link to them | high |
| Redirects | New redirect chains (2+ hops) and loops | medium |
| Sitemap | Pages that left the sitemap | medium |
| Meta description | Removed (medium), reworded (low) | medium / low |
| Page text | Body text mostly rewritten (low) or shrunk by half (high) | low / high |
| Hygiene | Duplicate titles, length limits, heading order, orphan pages | medium / low |

Also reported: **fixed** issues (what got better) and a 0-100 health score with its change since last time.

### Sample output

```json
{
  "type": "summary", "site": "https://example.com", "isBaseline": false,
  "headline": "2 critical, 0 high, 1 medium, 0 low changes since the last crawl (score 83 -> 78).",
  "whatBroke": [
    "CRITICAL: https://example.com/usage was HTTP 200, now HTTP 404",
    "CRITICAL: NEW noindex on https://example.com/install",
    "MEDIUM: https://example.com/usage was in the sitemap last run and is gone now"
  ],
  "severityCounts": { "critical": 2, "high": 0, "medium": 1, "low": 0, "info": 0 },
  "score": 78, "previousScore": 83, "scoreDelta": -5
}
```

```json
{ "type": "change", "kind": "title-changed", "severity": "medium",
  "url": "https://example.com/changelog",
  "summary": "Title changed on https://example.com/changelog: \"Changelog\" -> \"Release notes\"",
  "before": "Changelog", "after": "Release notes" }
```

```json
{ "type": "issue", "severity": "high", "regression": "NEW", "category": "canonical",
  "key": "canonical-elsewhere:https://example.com/changelog", "detail": { "canonicalTargets": ["https://example.com/index"] } }
```

Row types: `summary` (one per site), `change` (page-level before/after), `issue` (current problems, marked
`NEW` / `EXISTING`), `fixed`, `page-error`, `error`.

### How it works

1. Crawl up to `maxPages` pages (sitemap first, topped up from internal links) and status-check up to `maxLinksChecked` more internal links, following redirects hop by hop.
2. Audit every page and store a compact snapshot: status, title, meta description, H1, canonical, robots directives, JSON-LD types, a fingerprint of the body text.
3. Compare with the snapshot saved by the previous run (named key-value store, one record per site).
4. Output ranked findings, update the stored baseline, optionally send the alert.

**First run = baseline.** It outputs everything it finds but reports no regressions. Schedule the Actor (daily, or
after every deploy via an Apify webhook/API call) and the second run onwards shows what changed.

### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | `https://example.com` | Sites to audit. Each has its own baseline |
| `maxPages` | 40 | Pages fetched and audited in full |
| `maxLinksChecked` | 150 | Extra internal links that only get a status check |
| `useSitemap` | true | Discover pages from sitemap.xml and detect orphan pages |
| `includePatterns` / `excludePatterns` | empty | Substring filters for internal URLs |
| `detectContentChanges`, `contentChangePercent` | true, 25 | Body text change detection and its threshold |
| `onlyRegressions` | false | After the baseline, output only changes and new/fixed issues |
| `resetBaseline` | false | Forget the stored crawl and start over |
| `webhookUrl`, `alertMinSeverity` | empty, high | Slack/Discord/any webhook; fires only on re-runs that find regressions |
| `titleMin/MaxLength`, `descMin/MaxLength` | 15/60, 50/160 | Length thresholds |
| `stateStoreName` | `seo-regression-auditor-state` | Name of the baseline store; use one per client |

Run with `{}` to try it on example.com.

### Pricing

Pay per event:

| Event | Price | When |
|---|---|---|
| Site audited | $0.01 | per site per run |
| Regression found | $0.01 | per **critical or high** regression that is new this run, **capped at 20 per site per run** ($0.20 max) |

Baseline runs, clean re-runs, and medium/low/info findings only cost the $0.01 site fee. A clean re-run of a 40-page
site is $0.01 total. Findings that are two views of the same fault (for example "H1 removed" and "missing H1" on one page) are billed once.

### Use cases

- **After every release:** schedule or trigger a run and get a short list instead of a spreadsheet.
- **Agencies:** one run, many client sites, one baseline store name per client.
- **Migrations and redesigns:** compare before and after, catch lost canonicals, redirects and noindex.
- **Watch someone else's site:** note when a competitor changes titles, H1s or structured data.

### FAQ

**Does it run JavaScript?** No. It audits the HTML a crawler receives without rendering. Content added by JavaScript is not seen (that is also what most AI and many search crawlers see first).

**How many pages can it audit?** Up to 2000 audited pages and 5000 status-checked links per site per run. Larger sites: use `includePatterns` per section.

**Is the score comparable with Lighthouse or other tools?** No. It is this Actor's own 0-100 health score (penalties per issue type) and is meant for comparing a site with its own past, not with other sites.

**Why was nothing reported on the first run?** The first run is a baseline. Run it again later.

**Can it miss a regression?** Yes, anything outside the crawled sample (`maxPages`, `maxLinksChecked`) and anything that needs JavaScript or a login. Body text changes are estimated with a similarity sketch, not an exact diff. Orphan-page checks are only as good as the sitemap and the pages crawled.

**Does it respect robots.txt?** The crawl is an owner-style audit and fetches the pages you point it at. Disallowed pages are reported as an indexability finding rather than skipped. Only use it on sites you are allowed to audit.

### Integrations

Works with Make, Zapier, n8n, Slack/Discord webhooks, the Apify API and scheduler, and Apify's MCP server for agents.
Trigger it from your CI/CD after a deploy with the Apify API, then read `summary.whatBroke`.

If it caught something for you, a review on the Store page helps others find it. Issues and ideas are welcome on the Actor page.

# Actor input Schema

## `startUrls` (type: `array`):

Full site URLs to crawl and audit, e.g. https://example.com. Each site is audited on its own and stored under its own baseline, so the second run of the same URL shows what broke since the first.

## `maxPages` (type: `integer`):

How many pages get fetched and audited in full (titles, canonicals, headings, schema, indexability, content). Prefers sitemap.xml URLs and tops up from internal links.

## `useSitemap` (type: `boolean`):

Discover pages from sitemap.xml (and sitemap indexes, one level deep) first. Also used for orphan-page detection under Indexability.

## `maxLinksChecked` (type: `integer`):

On top of the crawled pages, this many additional internal links found on those pages get a status-only request for the broken-link and redirect-chain audits (no content is stored for these).

## `includePatterns` (type: `array`):

Crawl and link-check only internal URLs containing one of these substrings, e.g. /blog/, /products/. Empty means the whole site. The start URL is always crawled.

## `excludePatterns` (type: `array`):

Skip internal URLs containing any of these substrings, e.g. /cart, ?sort=, /tag/.

## `detectContentChanges` (type: `boolean`):

Compare each page's visible text with the last crawl and report pages whose text was largely rewritten or shrank by half (type change, kind content-changed or content-shrunk). Digits are ignored so clocks and counters do not trigger it.

## `contentChangePercent` (type: `integer`):

Report a content-changed row when roughly this percentage of a page's text differs from the last crawl. Uses a compact similarity sketch, so it is an estimate.

## `onlyRegressions` (type: `boolean`):

After the baseline run, output only what changed since last time (page changes plus NEW and FIXED issues) and skip issues that were already there. The baseline run always outputs everything found.

## `resetBaseline` (type: `boolean`):

Forget the stored audit for these sites and start fresh — this run becomes the new baseline with no regressions reported.

## `webhookUrl` (type: `string`):

Slack, Discord or any webhook that accepts JSON. Receives one message per site, only when a later run finds regressions at or above the alert severity. Never fires on the baseline run. Empty disables it.

## `alertMinSeverity` (type: `string`):

Lowest severity that triggers the webhook. critical: lost indexability or a page that now errors. high: removed H1, title, canonical or structured data, broken links. medium and low are smaller changes.

## `titleMinLength` (type: `integer`):

A <title> shorter than this is flagged as too short.

## `titleMaxLength` (type: `integer`):

A <title> longer than this is flagged as too long (roughly where Google starts truncating it in results).

## `descMinLength` (type: `integer`):

A meta description shorter than this is flagged as too short.

## `descMaxLength` (type: `integer`):

A meta description longer than this is flagged as too long.

## `maxItems` (type: `integer`):

Hard cap on issue/fixed rows pushed per site. 0 = no cap.

## `concurrency` (type: `integer`):

How many requests to run in parallel while crawling and link-checking.

## `timeoutMs` (type: `integer`):

Per-request timeout for every crawl, link-check and redirect hop.

## `stateStoreName` (type: `string`):

Named key-value store holding the last crawl of each site (the baseline). Use a different name per client to keep their baselines separate. The default is kept from the earlier version of this Actor so existing baselines still work.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com"
  ],
  "maxPages": 40,
  "useSitemap": true,
  "maxLinksChecked": 150,
  "includePatterns": [],
  "excludePatterns": [],
  "detectContentChanges": true,
  "contentChangePercent": 25,
  "onlyRegressions": false,
  "resetBaseline": false,
  "webhookUrl": "",
  "alertMinSeverity": "high",
  "titleMinLength": 15,
  "titleMaxLength": 60,
  "descMinLength": 50,
  "descMaxLength": 160,
  "maxItems": 0,
  "concurrency": 5,
  "timeoutMs": 15000,
  "stateStoreName": "seo-regression-auditor-state"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://example.com"
    ],
    "maxPages": 40,
    "useSitemap": true,
    "maxLinksChecked": 150,
    "includePatterns": [],
    "excludePatterns": [],
    "detectContentChanges": true,
    "contentChangePercent": 25,
    "onlyRegressions": false,
    "resetBaseline": false,
    "webhookUrl": "",
    "alertMinSeverity": "high",
    "titleMinLength": 15,
    "titleMaxLength": 60,
    "descMinLength": 50,
    "descMaxLength": 160,
    "maxItems": 0,
    "concurrency": 5,
    "timeoutMs": 15000,
    "stateStoreName": "seo-regression-auditor-state"
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/seo-audit-regression-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://example.com"],
    "maxPages": 40,
    "useSitemap": True,
    "maxLinksChecked": 150,
    "includePatterns": [],
    "excludePatterns": [],
    "detectContentChanges": True,
    "contentChangePercent": 25,
    "onlyRegressions": False,
    "resetBaseline": False,
    "webhookUrl": "",
    "alertMinSeverity": "high",
    "titleMinLength": 15,
    "titleMaxLength": 60,
    "descMinLength": 50,
    "descMaxLength": 160,
    "maxItems": 0,
    "concurrency": 5,
    "timeoutMs": 15000,
    "stateStoreName": "seo-regression-auditor-state",
}

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/seo-audit-regression-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://example.com"
  ],
  "maxPages": 40,
  "useSitemap": true,
  "maxLinksChecked": 150,
  "includePatterns": [],
  "excludePatterns": [],
  "detectContentChanges": true,
  "contentChangePercent": 25,
  "onlyRegressions": false,
  "resetBaseline": false,
  "webhookUrl": "",
  "alertMinSeverity": "high",
  "titleMinLength": 15,
  "titleMaxLength": 60,
  "descMinLength": 50,
  "descMaxLength": 160,
  "maxItems": 0,
  "concurrency": 5,
  "timeoutMs": 15000,
  "stateStoreName": "seo-regression-auditor-state"
}' |
apify call that_red_bird/seo-audit-regression-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/seo-audit-regression-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/22bjhvH4qEoon4JT6/builds/L6VLizUscItyD8j4R/openapi.json