Website SEO Audit & Regression Monitor

Audit complete public websites for deterministic technical SEO issues and monitor exactly which verified problems appear, recover, or change between scheduled runs.

This Actor is an HTTP-first Website SEO Audit and Technical SEO Audit tool for agencies, in-house SEO teams, developers, ecommerce teams, CI pipelines and AI agents. It does not invent a 0–100 score and does not promise rankings. Every finding has a stable issue ID, compact evidence and a documented condition. Monitor mode compares only complete trustworthy observations with the last successful Apify KVS state.

Why run it twice?

Use a one-time snapshot before a release, then schedule monitor mode daily, weekly, or after each deployment. The first successful run is a quiet BASELINE. Later runs emit webhook-ready events such as:

• BECAME_NOINDEX, BECAME_NON_200, TITLE_REMOVED, CANONICAL_CHANGED;
• BROKEN_LINK_INTRODUCED, DUPLICATE_TITLE_INTRODUCED;
• BROKEN_LINK_RESOLVED, DUPLICATE_TITLE_RESOLVED, and other recoveries;
• PAGE_ADDED, PAGE_REMOVED_PENDING, and conservative PAGE_REMOVED_CONFIRMED.

A blocked page, failed sitemap, truncated crawl, parser failure, or partial link check is never an SEO regression and never overwrites the previous good state.

Website SEO audit use cases

• Release and CI checks: audit production or staging after a deployment and route verified noindex, non-200, title, canonical, sitemap, duplicate-metadata, and broken-link regressions to the team that owns the change.
• Agency client monitoring: run up to 50 sites in one batch, keep each failure isolated, and use externalId to join Dataset rows to a client, property, or environment.
• Technical SEO inventories: export normalized page facts and deterministic issue evidence to JSON, CSV, Excel, dashboards, or an API workflow without interpreting an opaque score.
• Recovery verification: receive explicit recovery events after a fix instead of diffing two full exports by hand.

Reusable Task examples

• Run a Full Website Technical SEO Audit: replace one public website URL, choose the page ceiling and export deterministic page findings from the pages view.
• Monitor Technical SEO Regressions: keep the same Task, URL, scope and audit profile; the first complete run creates a quiet baseline and later runs emit events in changes.
• Bulk Website SEO Audit for Agencies: replace the reserved sample domains and externalId values with a client portfolio, then review one isolated status and summary per site.

The examples use bounded charge and Dataset-item limits. They remain separate workflows: a full single-site inventory, a recurring release monitor, and a reusable multi-client batch.

Quick start: basic SEO audit

{
  "sites": [
    {"url": "https://example.com", "externalId": "production"}
  ],
  "mode": "snapshot",
  "outputMode": "all",
  "maxPagesPerSite": 100,
  "respectRobotsTxt": true,
  "auditProfile": "standard"
}

Snapshot mode always emits the per-site row plus every page observation. It never reads or writes monitor state.

Full sitemap audit

{
  "sites": [{"url": "https://shop.example.com"}],
  "mode": "snapshot",
  "outputMode": "all",
  "maxPagesPerSite": 1000,
  "includePatterns": ["*/products/*", "*/collections/*"],
  "excludePatterns": ["*/cart*", "*?preview=*"],
  "auditProfile": "standard",
  "proxyConfiguration": {"useApifyProxy": false}
}

Discovery order is robots.txt sitemap declarations → conventional /sitemap.xml → bounded same-site internal-link crawl. Sitemap indexes, .xml.gz, repeated URLs and recursion are handled with strict file, depth, compressed-body and decompressed-body limits. Hitting a limit makes discovery incomplete instead of pretending that the returned URL prefix is the entire site.

The supplied start page is added when it is not in a sitemap and the page budget has room. includePatterns and excludePatterns are case-sensitive globs matched against both the full URL and its path.

Monitor mode

{
  "sites": [
    {"url": "https://example.com", "externalId": "production"},
    {"url": "https://staging.example.com", "externalId": "staging"}
  ],
  "mode": "monitor",
  "outputMode": "changesOnly",
  "maxPagesPerSite": 1000,
  "respectRobotsTxt": true,
  "auditProfile": "standard"
}

changesOnly emits one status row per supplied site, event rows, and any failed/partial/blocked page rows needed to diagnose why state was retained. Use all when you also need the full trustworthy page inventory on every monitor run. A broken site is isolated from the rest of the batch.

Per-site status is explicit:

Status	Meaning	Can advance state?
SUCCESS	Discovery and every requested audit surface are trustworthy	yes
PARTIAL	Some pages, discovery, or broken-link checks are incomplete	no
FAILED	The site could not be verified	no
BLOCKED	The target or its robots policy blocked the check	no
INVALID_SITE	URL syntax or the public-network safety boundary rejected input	no

NO_RESULTS is represented by a successful, complete site row with zero pages after intentional filters. It is never inferred from a transport or parser failure.

Quiet baseline

The first complete monitor run stores a compact last-successful state and returns:

{
  "record_type": "site",
  "status": "SUCCESS",
  "monitoring_status": "BASELINE",
  "summary": {"events": 0},
  "state_saved": true
}

Detail rows are delivered and any pay-per-event charge succeeds before the KVS candidate is committed. The site row is written last, so state_saved reports the actual per-site outcome. RUN_SUMMARY.states_saved and states_not_advanced provide the run totals.

Regression event: became noindex

{
  "record_type": "event",
  "event_type": "SEO_REGRESSION",
  "issue_id": "BECAME_NOINDEX",
  "severity": "critical",
  "source_url": "https://example.com/products/widget",
  "before": true,
  "after": false,
  "changed_fields": ["indexable", "issues"]
}

Broken-link regression

Internal links are extracted while pages are audited, deduplicated globally, and each unique target is checked at most once per run. An unverified target is unknown, not broken.

{
  "event_type": "SEO_REGRESSION",
  "issue_id": "BROKEN_LINK_INTRODUCED",
  "severity": "critical",
  "before": null,
  "after": [
    {"url": "https://example.com/old-offer", "http_status": 404}
  ],
  "changed_fields": ["issues.BROKEN_INTERNAL_LINK"]
}

SEO recovery

{
  "event_type": "SEO_RECOVERY",
  "issue_id": "BROKEN_LINK_RESOLVED",
  "severity": "critical",
  "before": [
    {"url": "https://example.com/old-offer", "http_status": 404}
  ],
  "after": null,
  "changed_fields": ["issues.BROKEN_INTERNAL_LINK"]
}

Deterministic audit rules

Thresholds are part of audit_rules_version=1.1.0: title 30–60 characters and meta description 70–160 characters. Whitespace is normalized before length checks. Changing a material condition bumps the rules version and causes a quiet RULES_REBASELINE, never hundreds of deployment-created regressions.

Issue ID	Severity	Exact condition
HTTP_STATUS_ERROR	critical	verified final status is outside 200–399
REDIRECTED	info	one or more validated HTTP redirects changed the requested URL
NOINDEX	critical	server HTML meta robots contains noindex
X_ROBOTS_NOINDEX	critical	final X-Robots-Tag contains noindex
TITLE_MISSING	critical	normalized <title> is absent/empty
TITLE_TOO_SHORT / TITLE_TOO_LONG	warning	title is below 30 / above 60 characters
META_DESCRIPTION_MISSING	warning	normalized meta description is absent/empty
META_DESCRIPTION_TOO_SHORT / META_DESCRIPTION_TOO_LONG	warning	description is below 70 / above 160 characters
H1_MISSING / MULTIPLE_H1	warning	zero / more than one <h1> in server HTML
CANONICAL_MISSING	warning	canonical link is absent/empty
CANONICAL_INVALID	critical	canonical cannot resolve to a safe HTTP(S) URL
CANONICAL_EXTERNAL	warning	canonical host differs from the final page host
LANG_MISSING	warning	<html lang> is absent/empty
VIEWPORT_MISSING	warning	non-empty viewport meta tag is absent
HREFLANG_INVALID	warning	language syntax or alternate HTTP(S) URL is invalid
HREFLANG_MISSING_RETURN	warning	both pages were audited, but the alternate does not point back
STRUCTURED_DATA_MISSING	info	no valid JSON-LD @type was extracted
STRUCTURED_DATA_INVALID_JSON	warning	at least one JSON-LD block is invalid JSON
IMAGE_ALT_MISSING	warning	an image lacks the alt attribute; alt="" is not treated as missing
BROKEN_INTERNAL_LINK	critical	a deduplicated same-site link has a verified HTTP status ≥400
DUPLICATE_TITLE	warning	two or more indexable audited pages have the same case-folded title
DUPLICATE_META_DESCRIPTION	warning	two or more indexable audited pages have the same case-folded description
MULTIPLE_URLS_SAME_CANONICAL	warning	two or more audited URLs share one canonical URL
SITEMAP_URL_NON_200	critical	a sitemap page has a verified non-2xx status
SITEMAP_URL_NOINDEX	critical	a sitemap page is deterministically non-indexable
INDEXABLE_URL_NOT_IN_SITEMAP	warning	emitted only when both sitemap and internal crawl scope are complete
ORPHAN_SITEMAP_URL	warning	emitted only when the complete internal-link graph cannot reach a sitemap URL

The core profile keeps only HTTP status, redirects, indexability, title, meta description, H1 and canonical presence. standard enables every rule above and bounded broken-link checks.

Page record contract

Every successful page observation contains:

record_type, source, source_id, source_url, scraped_at, schema_version, fingerprint
site_id, external_id, status, http_status, final_url
title, meta_description, h1, headings, canonical
robots_meta, x_robots_tag, indexable, hreflang, schema_types
internal_links_count, external_links_count, images_count, images_without_alt
issues, audit_rules_version, in_sitemap, proxy_used, error

fingerprint hashes normalized semantic SEO fields and issue evidence. It excludes timestamps, latency, run IDs, request IDs and transient headers. Dataset rows are append-only observations.

Removal and regression safety

PAGE_REMOVED_CONFIRMED requires two consecutive complete successful observations in which a previously present page is absent. The first absence emits PAGE_REMOVED_PENDING. Failed, partial, blocked, truncated, sitemap-error and parser-error runs do not increment confirmation.

Only two trustworthy successful page observations are compared. A Cloudflare 403 or challenge therefore returns BLOCKED/PARTIAL; it cannot emit BECAME_NON_200, TITLE_REMOVED, H1_REMOVED, or STRUCTURED_DATA_REMOVED. If discovery changes between sitemap and internal-link scope, the Actor returns DISCOVERY_CHANGED and retains the old state instead of comparing incompatible page sets.

State uses one named Apify KVS. It stores compressed, immutable versions of compact last-successful page fields, site issue keys and removal tombstones. The newest observation wins even when overlapping runs finish out of order; five versions are retained per site and audit scope. Saved Apify Tasks are isolated from each other automatically; direct Actor runs share the direct-run scope. There is no external database and no raw-HTML history.

The default Dataset contains site, page, and event records. Apify Dataset views project different field sets but cannot filter rows by record_type; use that field in API consumers when you need only one record kind.

Robots, proxies, and responsible use

• respectRobotsTxt defaults to true; robots is fetched once per site and cached.
• A missing robots file (404/410) permits normal public crawling. A robots timeout/5xx is unknown, not an empty allow-all policy.
• Sitemap parsing does not imply permission to fetch every listed URL; disallowed page paths are not requested.
• Direct HTTP is always first. An optional customer-selected Apify Proxy is used only after a blocked or transport-failed direct request. Custom proxy URLs are rejected; residential proxy escalation is never automatic.
• Login, CAPTCHA solving, paywall bypass, account-dependent pages and private-network access are not supported.
• Audit only public websites you are authorized to test. Review the target's terms and use a reasonable page limit and schedule. The Actor extracts technical metadata, not contact or profile data.

Static HTML limitation

The Actor audits the HTML and headers returned by the server; it does not execute JavaScript. Client-only metadata can therefore be absent from results. This is intentional: a 40-site PoC found server-delivered SEO metadata on 38 of 39 successful ordinary homepages, so browser-by-default cost was not justified. Core Web Vitals, Google Search Console data, ranking positions and rendered-only content are outside scope.

Daily/weekly schedule example

Save the monitor input as an Actor Task, or create a schedule through the API. This example runs the Actor daily at 06:00 UTC; replace the placeholders and keep the API token in an environment variable.

$body = @{
  name = "daily-website-seo-regressions"
  isEnabled = $true
  isExclusive = $true
  cronExpression = "0 6 * * *"
  timezone = "UTC"
  actions = @(
    @{
      type = "RUN_ACTOR"
      actorId = "YOUR_ACTOR_ID"
      runInput = @{
        body = '{"sites":[{"url":"https://example.com","externalId":"production"}],"mode":"monitor","outputMode":"changesOnly","maxPagesPerSite":1000,"respectRobotsTxt":true,"auditProfile":"standard"}'
        contentType = "application/json; charset=utf-8"
      }
      runOptions = @{ build = "latest"; memoryMbytes = 1024; maxTotalChargeUsd = 5 }
    }
  )
} | ConvertTo-Json -Depth 8

Invoke-RestMethod -Method Post `
  -Uri "https://api.apify.com/v2/schedules" `
  -Headers @{ Authorization = "Bearer $env:APIFY_TOKEN" } `
  -ContentType "application/json" `
  -Body $body

Change the cron expression to 0 6 * * 1 for Monday at 06:00 UTC.

Webhook regression alert example

Apify webhooks fire on run status, not on an individual Dataset event. Point a successful-run webhook at your alert receiver; the receiver reads resource.defaultDatasetId, filters record_type=event, and sends an alert only when event_type=SEO_REGRESSION.

$hooks = @(
  @{
    eventTypes = @("ACTOR.RUN.SUCCEEDED")
    requestUrl = "https://alerts.example.com/apify/seo-regressions"
    payloadTemplate = '{"runId":"{{resource.id}}","datasetId":"{{resource.defaultDatasetId}}","storeId":"{{resource.defaultKeyValueStoreId}}"}'
  }
) | ConvertTo-Json -Compress
$encoded = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($hooks))

Invoke-RestMethod -Method Post `
  -Uri "https://api.apify.com/v2/actors/YOUR_ACTOR_ID/runs?webhooks=$([Uri]::EscapeDataString($encoded))" `
  -Headers @{ Authorization = "Bearer $env:APIFY_TOKEN" } `
  -ContentType "application/json" `
  -Body '{"sites":[{"url":"https://example.com"}],"mode":"monitor","outputMode":"changesOnly"}'

API run

curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~website-seo-audit-monitor/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sites":[{"url":"https://example.com"}],"mode":"snapshot","outputMode":"all","maxPagesPerSite":100}'

Use with AI agents through Apify MCP

Expose the Actor as a typed tool in an authenticated MCP-compatible client:

https://mcp.apify.com?tools=automa-flow/website-seo-audit-monitor

Example prompt:

Run automa-flow/website-seo-audit-monitor for these public websites. Return
deterministic technical SEO findings grouped by severity and stable issue ID,
keep incomplete crawls explicit, and cite each affected page URL.

After publication the same URL becomes discoverable to Store users. The Actor does not invent an SEO score, rankings, Core Web Vitals, or JavaScript-rendered metadata.

Cost and billing

The implementation bills two units. page-check is one successfully verified page, including an unchanged monitor page. site-check is one website that produced at least one verified page, and it covers the fixed per-site work: robots.txt, sitemap discovery, sitemap-index traversal and monitor state comparison. Neither unit bills retries, FAILED, BLOCKED, INVALID_SITE, duplicate internal-link checks, or proxy fallback attempts. A website that yields no verified page is free entirely, and regression events are not separately double-billed.

The launch price is $0.003 per successful page check ($3 per 1,000 pages) plus $0.01 per successful website check, with platform usage included and no run-start fee. A one-site 100-page audit therefore has a maximum event price of $0.31, a 1,000-page audit $3.01, and a 10,000-page audit $30.01. On a 1,000-page site the website check is 0.3% of the bill; on a 20-page blog monitor it is the part that pays for the sitemap and state work. RUN_SUMMARY.charged_site_checks and RUN_SUMMARY.charged_page_checks are the authoritative logical charge counts. Run economics are also recorded in RUN_SUMMARY; use scripts/benchmark.py for the controlled CPU/memory lower bound.

The Actor checks billing capacity before opening the next site's HTTP client. If no capacity remains, that batch item is returned as FAILED with CHARGE_LIMIT_REACHED, makes zero target requests, and cannot advance monitor state. When a site's exact page count is not known until after discovery, the Actor treats that site atomically: either its website check and every successful page check are charged and delivered together, or nothing is charged for that site and its previous monitor state is retained.

Website SEO audit FAQ

Does this SEO audit execute JavaScript?

No. It audits server-delivered HTML and HTTP headers. This keeps scheduled runs fast and deterministic, but client-only metadata is outside scope.

Does it report Core Web Vitals or Google rankings?

No. Core Web Vitals require field or browser measurements, and rankings require a separate search data source. Empty fields never imply that either surface was checked.

What happens when a site blocks the crawler or a crawl is incomplete?

The site or page is BLOCKED, FAILED, or PARTIAL, its error remains visible in the Dataset, and monitor state is retained. An unverifiable response cannot become an SEO regression.

Can I use it as an SEO monitoring API?

Yes. Save monitor input as an Apify Task, schedule it, and consume record_type=event rows through the Dataset API or a successful-run webhook. The first complete run creates a quiet baseline.

Known limits

• Maximum 50 sites and 10,000 pages per site.
• HTTP(S) ports 80/443 only; credentials, localhost, private, reserved, link-local and metadata addresses are blocked. Direct requests connect to the exact validated public IP, closing DNS rebinding between validation and TCP connect. Every redirect is revalidated and must stay on the input site (apex and www are treated as the same site).
• Response, redirect, retry, sitemap file/depth, gzip decompression, page and unique-link limits are strict. HTML link, heading, hreflang, structured-data and captured-text collections are bounded; exceeding a bound is explicit PARTIAL/FAILED, never silent truncation.
• No JS rendering, Core Web Vitals, ranking claims, LLM recommendations, arbitrary SEO folklore, external database, universal anti-bot bypass or historical raw-page store.