# SEO Site Audit Crawler: On-Page SEO, Broken Links & Redirects (`subtleasianmerch/seo-site-audit`) Actor

Crawl a website with plain HTTP, respecting robots.txt, and audit every page: titles, meta descriptions, headings, canonicals, noindex, hreflang, structured data, alt text, broken links, redirect chains and duplicates. One row per page, a site summary and score, and alerts for new issues.

- **URL**: https://apify.com/subtleasianmerch/seo-site-audit.md
- **Developed by:** [Public Data Tools](https://apify.com/subtleasianmerch) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.40 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SEO Site Audit Crawler: On-Page SEO, Broken Links & Redirects

Crawl a whole website the way a search engine does, starting from its home page or sitemap, and get an on-page SEO audit of every page: title, meta description, headings, canonical, noindex, hreflang, structured data, Open Graph tags, image alt text and word count, plus broken links, redirect chains and duplicate titles and descriptions across the site. Each address is one row with its status code and a plain-English list of issues (error, warning or notice); each site gets a summary with a site score. The crawler uses plain HTTP, no browser, respects robots.txt and keeps a gentle pace, so runs are fast and cheap.

- 🕷️ **Crawls the whole site** from a start page and its sitemap: same site only (subdomains optional), with page, depth and pattern limits.
- 🏷️ **On-page SEO per page:** title and meta description (missing, too long or short, duplicate), H1 and H2, canonical, meta robots and X-Robots-Tag, hreflang, `<html lang>`, viewport, Open Graph and Twitter cards, JSON-LD types and errors, images without alt text, word count.
- 🔗 **Broken links and redirects:** every 404 and 500 with the pages that link to it, redirect chains and loops hop by hop, links that go through redirects, and optionally links to other sites.
- 🗺️ **Sitemap checks:** orphan pages, sitemap entries that redirect, break or are noindex, and indexable pages missing from the sitemap.
- 🤖 **Polite by design:** robots.txt and its Crawl-delay are always respected, one request at a time per site, an honest user agent, and a site that blocks crawlers is left alone.
- 🔔 **Monitoring:** schedule it with a monitor ID and every row shows the issues that are new or fixed since the last crawl.
- 💸 **$0.003 per audited page** ($3 per 1,000). Redirects, broken addresses, files, addresses robots.txt blocks and link checks are listed free.

**Try it:** the input comes pre-filled with crawlee.dev and a limit of 20 pages. Click **Start**; the audit takes under a minute. Then put in your own site.

### Who it's for

- **SEO consultants and agencies:** a technical audit of a client's site in minutes, as a spreadsheet you can sort and filter, with the summary for the report.
- **Site owners and marketers:** find the broken links, missing titles and duplicate descriptions before Google does, and watch for new ones every week.
- **Developers:** check a release or a migration didn't break links, redirects, canonicals or hreflang.
- **Content teams:** list thin pages, missing meta descriptions and images without alt text, page by page.
- **Data teams:** feed site health into dashboards through the Apify API.

### What you get

**One row per address the crawl found.** The `type` says what it is:

| Type | What it is | Charged |
|---|---|---|
| `page` | An HTML page that answered 200, fully audited | Yes |
| `redirect` | An address that redirects, with the whole chain | Free |
| `error` | An address that answered 4xx or 5xx, with the pages that link to it | Free |
| `failed` | An address that didn't answer (time-out, DNS, refused connection) | Free |
| `file` | A PDF, image or other file that isn't HTML | Free |
| `disallowed` | An address robots.txt blocks; not fetched | Free |
| `blocked` | Where a site refused the crawler (bot check, HTTP 403 or 429) | Free |

**For every page:** status code, response time, size, depth (clicks from the start page), the pages that link to it, whether it's in the sitemap and whether it's indexable; title and meta description with their lengths; the first H1, the number of H1s and H2s; canonical (self, other, missing or conflicting) and the canonical target's status; meta robots, X-Robots-Tag, noindex and nofollow; hreflang links with their status and whether they link back; `<html lang>`, viewport, Open Graph tags, Twitter card; structured data types (JSON-LD and microdata) and JSON-LD errors; images and images without alt text (with examples); word count; internal, external and nofollow link counts; broken links on the page; links to redirects; mixed content (http resources on https pages); and the `issues` list with counts by severity.

**Issues** come with a code, a severity and a message, for example `title-too-long` (warning): "The title is 72 characters; search results usually show about 60." There are 68 kinds (16 errors, 31 warnings, 21 notices); the most common are missing or duplicate titles and descriptions, missing H1, broken internal links, redirect chains, canonical and hreflang problems, noindex pages in the sitemap, orphan pages, images without alt text, thin pages and mixed content.

**The run summary** (`RUN_SUMMARY` in the run's key-value store), per site: pages audited, status code counts, issue counts by kind, groups of duplicate titles, descriptions and H1s, broken addresses with the pages that link to them, redirect chains, addresses robots.txt blocks, sitemap addresses the crawl didn't reach (and why), external link results, and a **site score** out of 100. Each audited page starts at 100 and loses 15 points for each kind of error and 5 for each kind of warning it has (notices don't count), down to 0; the site score is the average over the audited pages.

### How to use it

#### In Apify Console

1. Enter a **start page**, usually the site's home page. You can add more sites; they're crawled side by side.
2. Set **Pages per site** (100 by default). Pages nearest the start page are audited first.
3. Optionally narrow the crawl with **Link depth**, **Only addresses matching** or **Skip addresses matching** (for example `*/tag/*`), or include subdomains.
4. Click **Start**. At the default pace a site is crawled at about one page a second.
5. Open the **Pages**, **Issues**, **Links and redirects** or **Tags** view, or download the results as Excel, CSV or JSON. The **Storage → Key-value store → RUN_SUMMARY** record has the site summary.

#### Check links to other sites

Turn on **Check links to other sites** to check, by status only, the links that lead off the site (up to 300 by default). Each of those sites' robots.txt and pace are respected. A site that turns the check away (HTTP 401, 403, 429) is listed as "refused" in the summary, not as broken, because many sites answer bots that way.

#### Get alerts for new issues (monitoring)

1. Enter a **Monitor ID**, for example `my-site-weekly`.
2. Schedule the Actor (Schedules → create, e.g. weekly) with the same input.

Each run then compares every address with the last crawl of the same site: rows get `isNewPage`, `newIssues` and `fixedIssues`, and the summary counts them. Use the **Changes since last crawl** view, or connect the run to Slack or email with Apify's integrations.

#### With the Apify API

```bash
curl -X POST "https://api.apify.com/v2/acts/subtleasianmerch~seo-site-audit/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://www.example.com"], "maxPagesPerSite": 200}'
```

### Example input

```json
{
  "startUrls": ["https://www.example.com"],
  "maxPagesPerSite": 500,
  "maxDepth": 5,
  "sitemapPages": "crawl",
  "excludePatterns": ["*/tag/*", "?sort="],
  "checkExternalLinks": true,
  "maxExternalLinks": 300,
  "delayMs": 500,
  "monitorId": "example-weekly"
}
```

### Output

A real row from a test run on Apify (2 November 2026, docs.apify.com), shortened:

```json
{
  "url": "https://docs.apify.com/proxy",
  "type": "page",
  "statusCode": 200,
  "finalUrl": "https://docs.apify.com/proxy",
  "redirectHops": 0,
  "responseTimeMs": 4,
  "sizeBytes": 76878,
  "depth": 1,
  "inlinks": 99,
  "inSitemap": true,
  "indexable": true,
  "indexability": "indexable",
  "title": "Proxy | Platform | Apify Documentation",
  "titleLength": 38,
  "metaDescription": "Learn to anonymously access websites in scraping/automation jobs. Improve data outputs and efficiency of bots, and access websites from various geographies.",
  "metaDescriptionLength": 156,
  "h1": "Proxy",
  "h1Count": 1,
  "h2Count": 9,
  "canonicalUrl": "https://docs.apify.com/proxy",
  "canonicalStatus": "self",
  "noindex": false,
  "hreflang": [
    { "lang": "en", "url": "https://docs.apify.com/proxy", "statusCode": 200, "returnLink": true },
    { "lang": "x-default", "url": "https://docs.apify.com/proxy", "statusCode": 200, "returnLink": true }
  ],
  "htmlLang": "en",
  "viewport": true,
  "twitterCard": "summary_large_image",
  "structuredDataTypes": [],
  "images": 3,
  "imagesWithoutAlt": 0,
  "wordCount": 1747,
  "internalLinks": 38,
  "externalLinks": 34,
  "brokenLinks": [
    { "url": "https://docs.apify.com/residential_proxy", "statusCode": 404, "error": null, "internal": true }
  ],
  "brokenLinkCount": 1,
  "linksToRedirects": 2,
  "issues": [
    { "code": "broken-internal-links", "severity": "error", "message": "1 link to broken pages on the site: /residential_proxy (HTTP 404)." },
    { "code": "structured-data-missing", "severity": "notice", "message": "No structured data (JSON-LD or microdata)." },
    { "code": "links-to-redirects", "severity": "notice", "message": "2 links to addresses that redirect (e.g. /cli, /sdk/js/docs/guides/proxy-management); linking to the final address saves a hop." }
  ],
  "issueCount": 3,
  "errorCount": 1,
  "warningCount": 0,
  "noticeCount": 2,
  "site": "docs.apify.com",
  "crawledAt": "2026-11-02T18:37:37.748Z"
}
```

The broken address has a row of its own (free): `{ "url": "https://docs.apify.com/residential_proxy", "type": "error", "statusCode": 404, "linkedFrom": ["https://docs.apify.com/proxy"], "issues": [{ "code": "http-4xx", "severity": "error", "message": "HTTP 404. 1 crawled page links to it (e.g. /proxy)." }] }`. A redirect row lists its chain, for example `"redirectChain": [{ "url": "https://crawler-test.com/redirects/redirect_2", "statusCode": 301, "redirectsTo": "…" }, …]`, with `finalUrl` and `finalStatusCode`.

### Pricing

- **$0.003 per audited page** ($3 per 1,000 pages). 500 pages cost $1.50.
- **Free:** redirects, error pages, addresses that don't answer, files, addresses robots.txt blocks, sites that refuse the crawler, status checks of links beyond your page limit, and external link checks.
- A run stops cleanly at your maximum cost per run. Apify's platform usage is included in the price.

### Good to know

- **No browser.** The crawler reads the HTML the server sends, as search engines do on their first pass. Pages built only with JavaScript (some single-page apps) show little text, few links and missing tags here, even if they look fine in a browser. For speed and Core Web Vitals, use the Lighthouse Audit Actor (below).
- **robots.txt always applies.** Addresses it blocks are listed but never fetched. If robots.txt can't be read (the server answers 5xx), the site is treated as closed to crawlers, as the robots.txt standard says. A Crawl-delay is honoured up to 60 seconds; a site asking for more isn't crawled.
- **Gentle pace.** One request at a time per site, at least 500 ms apart by default (you can choose 250 ms to 60 s), and longer if robots.txt asks. At the default pace a site goes at roughly 1,000 to 4,000 pages an hour, depending on how fast it answers (in tests: 100 pages of docs.apify.com in 87 seconds). The run's time limit is 1 hour by default; for big sites, raise it under **Run options → Timeout**. The crawl stops about 2–3 minutes before the limit and delivers what it found.
- **Sites that refuse crawlers are left alone.** A bot check (Cloudflare, DataDome and similar), a 403 on the first page, five 401/403 answers in a row, or HTTP 429 that continues after waiting stop that site's crawl at once; it's listed as blocked, free. The Actor doesn't use proxies or disguises to get past blocks.
- **Limits:** up to 5,000 audited pages per run in all, and up to twice your page limit in requests per site. Once the page limit is reached, the links that the audited pages point to are still checked for status (HEAD requests, free), so their broken links are found; pages beyond the limit aren't audited. Sitemaps: up to 20 files, 100,000 addresses, 30 MB or 60 seconds of reading. HTML pages larger than 5 MB are read up to 5 MB.
- **Addresses are normalised:** the `#fragment` is dropped, and tracking parameters (utm\_\*, gclid, fbclid and similar) are removed unless you turn that off. `/page` and `/page/` stay different addresses, because servers treat them differently; when both answer, the duplicate title check shows it.
- **Duplicates** are counted among indexable pages only: a page that is noindex or whose canonical points elsewhere is meant to be a copy.
- **Length limits** for titles (30–60 characters) and meta descriptions (70–160) are the usual guides; search engines actually cut by pixel width.
- **Nofollow links are followed** within the site, so broken links behind them are found too.

### FAQ

**Does it work on any website?** On public sites that allow crawling. Pages behind a login, and sites that block crawlers, can't be audited.

**Will it slow my site down?** No more than a single visitor: one request at a time, with a pause after each.

**Can I audit just one section?** Yes: start at that section and use **Only addresses matching**, for example `https://www.example.com/blog/*`.

**What about orphan pages?** With **Sitemap pages** set to "crawl" (the default), pages only the sitemap lists are audited after the linked pages, and flagged as orphans when the crawl finished without reaching the page limit.

**Why is a page I can see in my browser almost empty here?** It's probably built with JavaScript in the browser. The crawler sees the HTML the server sends.

**What does a broken link cost?** Nothing: only audited HTML pages are charged.

**Can I get the results in Google Sheets?** Yes: export the dataset as CSV or Excel and open it in Google Sheets, or connect it with Make, Zapier or n8n.

### Related Actors

More Actors from the same developer that work well with this one:

| Actor | Use it to |
|---|---|
| [Lighthouse Audit](https://apify.com/subtleasianmerch/lighthouse-audit) | score the same pages for speed, Core Web Vitals and accessibility |

Questions, bugs or ideas? Open an issue on this Actor's **Issues** tab.

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts, usually a site's home page, e.g. https://www.example.com. "example.com" works too (https is added). Links are followed within the same site (www and non-www count as one). Several sites can be crawled in one run.

## `maxPagesPerSite` (type: `integer`):

At most this many HTML pages are audited (and charged) per site, nearest the start page first. Redirects, error pages and files are listed too, free. A run audits at most 5,000 pages in all.

## `sitemapUrls` (type: `array`):

Optional. Without any, the sitemaps listed in each site's robots.txt are read (or /sitemap.xml). XML, .xml.gz, text sitemaps and sitemap indexes work. A sitemap on its own (no start page) is fine too: its pages become the start pages.

## `sitemapPages` (type: `string`):

What to do with the pages the sitemaps list. Crawl: after the linked pages, also audit sitemap pages no crawled page links to (orphan pages). Compare: only mark which pages are in the sitemap and list sitemap pages the crawl didn't reach. Ignore: don't read sitemaps.

## `maxDepth` (type: `integer`):

Optional. How many clicks from the start page to follow: 0 audits only the start pages, 1 adds the pages they link to, and so on. Empty: no limit (the page limit still applies).

## `includeSubdomains` (type: `boolean`):

Also crawl subdomains of the start page's site, e.g. blog.example.com and shop.example.com for example.com. Each subdomain's own robots.txt is respected.

## `includePatterns` (type: `array`):

Optional. Only crawl addresses that match one of these patterns. \* stands for any characters, e.g. https://www.example.com/blog/\*. A pattern without \* matches addresses that contain it, e.g. /blog/. Start pages are always crawled.

## `excludePatterns` (type: `array`):

Optional. Don't crawl addresses that match one of these patterns (same rules as above), e.g. */tag/* or ?sort=.

## `removeTrackingParams` (type: `boolean`):

Treat /page?utm_source=x (and gclid, fbclid and similar) as /page, so one page isn't crawled many times. Other parameters are kept as they are.

## `checkExternalLinks` (type: `boolean`):

Also check whether links to other sites still work (status only, nothing is audited or charged there). Each site's robots.txt and pace are respected; a site that turns the check away (HTTP 401, 403, 429) is listed as "refused", not as broken.

## `maxExternalLinks` (type: `integer`):

With the option above: at most this many different external links are checked per run, in the order they were found.

## `delayMs` (type: `integer`):

Requests to a site go one at a time, with at least this pause after each answer (500 ms means at most 2 requests a second). If the site's robots.txt asks for a longer Crawl-delay, that is used instead. Sites in one run are crawled side by side.

## `monitorId` (type: `string`):

Give scheduled runs the same ID (letters, digits and dashes) to compare each crawl with the last one: every row then says whether the page is new and which issues are new or fixed since the last run. Leave empty for one-off audits.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxPagesPerSite": 20,
  "sitemapPages": "crawl",
  "includeSubdomains": false,
  "removeTrackingParams": true,
  "checkExternalLinks": false,
  "maxExternalLinks": 300,
  "delayMs": 500
}
```

# Actor output Schema

## `results` (type: `string`):

The dataset rows this run produced.

## `runSummary` (type: `string`):

Counts, costs and anything skipped, saved as RUN_SUMMARY.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev/"
    ],
    "maxPagesPerSite": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("subtleasianmerch/seo-site-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev/"],
    "maxPagesPerSite": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("subtleasianmerch/seo-site-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxPagesPerSite": 20
}' |
apify call subtleasianmerch/seo-site-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,subtleasianmerch/seo-site-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TKbv3uUyYW21UwIsE/builds/lT8zIDD6iadW9bhAh/openapi.json