# SEO Page Audit — Sitemap Crawl & On-Page Checker (`insight.solutions/seo-page-audit`) Actor

Point it at a domain. It reads robots.txt, walks the sitemap or crawls the site, then audits every page: title, description, H1s, canonical, hreflang, schema, images, links and speed. 23 issue codes, a transparent 0-100 score, and an optional broken-link check.

- **URL**: https://apify.com/insight.solutions/seo-page-audit.md
- **Developed by:** [Insight Solutions](https://apify.com/insight.solutions) (community)
- **Categories:** SEO tools, Marketing, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.60 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## SEO Page Audit — Sitemap Crawl & On-Page Checker

**Point it at a domain and get every page audited.** This Actor reads a site's `robots.txt`, walks its sitemap — nested indexes, gzip children and relative URLs included — or crawls the site when there is no sitemap, and then audits each page: title, meta description, headings, canonical, `hreflang`, structured data, images, links, response time, page weight. Every page comes back with a list of **issue codes** and a **0–100 score** built from published weights you can check against the table below.

No API key. No browser. No JavaScript execution. **$1.00 per 1,000 pages**, proxy included. `robots.txt`, sitemaps and link checks are free, pages that could not be read are free, and a run that reads nothing costs nothing at all.

### Try it in 30 seconds

```json
{
  "websites": ["basecamp.com"],
  "maxPages": 50
}
```

A bare domain is enough. The run reads `https://basecamp.com/robots.txt`, follows the sitemap it finds, and audits the first 50 URLs listed there. Want specific pages instead? Put them in `pageUrls` and nothing is discovered or crawled.

### What comes back

One `page` row per audited URL. This one is real — read from basecamp.com/pricing on 2026-09-11:

```jsonc
{
  "ok": true,
  "rowType": "page",                       // "page" | "site" | "broken-link" | "diagnostic"

  "input": "basecamp.com",
  "website": "basecamp.com",
  "url": "https://basecamp.com/pricing",
  "source": "sitemap",                     // "sitemap" | "crawl" | "input"
  "sitemapLastmod": "2026-05-26T12:00:00-07:00",

  // the response
  "statusCode": 200,
  "finalUrl": "https://basecamp.com/pricing",
  "redirectChain": [],
  "responseTimeMs": 243,
  "contentType": "text/html",
  "contentLength": 65209,                  // bytes of HTML
  "compressed": true,
  "lastModified": "Wed, 02 Sep 2026 23:45:00 GMT",
  "hsts": false,
  "xRobotsTag": null,

  // the markup
  "title": "Basecamp — Pricing",           // — decoded to what a reader sees
  "titleLength": 18,
  "metaDescription": "Start free with Basecamp — one project, five users, forever free…",
  "metaDescriptionLength": 128,
  "metaRobots": "index, follow",
  "canonical": "https://basecamp.com/pricing",
  "canonicalMatchesUrl": true,
  "lang": "en",
  "h1": ["No per-user fees, everyone's included. Simple fixed prices — another Basecamp exclusive."],
  "h1Count": 1, "h2Count": 9, "h3Count": 3,
  "wordCount": 2060,
  "textToHtmlRatio": 18.88,                // visible chars per 100 chars of markup
  "ogTitle": "Pricing",
  "ogDescription": "Start free with Basecamp — one project, five users, forever free…",
  "ogImage": "https://basecamp.com/assets/images/opengraph/basecamp.png?v=1.5.1",
  "twitterCard": "summary_large_image",
  "hreflang": [],
  "structuredDataTypes": ["WebPage", "WebSite", "ItemList", "SiteNavigationElement"],
  "hasViewport": true,
  "charset": "utf-8",
  "favicon": "https://basecamp.com/assets/images/general/favicon.svg?v=1.5.1",
  "internalLinkCount": 35,
  "externalLinkCount": 12,
  "nofollowCount": 0,
  "imageCount": 1,
  "imagesMissingAlt": 0,
  "imagesMissingAltExamples": [],
  "scriptCount": 10, "stylesheetCount": 48, "inlineStyleCount": 0,
  "noindex": false, "nofollowPage": false,
  "mixedContentCount": 0,
  "mixedContentExamples": [],

  // the verdict
  "issues": ["title-too-short"],
  "issueCount": 1,
  "score": 95,                             // 100 − 5 for title-too-short

  "error": null,
  "errorType": null,
  "scrapedAt": "2026-09-11T15:44:37.120Z",
  "sourceUrl": "https://basecamp.com/pricing"
}
```

Then one `site` row per website, written after the pages it summarises:

```jsonc
{
  "ok": true,
  "rowType": "site",
  "website": "basecamp.com",
  "url": "https://basecamp.com",
  "pagesAudited": 50,
  "pagesFailed": 0,
  "discovery": "sitemap",
  "robotsFound": true,
  "robotsBlocked": 0,
  "sitemapUrls": ["https://basecamp.com/sitemap.xml"],
  "sitemapUrlCount": 174,
  "statusCounts": { "200": 49, "404": 1 },
  "avgResponseMs": 268,
  "issueCounts": { "missing-description": 12, "thin-content": 4, "title-too-short": 1 },
  "score": 91,                             // mean of the 50 page scores
  "worstPages": [
    { "url": "https://basecamp.com/…", "score": 42, "issues": ["http-error", "missing-h1", "…"] }
  ],
  "linksChecked": 0,
  "linksUnchecked": 0,                     // links left in the queue when the time budget ran out
  "brokenLinkCount": 0
}
```

And, with `checkLinks` on, one `broken-link` row per dead internal link, naming the pages that point at it.

### The issue codes, and what each costs

Every page starts at **100**. Each issue found subtracts its weight. The total is floored at 0.

```
score = max(0, 100 − Σ weight(issue))
```

That is the whole formula. No curve, no weighting by page importance, nothing calibrated against anything. It is a checklist with a running total, published here so that fixing a missing `<h1>` and predicting exactly ten points is possible — and so that if you disagree with a weight you can recompute the column yourself from the `issues` array, which is the part actually worth acting on.

| Code | −pts | Fires when |
|---|---:|---|
| `http-error` | 40 | The page answered 4xx or 5xx — a 404 in the sitemap, a 500 on a live URL. It still gets a full row: a broken page you can see is the point. (A 401, 403 or 429 is a *refusal*, not a page; those are free diagnostic rows instead) |
| `noindex` | 25 | The robots meta tag or the `X-Robots-Tag` header says `noindex` or `none` |
| `missing-title` | 20 | No `<title>`, or an empty one |
| `missing-description` | 10 | No `<meta name="description">` |
| `missing-h1` | 10 | No `<h1>` on the page |
| `duplicate-title` | 10 | Another audited page on the same site publishes this exact title |
| `thin-content` | 10 | Fewer than **200** words of visible body text |
| `mixed-content` | 8 | An https page loads at least one asset over plain http |
| `missing-viewport` | 5 | No viewport meta tag, so the page will not adapt to a phone |
| `multiple-h1` | 5 | More than one `<h1>` |
| `missing-canonical` | 5 | No `rel="canonical"` link |
| `canonical-mismatch` | 5 | The canonical points somewhere other than the URL that served the page |
| `images-missing-alt` | 5 | At least one `<img>` has no `alt` attribute at all |
| `slow-response` | 5 | The page took longer than **2,000 ms** to answer |
| `duplicate-description` | 5 | Another audited page on the same site publishes this exact description |
| `title-too-long` | 5 | Title longer than **60** characters |
| `title-too-short` | 5 | Title shorter than **20** characters |
| `missing-lang` | 3 | No `lang` attribute on `<html>` |
| `description-too-long` | 3 | Description longer than **160** characters |
| `no-structured-data` | 3 | No JSON-LD on the page |
| `not-compressed` | 3 | The response was sent uncompressed — and only when the headers prove it |
| `redirected` | 3 | The URL in the sitemap (or the one you sent) redirected somewhere else |
| `large-html` | 3 | The HTML document alone is over **1 MB**, before any assets load |

Four of those are more careful than they look, and it matters:

- **`alt=""` is not a missing alt.** It is the correct, deliberate markup for a decorative image. Counting it would file a finding against every site that gets accessibility right. Only a missing attribute counts.
- **`mixed-content` counts assets, not links.** An `<a href="http://…">` is the reader's choice and no browser warns about it. The captured apify.com pricing page contains exactly one `http://` URL — the LinkedIn link in the footer — and correctly reports zero mixed content.
- **`canonical-mismatch` normalises before comparing.** A canonical differing only by a trailing slash or a fragment is not a mismatch.
- **`not-compressed` fires only on proof.** The HTTP client inflates compressed responses transparently and deletes `content-encoding` and `content-length` when it does, so "no encoding header" is not evidence of anything. This fires only when a declared `content-length` matches the body that arrived — meaning nothing was inflated. When the headers do not settle it, `compressed` is `null` and no issue is raised. A false "you are not compressing" on every page of every site would be a worse bug than missing the occasional real one.

### Use cases

- **Pre-launch and post-migration QA** — run the sitemap after a deploy and diff the `issueCounts` histogram against last week's. A migration that quietly shipped `noindex` on 400 pages shows up as a single number.
- **Agency audits and pitches** — walk in with a scored page-by-page table of a prospect's site, sorted by `score`, with the ten worst named on the summary row.
- **Content-team backlog** — export the `issues` view, filter to `missing-description` and `thin-content`, and you have a prioritised list of pages to write.
- **Broken-link sweeps** — `checkLinks: true` on a site you are responsible for, then read the `links` view. Each row names the broken URL and up to ten pages that link to it, which is normally enough to identify the template at fault.
- **Duplicate-content detection** — `duplicate-title` and `duplicate-description` are computed across the whole site, so faceted listings and paginated archives that all share one title are found in one run.
- **Competitive teardown** — audit a competitor's sitemap and compare `wordCount`, `structuredDataTypes` and `textToHtmlRatio` distributions against your own.
- **Monitoring** — schedule it weekly against `maxPages: 100` and watch the site `score` as a single trend line.

### How it works, and why it keeps working

A website tells you which pages it has in two places, and this Actor reads both before it audits anything.

**1. `robots.txt`.** The first request of every website is `GET /robots.txt`. It does three jobs: it proves which of the four candidate addresses (`https://`, `https://www.`, `http://`, `http://www.`) the host actually answers on; it supplies the `Sitemap:` lines; and it supplies the `Disallow` rules, which are obeyed. Rule matching is the same longest-match-wins algorithm search engines use, with `*` and `$` patterns, matched against path *and* query — so `Disallow: /search?` blocks `/search?q=shoes` and leaves `/searching` alone, and `Allow: /private/public-notes` correctly overrides `Disallow: /private/`. Only the `User-agent: *` group is read; obeying Googlebot's rules would be pretending to be Googlebot.

**2. The sitemap.** Whatever `robots.txt` named, or failing that `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml` and `/sitemap/sitemap.xml` in that order. A `<sitemapindex>` is followed to its children, three levels deep, up to 50 sitemap files per site. Children ending `.xml.gz` are inflated. `<loc>` values are resolved against the sitemap's own URL, because the spec says they are absolute and real sitemaps disagree — basecamp.com's, captured live, lists `/`, `/2` and `/2/integrations`. `<lastmod>`, `<changefreq>` and `<priority>` ride along onto the page row. A guessed path that is not there is not a fault — a site with no sitemap is the case the crawl fallback exists for, and it produces no diagnostic row; a sitemap that `robots.txt` or a sitemap index *declared* and that then fails does.

**3. No sitemap? Crawl.** A site that answers nothing useful at any of those locations is crawled breadth-first from its home page: same host with `www.` folded away, HTML pages only, shallowest links first, fragments and `mailto:` and asset paths skipped, bounded by `maxPages` and by `robots.txt`.

Then each page is fetched with a browser `accept` header, redirects followed, a 20-second per-request timeout, up to `maxConcurrency` at once — and **150–400 ms apart per host, always**. A 500-page audit arriving as 500 simultaneous requests is a denial-of-service attempt with a friendly user agent; raising `maxConcurrency` overlaps the waiting, not the hammering.

Three decisions are worth spelling out, because they are what separates a useful audit from a noisy one:

**A 404 is a row, not an error.** A URL in a sitemap that answers 404 is one of the most useful things an audit can find. It comes back as a full `page` row — status, title, response time, the `http-error` issue, a score — and it is charged, because it is a finding. Two cases are deliberately *not* rows: a URL that answered nothing at all, and a URL that was **refused** (401, 403, 429, or a Cloudflare origin error). A refusal is very often about this exit IP rather than about the URL, and billing a row that scores 0 because a bot wall answered would be charging for the wall. Both are free diagnostic rows.

**Duplicate checks run at the end of a site.** `duplicate-title` and `duplicate-description` cannot be decided while pages are still arriving, so every page of a website is buffered until the site is finished, the title and description index is built, and then all the page rows are written at once followed by the site summary. The buffer holds extracted fields, never HTML.

**The word count is a word count.** Scripts, styles, SVG, `<template>` and the whole `<head>` are removed before counting, so a client-rendered page shipping 400 KB of JSON state in a script tag does not read as 60,000 words of content. The captured apify.com pricing page is 437 KB of markup carrying 2,109 real words, and `textToHtmlRatio: 2.81` says so.

### How it compares

- **The sitemap is the input, not a crawl budget.** Most page-level auditors make you either paste URLs one at a time or buy a crawler subscription. This takes a domain and does the discovery — including the gzip and nested-index cases that break naive readers.
- **The score is arithmetic you can audit.** Every weight is in the table above and in `src/audit.js` as one object. Nothing is hidden behind a model, and the `issues` array is always there to recompute from.
- **Priced by the page, with the plumbing free.** `robots.txt`, every sitemap file, every gzip child and every link check cost nothing. You pay for audited pages and one summary row.
- **Failures are free and legible.** A dead host, a URL that answers with a PDF, a bot wall, or a site that ran out the clock produces a diagnostic row with an `errorType` you can branch on — and no charge. A run that audits nothing at all finishes **FAILED** with the reason in its status message, never a green run containing an apology.
- **No browser.** Auditing is pure HTTP and targeted regular expressions, which is why a 5,000-page site costs single-digit dollars rather than hundreds — and why a page whose content only exists after client-side rendering is a documented limitation rather than a hidden one.

### Input reference

| Field | Type | Default | What it does |
|---|---|---|---|
| `websites` | array of strings | `["basecamp.com"]` | Bare domains or full URLs. Each is tried `https://` → `https://www.` → `http://`. Per site: robots.txt, then sitemap discovery, then up to `maxPages` page audits. Duplicates are read, and billed, once |
| `pageUrls` | array of strings | `[]` | Audit exactly these URLs — no sitemap, no crawl. URLs on the same host as a `websites` entry are folded into that site's audit rather than becoming a second one |
| `maxPages` | integer 1–5000 | `100` | Pages audited **per website**. This is the number that sets the price of a run |
| `checkLinks` | boolean | `false` | Also fetch every unique internal link found (a plain `GET`), and write a `broken-link` row for each one that answers 4xx, 5xx or nothing. $0.0002 per broken link; checking is free. Links the time budget cuts off are counted in `linksUnchecked`, never as broken |
| `maxLinksToCheck` | integer 1–5000 | `500` | Ceiling on the link check, per website |
| `respectRobots` | boolean | `true` | Obey the site's own `robots.txt`. Skipped URLs are counted as `robotsBlocked` and never fetched or billed. Turn off only for a site you own |
| `maxSitemapUrls` | integer 1–5000 | `5000` | Ceiling on URLs collected from sitemaps. Collection also stops as soon as there are enough to fill `maxPages` |
| `maxConcurrency` | integer 1–16 | `8` | Pages of one website in flight at once. Each worker keeps its own proxy session. Per-host spacing of 150–400 ms applies regardless |
| `maxRunSecs` | integer 30–3600 | `240` | Whole-run wall-clock budget. When it runs out the Actor keeps and bills what it has, and files a free diagnostic row saying how many URLs it never reached |
| `proxyConfiguration` | object | Apify datacenter | Switch the group to `RESIDENTIAL` for sites behind an aggressive bot wall. Datacenter is the default because it is cheaper and ordinary websites accept it |

At least one of `websites` and `pageUrls` must be non-empty. A run with neither finishes FAILED, having charged nothing.

### Output reference

Every row carries the same keys, whatever its type. `ok: true` is a `page`, `site` or `broken-link` row; `ok: false` is a free diagnostic row.

**Common to every row**

| Field | What it is |
|---|---|
| `ok`, `rowType` | Whether this row is a result, and which of the four kinds it is |
| `input`, `website` | The entry you supplied, and the registrable domain it resolved to |
| `url`, `sourceUrl` | The URL this row is about |
| `source` | Where the URL came from: `sitemap`, `crawl` or `input` on a page row; `site` on a summary; `link-check` on a broken link |
| `error`, `errorType`, `scrapedAt` | Why a row failed, if it did, and when the row was written |

**`page` rows**

| Field | What it is |
|---|---|
| `statusCode`, `finalUrl`, `redirectChain`, `responseTimeMs` | What the server answered, where it ended up, and how long it took |
| `contentType`, `contentLength`, `compressed`, `lastModified`, `hsts`, `xRobotsTag` | Response headers worth an audit column. `contentLength` is bytes of HTML received |
| `title`, `titleLength`, `metaDescription`, `metaDescriptionLength`, `metaRobots` | The tags search results are built from, entity-decoded |
| `canonical`, `canonicalMatchesUrl`, `lang`, `charset`, `hasViewport`, `favicon` | Indexing and rendering signals |
| `h1`, `h1Count`, `h2Count`, `h3Count` | Heading text and structure |
| `wordCount`, `textToHtmlRatio` | Visible body text, and how much of the document it is |
| `ogTitle`, `ogDescription`, `ogImage`, `twitterCard` | Social cards |
| `hreflang`, `structuredDataTypes` | Alternates as `{lang, href}`, and every JSON-LD `@type` including nested ones |
| `internalLinkCount`, `externalLinkCount`, `nofollowCount` | Unique links, split by host with `www.` folded away |
| `imageCount`, `imagesMissingAlt`, `imagesMissingAltExamples` | Images, and up to twenty `src` values with no `alt` attribute |
| `scriptCount`, `stylesheetCount`, `inlineStyleCount` | Front-end weight |
| `noindex`, `nofollowPage`, `mixedContentCount`, `mixedContentExamples` | Indexing directives and http assets on an https page |
| `issues`, `issueCount`, `score` | The verdict |
| `sitemapLastmod`, `sitemapChangefreq`, `sitemapPriority` | What the sitemap said about this URL, when it came from one |

**`site` rows**

| Field | What it is |
|---|---|
| `pagesAudited`, `pagesFailed` | How many URLs produced a row, and how many could not be read |
| `discovery` | `sitemap`, `crawl` or `input` |
| `robotsFound`, `robotsBlocked` | Whether robots.txt was readable, and how many URLs it kept out |
| `sitemapUrls`, `sitemapUrlCount` | Which sitemap files were read, and how many URLs came out of them |
| `statusCounts`, `avgResponseMs` | Status histogram and mean response time |
| `issueCounts` | How many pages triggered each issue code, heaviest first |
| `score` | The rounded mean of the page scores |
| `worstPages` | The ten lowest-scoring pages, each with its score and issues |
| `linksChecked`, `linksUnchecked`, `brokenLinkCount` | Link-check totals, zero unless `checkLinks` is on. `linksUnchecked` is what the time budget cut off |

**`broken-link` rows** carry `url`, `statusCode`, `error` and `foundOn` — up to ten audited pages that link to the dead URL.

`errorType` on a diagnostic row is one of:

| Value | Meaning | Charged? |
|---|---|---|
| `unreachable` | Nothing answered — a dead host, a connection that failed, a domain that does not resolve | No |
| `blocked` | The request was refused — 401, 403, 429, a Cloudflare origin error, or an anti-bot interstitial served in place of the page | No |
| `not-html` | The URL answered with a file rather than a page — a PDF, an image, a JSON feed | No |
| `timeout` | The run's `maxRunSecs` budget ran out before this URL was read | No |
| `invalid-input` | The entry was not a fetchable website or URL — an email address, an IP, a bare word | No |

### Pricing

**$1.00 per 1,000 pages.** Pay-per-event, with the proxy already inside that number — there is no separate proxy line on your bill for this Actor.

| Event | What triggers it | FREE | Starter | Scale | Business |
|---|---|---|---|---|---|
| **Page audited** *(primary)* | One audited page written to your dataset | $0.001 | $0.001 | $0.0008 | $0.0006 |
| Site summary | One website that produced at least one page | $0.002 | $0.002 | $0.002 | $0.002 |
| Broken link found | One internal link that failed, only when `checkLinks` is on | $0.0002 | $0.0002 | $0.0002 | $0.0002 |
| Run started | Once per run, after the first page is audited | $0.001 | $0.001 | $0.001 | $0.001 |

**Worked example.** A weekly audit of a 500-page site with link checking on, finding 30 broken links:

- 500 pages × $0.001 = **$0.500**
- 1 site summary × $0.002 = **$0.002**
- 30 broken links × $0.0002 = **$0.006**
- 1 run start = **$0.001**
- **Total: $0.509**

What you are never charged for: `robots.txt`, any sitemap file however many there are, any link check (only a *broken* link produces a billed row), a URL that answered nothing, a URL that answered with a file, a URL that was refused with 401/403/429 or behind a bot wall, a URL `robots.txt` disallowed, an entry that was not a website, or anything the run never reached before `maxRunSecs`. If a whole run comes back empty it finishes FAILED and bills **nothing at all**, start fee included.

Set `ACTOR_MAX_TOTAL_CHARGE_USD` on a run and the Actor stops fetching once the ceiling is in sight, rather than handing you rows it cannot bill or billing you for rows it cannot hand over. It finishes SUCCEEDED with the ceiling named in its status message, and everything already delivered is yours.

### Limits, and the ones that might bite

**No browser, so the audit is of the HTML the server returned.** A page whose body copy is rendered client-side will read as `thin-content` with a low `textToHtmlRatio`, and that is an honest description of what a crawler receives — but it is not what a human sees, and treating it as a content problem would be wrong. The `wordCount` and `textToHtmlRatio` columns together are the tell: a low ratio on a big document means a framework, not an empty page.

**`sitemapUrlCount` is what was read, not everything the site publishes.** Sitemap collection stops as soon as there are enough URLs to fill `maxPages`, at 50 sitemap files, at three levels of index nesting, or at `maxSitemapUrls`. Reading a news site's entire sitemap tree to audit 100 pages would be rude as well as slow. Raise `maxPages` and `maxSitemapUrls` together when you want a complete count.

**`redirectChain` has two ends, not every hop.** The HTTP client follows redirects itself and hands back only the URL it finished on, so a page that went through three hops reports the URL requested and the URL delivered. The `redirected` issue and `finalUrl` are correct; the intermediate hops are not visible.

**Duplicate checks are within one run and one site.** `duplicate-title` compares the pages *this run audited on this website*. Two pages sharing a title where only one was audited will not be flagged, and a title shared across two different domains in the same run is not a duplicate.

**`compressed` can be `null`.** See the `not-compressed` note above. When the response carried neither `content-encoding` nor `content-length` and the body was empty, nothing can be concluded, and the column says so rather than guessing.

**A crawl is not a full-site crawl.** The fallback follows internal HTML links breadth-first from the home page up to `maxPages`. It does not submit forms, does not paginate listings by guessing URL patterns, and does not leave the host. A site with no sitemap and a JavaScript-only navigation will yield one page: its home page.

**Scores are a checklist total, not a ranking prediction.** Nothing here measures backlinks, domain authority, search intent, competitor positioning or whether the content is any good. A page can score 100 and rank nowhere. The score is for triage — sort by it, then read the `issues`.

**Sites behind aggressive bot management may refuse the default proxy.** The datacenter default is right for the overwhelming majority of the web and wrong for the small slice that fingerprints every request. Those pages come back as free `blocked` diagnostic rows; switch `proxyConfiguration` to `RESIDENTIAL` and re-run if you need them.

**The upstream format may change.** Sites are rebuilt, sitemaps move, and a site that published a clean `sitemap.xml` last month can start serving an index of gzip children this month. When that happens the discovery path changes rather than breaking — the `discovery` and `sitemapUrls` columns on the site row are how you see which path was taken.

### Use it from an AI agent, or from code

One JSON object in, one flat array out — the shape agent runtimes want. The Actor runs with **limited permissions**, uses **pay-per-event** pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. The **Integrations** tab pushes results to Slack, a webhook, Zapier, Make, Google Sheets, Snowflake or BigQuery.

```bash
curl -X POST "https://api.apify.com/v2/acts/insight.solutions~seo-page-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"websites":["basecamp.com"],"maxPages":50}'
```

```python
## pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/seo-page-audit").call(run_input={
    "websites": ["basecamp.com"],
    "maxPages": 200,
    "checkLinks": True,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "page" and row["score"] < 80:
        print(row["score"], row["url"], "→", ", ".join(row["issues"]))
    elif row["rowType"] == "site":
        print("site score:", row["score"], "over", row["pagesAudited"], "pages")
    elif row["rowType"] == "broken-link":
        print("broken:", row["url"], "linked from", row["foundOn"][0])
```

The four dataset views — **Overview**, **Pages with issues**, **Broken links** and **Site summaries** — are set up in the Console so each row type exports as its own clean table.

### FAQ

**How does it find the pages?**
`robots.txt` first, for the `Sitemap:` lines and the crawl rules. Then those sitemaps, or `/sitemap.xml` and three other usual locations. Sitemap indexes are followed to their children, gzip children are inflated, and relative `<loc>` values are resolved. Only if none of that yields anything does it crawl the site from its home page.

**Does it respect robots.txt?**
Yes, by default, using the `User-agent: *` group and the longest-match-wins rule search engines use. Disallowed URLs are never fetched, counted as `robotsBlocked` on the site row, and never billed. `respectRobots: false` exists for sites you own — for example a staging site that blocks everything.

**How fast is it?**
Requests to one host are spaced 150–400 ms apart, so about three pages a second per site plus the time each page takes to answer, overlapped across `maxConcurrency` workers. 50 pages of a typical site is well under a minute. `maxRunSecs` is the hard stop.

**Why did a page I know exists not get audited?**
Four possibilities, in order of likelihood: it is not in the sitemap; `maxPages` was reached before the run got to it; `robots.txt` disallows it; or it answered with something that is not a web page. The site row's `sitemapUrlCount`, `pagesAudited` and `robotsBlocked` distinguish the first three, and there is a free diagnostic row for the fourth.

**What counts as a duplicate title?**
An exact string match, after entity decoding and whitespace collapsing, between two pages audited on the same website in the same run. Near-duplicates are not detected — "Shoes | Acme" and "Shoes - Acme" are two different titles.

**Can I audit just a handful of specific pages?**
Yes. Put them in `pageUrls` and leave `websites` empty. No sitemap is fetched and no crawl happens; only `robots.txt` is read, because the Disallow rules still apply.

**Does `checkLinks` check external links too?**
No. Only internal links, deduplicated across the whole site, capped at `maxLinksToCheck`. Checking outbound links would multiply the request count against sites that did not ask to be crawled.

**Can I audit 5,000 pages?**
Yes, that is the per-website ceiling. Raise `maxRunSecs` to match — 5,000 pages at three a second is roughly half an hour — and bound the spend with `ACTOR_MAX_TOTAL_CHARGE_USD`. At $0.001 per page the whole site is $5.

**What happens if one website in a list fails?**
The others still run. The failed one produces a free diagnostic row and the run finishes `SUCCEEDED`. If *every* website fails, the run finishes `FAILED` and you are billed nothing at all.

**Is the data fresh?**
Live. Every run reads the site at that moment; nothing is cached, and there is no index behind this.

### Legal and data-protection notes

- **Public pages only.** Every URL fetched is one a browser would load without signing in. The Actor never logs in, never accepts cookies or session tokens, never takes an API key belonging to anyone else, and never touches anything behind authentication.
- **It obeys the site's own instructions by default.** `robots.txt` is read before anything else and its `Disallow` rules are respected unless you deliberately turn that off for a site you control.
- **It is a polite crawler.** One request at a time per host with 150–400 ms between them, a 20-second per-request timeout, and a whole-run budget. It does not retry a refusal in a loop.
- **This is not personal data.** An on-page SEO record describes a web page's markup, not a person. Nothing in the output identifies an individual, and no contact details, names or user records are collected.
- **Scores are observations, not advice.** "This page has no meta description" is what was seen. What to do about it depends on the page, and the issue list is there precisely so the number never has to be taken on faith.
- **Not affiliated with any website audited or named in these docs.** All trademarks belong to their respective owners.

### Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

**Video, audio & social**

- [YouTube Transcript API](https://apify.com/insight.solutions/youtube-transcript-api) — captions as timed segments, text, SRT or VTT, with language fallback and translation.
- [YouTube Comments API](https://apify.com/insight.solutions/youtube-comments-api) — comments and replies with likes, pinned and hearted flags, newest or top sort.
- [YouTube Channel API](https://apify.com/insight.solutions/youtube-channel-api) — a channel's videos, Shorts and live streams, plus YouTube search.
- [Podcast Search, Episodes & Charts API](https://apify.com/insight.solutions/podcast-api) — Apple Podcasts search, charts and full episode feeds.
- [Bluesky Scraper](https://apify.com/insight.solutions/bluesky-scraper) — profiles, posts, followers and follows from the public AT Protocol API.
- [Telegram Channel Scraper](https://apify.com/insight.solutions/telegram-channel-scraper) — posts, views and channel stats from public Telegram channels.
- [Substack Scraper](https://apify.com/insight.solutions/substack-scraper) — posts with full free text, comments and publication profiles.

**News, documents & the web**

- [Google News Search, Topics & Real Article URLs](https://apify.com/insight.solutions/google-news-api) — news search and topic feeds with the publisher's real URL decoded.
- [Website to Markdown — Content Extractor for LLMs & RAG](https://apify.com/insight.solutions/website-content-extractor) — any site as clean Markdown, text and heading-aware chunks.
- [Internet Archive API](https://apify.com/insight.solutions/internet-archive-api) — archive.org search, item metadata, files and reviews.
- [Wayback Machine Toolkit](https://apify.com/insight.solutions/wayback-toolkit) — archived URL inventories, snapshots and text diffs between dates.
- [Website Technology Detector](https://apify.com/insight.solutions/website-tech-detector) — the tech stack behind any site, with the evidence for each detection.
- [Domain Intelligence API](https://apify.com/insight.solutions/domain-intelligence-api) — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
- [Keyword Suggestions API](https://apify.com/insight.solutions/keyword-suggestions-api) — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
- [Website Contact Extractor](https://apify.com/insight.solutions/website-contact-extractor) — emails, phone numbers and social profiles from any list of websites.

**Business, finance & jobs**

- [Congress & Insider Trades API](https://apify.com/insight.solutions/congress-insider-trades-api) — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
- [SEC EDGAR API](https://apify.com/insight.solutions/sec-edgar-api) — filings, XBRL financials and full-text search by ticker or CIK.
- [Y Combinator Companies, Batches & Founders](https://apify.com/insight.solutions/yc-companies-directory) — the YC directory with founders and social links, filterable by batch, industry and hiring status.
- [Career Site Jobs API](https://apify.com/insight.solutions/ats-jobs-api) — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
- [New Job Postings Monitor](https://apify.com/insight.solutions/job-postings-monitor) — new, closed and changed postings on the career sites you watch.
- [Shopify Products API](https://apify.com/insight.solutions/shopify-products-api) — any Shopify store's catalogue, variants, prices and stock signals.

**Apps & games**

- [App Store & Google Play Reviews API](https://apify.com/insight.solutions/app-reviews-api) — reviews from both stores with ratings, versions and developer replies.
- [App Store Top Charts & App Search API](https://apify.com/insight.solutions/app-charts-api) — Apple top charts by country and genre, plus app search and details.
- [Steam Reviews API](https://apify.com/insight.solutions/steam-reviews-api) — Steam reviews with playtime, helpfulness and game details.
- [Steam Game Data API](https://apify.com/insight.solutions/steam-store-stats-api) — prices, tags, review scores, live player counts and top charts.

# Actor input Schema

## `websites` (type: `array`):

One entry per site to audit. A bare domain is enough — `basecamp.com` — and so is a full URL. Each domain is tried on `https://`, then `https://www.`, then plain `http://`, so a host that only answers on one of them still gets read. For each site the Actor reads `robots.txt`, follows the `Sitemap:` lines it finds (or tries `/sitemap.xml` and three other usual locations), and audits up to **Maximum pages per website** URLs. A site that publishes no sitemap at all is crawled breadth-first from its home page instead. Email addresses and IP addresses are rejected for free.

## `pageUrls` (type: `array`):

Audit exactly these pages and nothing else — no sitemap, no crawl. Use it when you already know which URLs matter: the twelve landing pages you just shipped, the product pages a client complained about, a list exported from your analytics. URLs on the same host as an entry in **Websites** are folded into that site's audit rather than becoming a second one, so they still count toward its duplicate-title check and its site summary row.

## `maxPages` (type: `integer`):

How many pages to audit per website. This is the number that sets the price of a run: every audited page is one `page` event. The default of 100 is a useful sample of a mid-size site; raise it to cover a site completely, lower it to spot-check. Sitemap URLs are taken in the order the sitemap lists them; a crawl takes them breadth-first from the home page, shallowest links first.

## `checkLinks` (type: `boolean`):

Off by default. Turn it on and every unique internal link found on the audited pages is fetched once (a plain `GET` — `HEAD` stalls on some CDNs), and anything answering 4xx, 5xx or nothing at all gets a `broken-link` row naming the pages that link to it. Links are deduplicated across the whole site — a bad link in the footer of 400 pages is one check and one row — and pages that were audited anyway are skipped, because their status is already known. Billed at $0.0002 per link that turns out to be broken; checking costs nothing.

## `maxLinksToCheck` (type: `integer`):

The ceiling on the link check, per website. Only used when **Check internal links** is on. Unique internal links are collected across every audited page and then cut to this number.

## `respectRobots` (type: `boolean`):

On by default, and worth leaving on. The Actor reads the `User-agent: *` group of each site's `robots.txt` and skips any URL it disallows, using the same longest-match-wins rule search engines use, with `*` and `$` patterns supported. Skipped URLs are counted on the `site` row as `robotsBlocked` and are never fetched or billed. Turn it off only for a site you own or have permission to crawl — for example to audit a staging site that blocks everything.

## `maxSitemapUrls` (type: `integer`):

How many URLs to read out of a site's sitemaps before stopping. A large news site's sitemap tree runs to millions of entries and reading all of it to audit 100 pages would be rude as well as slow, so collection also stops as soon as there are enough URLs to fill **Maximum pages per website**. Sitemap indexes are followed three levels deep and at most 50 sitemap files are fetched per website.

## `maxConcurrency` (type: `integer`):

How many pages of one website are fetched at once. Each parallel worker keeps its own proxy session, so a page that refuses one exit IP does not burn the address the others are using. Requests to a single host are spaced 150–400 ms apart whatever this is set to, so raising it buys parallel parsing rather than a burst of simultaneous requests. Websites themselves are always audited one at a time.

## `maxRunSecs` (type: `integer`):

Wall-clock budget for the whole run. When it is reached the Actor stops fetching, keeps and bills for every row it already wrote, and files a free diagnostic row saying how many URLs it never reached. Nothing is charged for work that did not happen.

## `proxyConfiguration` (type: `object`):

Ordinary websites do not refuse datacenter address ranges the way the big platforms do, so the default is Apify's datacenter proxy and its cost is already inside the per-page price. If you are auditing a site behind an aggressive bot wall, switch the group to RESIDENTIAL here and nothing else changes — sessions still rotate automatically when an exit IP is refused. Clearing the field sends requests from the run's own address.

## Actor input object example

```json
{
  "websites": [
    "apify.com",
    "https://www.gov.uk"
  ],
  "pageUrls": [
    "https://basecamp.com/pricing",
    "https://basecamp.com/about"
  ],
  "maxPages": 50,
  "checkLinks": false,
  "maxLinksToCheck": 500,
  "respectRobots": true,
  "maxSitemapUrls": 5000,
  "maxConcurrency": 8,
  "maxRunSecs": 240,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One row per audited page — title, description, headings, canonical, structured data, images, links, speed, issue codes and a 0-100 score — plus a summary row per website and a row for every broken internal link. Delivered as JSON items in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "basecamp.com"
    ],
    "pageUrls": [],
    "maxPages": 50,
    "checkLinks": false,
    "maxLinksToCheck": 500,
    "respectRobots": true,
    "maxSitemapUrls": 5000,
    "maxConcurrency": 8,
    "maxRunSecs": 240,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("insight.solutions/seo-page-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websites": ["basecamp.com"],
    "pageUrls": [],
    "maxPages": 50,
    "checkLinks": False,
    "maxLinksToCheck": 500,
    "respectRobots": True,
    "maxSitemapUrls": 5000,
    "maxConcurrency": 8,
    "maxRunSecs": 240,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("insight.solutions/seo-page-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "basecamp.com"
  ],
  "pageUrls": [],
  "maxPages": 50,
  "checkLinks": false,
  "maxLinksToCheck": 500,
  "respectRobots": true,
  "maxSitemapUrls": 5000,
  "maxConcurrency": 8,
  "maxRunSecs": 240,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call insight.solutions/seo-page-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,insight.solutions/seo-page-audit"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IbdSbOhXIRfYOPMKq/builds/Rs16hk2iXdOZNgyxB/openapi.json