# Site Audit: SEO, Broken Links & AI-Crawler Check (`gapforge/site-audit`) Actor

Audit a public website over plain HTTP: on-page SEO, broken links, sitemap and robots.txt, llms.txt and AI-crawler rules, security headers and the TLS certificate. One row per page, with each issue's severity, reason and fix.

- **URL**: https://apify.com/gapforge/site-audit.md
- **Developed by:** [GapForge](https://apify.com/gapforge) (community)
- **Categories:** SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 page audits

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Site Audit check?

Site Audit crawls a website you name and reports what is wrong with it, page by page: on-page SEO basics, broken links, sitemap and robots.txt problems, whether AI crawlers may read the site (robots.txt rules for 12 AI bots, plus `llms.txt`), security headers and the TLS certificate. Each finding comes with a severity, a plain-words reason and a fix. The easiest way to try it: open the Input tab and click **Start**. The prefilled site returns results in well under a minute.

It works over plain HTTP: no browser, no login, no proxy. It reads the HTML the server sends, the way a search or AI crawler that does not run JavaScript reads it. It identifies itself honestly as `GapForgeSiteAudit` and follows the site's robots.txt, including Crawl-delay.

Turn on **Only new or fixed issues since last run**, add a schedule, and it becomes a monitor: a run reports just the pages whose issues changed. That suits agencies and freelancers who watch client sites, developers who want an audit step after each deploy, and site owners who want to hear about a broken link or an expiring certificate before a visitor does.

### Why use it

- **Several audits in one run.** On-page SEO, broken links, sitemap, robots.txt, AI-crawler access, `llms.txt`, security headers and TLS, in one dataset.
- **Findings you can act on.** A fixed list of documented issue codes. Each issue states its severity (`error`, `warning` or `notice`), what was found and how to fix it.
- **Documented, not a black box.** The tables below say exactly what each check tests. The **Limits** section says what the tool does not do.
- **Built for monitoring.** Monitor mode remembers the issues it has already reported and tells you what is new and what was fixed.
- **Polite by design.** It follows robots.txt, keeps to a per-host request rate you choose, and never logs in.

### What data do you get?

Two kinds of rows, told apart by `type`.

**One `site` row per site (free).** It is saved before the site's page rows.

| Field | Meaning |
|---|---|
| `site`, `url` | Scheme and host of the site, and the start address you gave |
| `reachable` | `false` when the site did not answer at all (also when its TLS certificate was rejected) |
| `auditedAt` | When the audit of this site started (UTC, ISO 8601) |
| `pagesAudited`, `pagesDelivered` | Page rows produced, and page rows saved and charged |
| `pagesSkippedByRobots`, `skippedUrls` | Addresses not audited because robots.txt disallows them or the address they redirect to (count, and the first 50) |
| `pagesFailed`, `failedUrls` | Addresses that did not answer (count, and the first 50). They are not charged |
| `maxPagesReached`, `chargeLimitReached` | `true` when the audit stopped at your page limit, or at your maximum charge |
| `internalLinksNotChecked` | Links to pages or files of the site that were not tested, for example because they lie beyond the page limit or the 200-per-site file-link limit. Links that robots.txt disallows are not counted here: disallowed pages are in `pagesSkippedByRobots`, disallowed files are not counted |
| `externalLinksChecked`, `externalLinksNotChecked` | Links to other sites that were tested, and links left untested because of the external link limit or because the other host's robots.txt disallows the request, cannot be read, or asks for more than 60 seconds between requests. Both are 0 when **Check links to other sites** is off |
| `robotsTxt`, `crawlDelaySeconds` | `found`, `missing`, `invalid` or `unreachable`, and the Crawl-delay that applies to this tool |
| `sitemaps`, `sitemapUrlCount`, `sitemapInvalidEntries`, `sitemapTruncated` | Sitemap files read, page addresses listed in them, invalid files or entries, and whether the sitemaps were too large to read in full |
| `llmsTxt`, `llmsFullTxt`, `llmsTxtValid` | Whether `/llms.txt` and `/llms-full.txt` exist, and whether `/llms.txt` has the basic format |
| `aiBots`, `aiBotsBlocked` | What robots.txt says to each AI crawler about the home page (`allowed`, `blocked` or `not-mentioned`), and the list of crawlers that may not fetch it |
| `httpsRedirect` | Whether the `http://` address redirects to `https://` |
| `securityHeaders`, `securityHeadersMissing` | For six security headers: whether the start page sends each one, and the list of missing ones |
| `tlsValid`, `tlsDaysLeft`, `tlsValidUntil`, `tlsIssuer` | TLS certificate: trusted and matching the host, days until it expires, expiry time, issuing authority |
| `issues`, `issueCount`, `errors`, `warnings`, `notices` | Site-level issues, and totals for the whole site (site-level issues plus the issues of all audited pages) |

**One `page` row per audited page.**

| Field | Meaning |
|---|---|
| `site`, `url`, `finalUrl` | The site, the address that was requested and the address after redirects |
| `status`, `redirectChain` | HTTP status of the final answer, and each redirect hop on the way (`url`, `status`) |
| `responseMs`, `contentType`, `bytes` | Time to receive the page as seen from the run, its Content-Type and the size of the HTML read |
| `title`, `titleLength` | Text of the `<title>` tag and its length in characters |
| `metaDescription`, `metaDescriptionLength` | Meta description and its length in characters |
| `canonical`, `canonicalUrl` | `self`, `other` or `missing`, and the address the canonical link points to |
| `metaRobots`, `xRobotsTag`, `noindex`, `nofollow` | The meta robots tag, the X-Robots-Tag header, and what they mean for indexing and link following |
| `h1Count`, `lang`, `hreflangCount` | Number of `<h1>` headings, the `<html lang>` value, number of hreflang links |
| `jsonLdTypes`, `jsonLdErrors` | `@type` values found in JSON-LD blocks, and the number of blocks that are not valid JSON |
| `imagesMissingAlt` | `<img>` tags with no `alt` attribute |
| `internalLinks`, `externalLinks` | Number of different link targets on the same site and on other sites |
| `brokenLinks` | Dead link targets found on the page, as `url`, `status` and `reason` |
| `inSitemap`, `depth` | Whether the page is listed in the sitemap, and how many clicks it is from the start address |
| `issues`, `issueCount`, `errors`, `warnings`, `notices` | The page's issues (`code`, `severity`, `message`, `fix`, `changeType`) and their counts |

The rows hold facts about pages and servers. Links that start with `mailto:` or `tel:` are ignored and never stored.

### The checks

What is tested, and where the result shows up.

| Check | What it tests | What it does not test |
|---|---|---|
| Pages and status | Each HTML page reached from the start address by links (and from the sitemap, if switched on): status, redirects, response time | Pages that need JavaScript, a form or a login to appear |
| Titles and descriptions | Presence, length and duplicates of `<title>` and the meta description among audited pages | Wording quality or keyword use |
| Headings, language, viewport | `<h1>` count, `<html lang>`, the viewport meta tag | Heading order below `<h1>` |
| Canonical and indexing | `rel="canonical"` (self, other, missing, conflicting), `noindex` and `nofollow` in meta robots and X-Robots-Tag | Canonicals sent in an HTTP `Link` header |
| Structured data | JSON-LD blocks: which `@type` values are present, and whether each block is valid JSON | Microdata and RDFa; whether the markup qualifies for a rich result |
| Images | `<img>` tags with no `alt` attribute | Whether the alt text is good; images added by scripts |
| Mixed content | Images, scripts, stylesheets, frames and media loaded over `http://` on an `https://` page | Resources requested by scripts or CSS |
| Internal links | Links between pages of the site: each target inside the crawl is fetched; file links (PDF, images and similar) get a light status request | Targets beyond your page limit (counted in `internalLinksNotChecked`) |
| External links (optional) | Links to other sites: a HEAD request, then GET if HEAD fails. A redirect counts as a working link | Where a redirect ends up; hosts whose robots.txt disallows the request |
| robots.txt | Present, readable and valid; Crawl-delay; rules for this tool | Rules for crawlers not listed below |
| Sitemaps | Sitemaps named in robots.txt plus `/sitemap.xml`; index files, gzip files and plain-text lists; invalid entries; pages missing from the sitemap | Image, video and news sitemap extensions |
| AI-crawler access (optional) | robots.txt rules for the home page for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and meta-externalagent | Blocks made by a firewall or CDN instead of robots.txt |
| llms.txt (optional) | `/llms.txt` and `/llms-full.txt` exist; `/llms.txt` starts with a `# ` title line and has a Markdown link | The quality or completeness of the file |
| Security headers (optional) | Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options, X-Frame-Options (or CSP `frame-ancestors`), Referrer-Policy and Permissions-Policy on the start page; the `http://` to `https://` redirect | Whether a policy's contents are strict enough; headers on other pages |
| TLS certificate (optional) | Trusted and matching the host; days until expiry; issuer | Protocol versions and cipher strength |

In `aiBots`, `not-mentioned` means robots.txt has no group for that crawler, so the general `*` rules apply to it. `aiBotsBlocked` lists the crawlers that may not fetch the home page under whichever rules apply to them.

### Issue codes

The list is fixed. `error` means broken for visitors or crawlers, `warning` means likely to hurt, `notice` means worth a look.

| Code | Severity | What triggers it | How to fix it |
|---|---|---|---|
| `HTTP_4XX` | error | The start URL or a URL listed in the sitemap answers with a 4xx status (for example 404). | Restore the page, redirect it to its replacement, or remove it from the sitemap and from links. |
| `HTTP_5XX` | error | The start URL or a URL listed in the sitemap answers with a 5xx status (server error). | Check the server logs for this URL and fix the error. |
| `REDIRECT_LOOP` | error | The start URL or a sitemap URL redirects in a loop or through more than 10 redirects. | Point the first URL straight at the final page. |
| `REDIRECT_OFFSITE` | warning | The start URL or a sitemap URL redirects to a different site. | If that is intended, audit the other site instead; otherwise fix the redirect. |
| `REDIRECT_CHAIN` | notice | The page was reached through 2 or more redirects in a row. | Redirect the first URL straight to the final page. |
| `SITEMAP_URL_REDIRECTS` | notice | A URL listed in the sitemap redirects to this page. | List the final URL in the sitemap. |
| `BROKEN_INTERNAL_LINK` | error | The page links to a URL on the same site that answers 4xx or 5xx (401, 403 and 429 are not counted), redirects in a loop, or does not answer. | Fix or remove each link listed in brokenLinks. |
| `BROKEN_EXTERNAL_LINK` | warning | With external link checking on: the page links to a URL on another site that answers 4xx or 5xx (401, 403 and 429 are not counted) or does not answer. | Update or remove each link listed in brokenLinks. |
| `TITLE_MISSING` | error | The page has no `<title>`, or it is empty. | Add a unique, descriptive `<title>`. |
| `TITLE_TOO_LONG` | notice | The title is longer than 60 characters. | Shorten the title; search results usually cut long titles. |
| `TITLE_DUPLICATE` | warning | Another audited page that can be indexed has the same title. | Give each page its own title. |
| `META_DESCRIPTION_MISSING` | warning | The page has no meta description, or it is empty. | Add a meta description that summarizes the page. |
| `META_DESCRIPTION_TOO_LONG` | notice | The meta description is longer than 160 characters. | Shorten the meta description; search results usually cut long ones. |
| `META_DESCRIPTION_DUPLICATE` | notice | Another audited page that can be indexed has the same meta description. | Give each page its own meta description. |
| `H1_MISSING` | warning | The page has no `<h1>` heading. | Add one `<h1>` that states the page's topic. |
| `H1_MULTIPLE` | notice | The page has more than one `<h1>` heading. | Keep one main `<h1>` unless the layout needs more. |
| `CANONICAL_MISSING` | notice | The page has no rel="canonical" link. | Add a canonical link that points to the page's preferred URL. |
| `CANONICAL_OTHER` | notice | The canonical link points to a different URL than the page itself. | Confirm this is intended: search engines are asked to index the other URL instead. |
| `CANONICAL_MULTIPLE` | warning | The page has canonical links that point to different URLs. | Keep one canonical link. |
| `NOINDEX` | notice | The page asks search engines not to index it (meta robots or X-Robots-Tag: noindex). | Remove noindex if the page should appear in search results. |
| `NOINDEX_IN_SITEMAP` | warning | The page is listed in the sitemap but asks not to be indexed. | Remove it from the sitemap, or remove noindex. |
| `NOFOLLOW` | notice | The page tells crawlers not to follow its links (meta robots or X-Robots-Tag: nofollow). This audit does not follow them either. | Remove nofollow if the linked pages should be discovered through this page. |
| `LANG_MISSING` | notice | The `<html>` tag has no lang attribute. | Add lang, for example `<html lang="en">`. |
| `VIEWPORT_MISSING` | notice | The page has no `<meta name="viewport">` tag. | Add a viewport meta tag so the page scales on phones. |
| `JSONLD_INVALID` | warning | A JSON-LD block (`<script type="application/ld+json">`) is not valid JSON. | Fix the JSON syntax of the structured data block. |
| `IMG_ALT_MISSING` | notice | One or more `<img>` tags have no alt attribute (an empty alt="" is accepted). | Add alt text, or alt="" for decorative images. |
| `MIXED_CONTENT` | warning | An https page loads an image, script, stylesheet, frame or media file over http. | Load all resources over https. |
| `NOT_IN_SITEMAP` | notice | The site has a sitemap, and this page can be indexed but is not listed in it. | Add the page to the sitemap, or ignore this if it is left out on purpose. |
| `SITE_UNREACHABLE` | error | The site did not answer (DNS, connection, TLS or timeout error). No page was audited and no page was charged. | Check the address and that the site is online, then run again. |
| `START_NOT_A_PAGE` | warning | The start URL answers, but not with an HTML page (for example a file), so it was not audited as a page. | Start from an address that serves a web page, such as the home page. |
| `ROBOTS_MISSING` | notice | robots.txt answers with a 4xx status (usually 404). | Add a robots.txt, even a short one, and list your sitemap in it. |
| `ROBOTS_UNREACHABLE` | error | robots.txt answers with a server error (5xx) or redirects without end. Crawlers treat the whole site as off limits, and so does this audit. | Make robots.txt answer 200 (or 404 if you have none). |
| `ROBOTS_INVALID` | warning | robots.txt is an HTML page, or has text but no line a crawler understands. | Serve a plain-text robots.txt with User-agent, Allow, Disallow or Sitemap lines. |
| `ROBOTS_BLOCKS_AUDIT` | warning | robots.txt does not allow this tool to fetch the start URL, so no page was audited. | To audit the site, allow the user agent GapForgeSiteAudit in robots.txt. |
| `CRAWL_DELAY_TOO_LONG` | warning | robots.txt asks for more than 60 seconds between requests, so no page was audited. | Lower Crawl-delay for the user agent GapForgeSiteAudit to audit the site. |
| `SITEMAP_MISSING` | warning | No sitemap was found (none listed in robots.txt and none at /sitemap.xml). | Publish a sitemap and list it in robots.txt with a Sitemap line. |
| `SITEMAP_INVALID_ENTRIES` | warning | A sitemap file could not be read, or lists entries that are not valid URLs of this site. | Fix or remove the invalid sitemap files and entries. |
| `LLMS_TXT_MISSING` | notice | With AI-readiness checks on: /llms.txt is absent (or answers with an HTML page). | Optional: publish /llms.txt, a short Markdown guide to the site for AI assistants. |
| `LLMS_TXT_FORMAT` | warning | With AI-readiness checks on: /llms.txt does not start with a "# " title line or has no Markdown link. | Start the file with a line "# Site name" and link the key pages as [name](url). |
| `AI_BOTS_BLOCKED` | notice | With AI-readiness checks on: robots.txt does not let one or more of the listed AI crawlers fetch the home page. | Nothing to fix if that is your choice; otherwise allow those user agents in robots.txt. |
| `NOT_HTTPS` | warning | With security checks on: the start page is served over http, not https. | Serve the site over https. |
| `HTTPS_REDIRECT_MISSING` | warning | With security checks on: the http:// address answers without redirecting to https://. | Redirect all http requests to https with a 301. |
| `HSTS_MISSING` | warning | With security checks on: the https start page sends no Strict-Transport-Security header. | Send Strict-Transport-Security, for example max-age=31536000. |
| `CSP_MISSING` | notice | With security checks on: the start page sends no Content-Security-Policy header. | Add a Content-Security-Policy that lists the sources the site needs. |
| `CONTENT_TYPE_OPTIONS_MISSING` | notice | With security checks on: the start page does not send X-Content-Type-Options: nosniff. | Send X-Content-Type-Options: nosniff. |
| `FRAME_PROTECTION_MISSING` | notice | With security checks on: the start page sends neither X-Frame-Options nor a Content-Security-Policy with frame-ancestors. | Send X-Frame-Options: SAMEORIGIN or a CSP frame-ancestors rule. |
| `REFERRER_POLICY_MISSING` | notice | With security checks on: the start page sends no Referrer-Policy header. | Send Referrer-Policy, for example strict-origin-when-cross-origin. |
| `PERMISSIONS_POLICY_MISSING` | notice | With security checks on: the start page sends no Permissions-Policy header. | Send a Permissions-Policy that switches off the browser features the site does not use. |
| `TLS_INVALID` | error | With security checks on: the TLS certificate is expired, not trusted or does not match the host name. No page can then be read, so the site row also says `SITE_UNREACHABLE` and no page is audited. | Renew or reinstall the certificate. |
| `TLS_EXPIRES_SOON` | warning | With security checks on: the TLS certificate expires in 30 days or fewer. | Renew the certificate. |

### How to use it (step by step)

1. Open the Input tab. The prefilled site already works.
2. In **Sites to audit**, add one address per site, for example `https://www.your-site.com`. The audit starts there and stays on that host (with or without `www`).
3. Set **Maximum pages per site**. You pay per page row, so this also caps the cost.
4. Optional: switch on **Check links to other sites**, and switch **Security checks** or **AI-readiness checks** off if you do not need them.
5. For monitoring, turn on **Only new or fixed issues since last run** and add a schedule (weekly works well). Use a different **State key** for each schedule so their memories stay separate.
6. Click **Start**. Results appear in the Output tab as tables (Overview, Pages, Issues, plus All fields) and can be exported as JSON, CSV or Excel.

### Monitor mode

With **Only new or fixed issues since last run** on, the tool remembers which issues each page had. A page row is saved (and charged) when the page has an issue it did not have last time, or when an issue it had is gone. Each issue then carries a `changeType`: `new`, `unchanged` or `resolved`. A broken-link issue is remembered together with its list of dead links: when that list changes (a link breaks, or one of several is fixed), the issue is reported as `new` again with the current list, and `resolved` appears when the page has no such broken link left. On a page, `resolved` can also mean that the dead link was not tested in this run (it lay beyond the page, file-link or external-link limit, or **Check links to other sites** was switched off); when a limit was the cause, the site row's `internalLinksNotChecked` or `externalLinksNotChecked` counts those links. `AI_BOTS_BLOCKED` works the same way with its list of crawlers. A page with no change is checked but not saved and not charged. The first run with a state key reports each current issue as `new`. In a normal run `changeType` is `snapshot`.

The site row is saved in each run, free, with its counts for the whole site. A site-level issue is reported as `resolved` only when its check ran in this run and no longer finds it; when the site cannot be reached, or a check is switched off, the earlier site-level issues are kept as they were. The memory is a named key-value store in your own Apify account called `site-audit-state-KEY`, where KEY is your state key. Delete it to start fresh.

### How much does a site audit cost?

You pay per page audited, after the page row is saved:

- **$0.005 per page row**, which is **$5 per 1,000 pages audited** (less on paid Apify plans, down to $3.50 per 1,000 pages; the Pricing tab shows your price).
- Apify also charges a small start fee of $0.00005 per run.
- The site summary row is free. A site that cannot be reached and a page that does not answer add no page charge; such a run costs only the start fee.
- A start address or sitemap address that answers with an error, loops or leaves the site is a page row too (it carries the finding) and is charged like one.

Examples: auditing a 100-page site costs about $0.50. A weekly monitor run over the same site that finds 4 pages with new or fixed issues costs about $0.02. Apify's free plan includes $5 of monthly credit, enough to audit 999 pages in one run. The tool stops cleanly when it reaches the maximum charge you set for the run and tells you so in the status message.

### Input example

```json
{
  "startUrls": ["https://www.your-site.com"],
  "maxPagesPerSite": 100,
  "useSitemap": true,
  "maxRequestsPerSecond": 2,
  "checkExternalLinks": false,
  "maxExternalLinksPerSite": 200,
  "checkSecurityHeaders": true,
  "checkAiReadiness": true,
  "onlyChanges": false,
  "stateKey": "weekly-my-site"
}
```

### Output example

A page row:

```json
{
  "type": "page",
  "site": "https://example.com",
  "url": "https://example.com/pricing",
  "finalUrl": "https://example.com/pricing",
  "status": 200,
  "redirectChain": [],
  "responseMs": 182,
  "contentType": "text/html; charset=utf-8",
  "bytes": 18734,
  "title": "Pricing",
  "titleLength": 7,
  "metaDescription": null,
  "metaDescriptionLength": 0,
  "canonical": "self",
  "canonicalUrl": "https://example.com/pricing",
  "metaRobots": null,
  "xRobotsTag": null,
  "noindex": false,
  "nofollow": false,
  "h1Count": 1,
  "lang": "en",
  "hreflangCount": 0,
  "jsonLdTypes": ["Product"],
  "jsonLdErrors": 0,
  "imagesMissingAlt": 2,
  "internalLinks": 14,
  "externalLinks": 3,
  "brokenLinks": [{"url": "https://example.com/old-plan", "status": 404, "reason": "status 404"}],
  "inSitemap": true,
  "depth": 1,
  "issues": [
    {"code": "BROKEN_INTERNAL_LINK", "severity": "error", "message": "1 broken link(s) to this site, first: https://example.com/old-plan (status 404).", "fix": "Fix or remove each link listed in brokenLinks.", "changeType": "snapshot"},
    {"code": "META_DESCRIPTION_MISSING", "severity": "warning", "message": "The page has no meta description, or it is empty.", "fix": "Add a meta description that summarizes the page.", "changeType": "snapshot"},
    {"code": "IMG_ALT_MISSING", "severity": "notice", "message": "2 image(s) have no alt attribute.", "fix": "Add alt text, or alt=\"\" for decorative images.", "changeType": "snapshot"}
  ],
  "issueCount": 3,
  "errors": 1,
  "warnings": 1,
  "notices": 1
}
```

The values above illustrate the format.

### Integrations and API

Run it from the Apify API, the Apify CLI or the Apify clients for Python and JavaScript, on a schedule, or from automation tools that connect to Apify (for example Make, n8n or Zapier). A webhook can notify your system when a run finishes, so new issues can flow straight into a ticket tracker, a chat channel or a spreadsheet. AI agents can call it through the Apify MCP server; the input and output schemas tell them what to send and what comes back.

### FAQ and troubleshooting

#### Is it legal?

You audit a site you name. The tool fetches public pages the way a search crawler does, follows the site's robots.txt, keeps to a polite request rate and never logs in. Audit sites you own or are allowed to test, and check your own use against the laws and site terms that apply to you.

#### Will it overload my site?

No more than a small crawler would. Requests to one host are sent one at a time, at the rate set in **Maximum requests per second** (2 by default, 5 at most). If your robots.txt sets a Crawl-delay that is slower, the tool waits that long between requests.

#### Why are some pages missing?

Four possible reasons. The first three are shown in the site row. robots.txt disallows them (`skippedUrls`). The audit reached your page limit (`maxPagesReached`). The page did not answer (`failedUrls`). Or reaching the page takes JavaScript, a form or a login, which this tool does not use. Pages listed in the sitemap are found even when nothing links to them, as long as **Start from the sitemap too** is on.

#### How do I let the tool audit a site that blocks crawlers?

Add a group for it to the site's robots.txt: `User-agent: GapForgeSiteAudit`, then `Allow: /`. If a firewall or CDN blocks unknown crawlers, allow that user agent there too.

#### Why is a link reported as broken when it opens in my browser?

The target answered the tool with an error status, or did not answer within the time limit. A link to a host that does not answer at all, not even for its robots.txt, is listed as broken too. Some sites refuse automated requests; answers 401, 403 and 429 are therefore not counted as broken. The `reason` field in `brokenLinks` says what happened.

#### The run finished with 0 page rows in monitor mode.

No page had an issue appear or disappear since the last run with the same state key. No page was charged; the run costs only the start fee. Changes to site-level issues (for example a certificate entering its last 30 days) are still in the free site row, tagged `new` or `resolved`, and the status message then says "the site row of 1 site(s) has new or resolved issues".

#### It stopped early.

The status message says why. Most often the run reached your maximum charge; raise the limit and run again.

#### Is this an official product of a search engine or an AI company?

No. This tool is not affiliated with or endorsed by Apify, Google, OpenAI, Anthropic, Perplexity, Apple, Amazon, Meta, Common Crawl, ByteDance or any site it audits. The crawler names in the AI-crawler check are the user agents those companies publish.

#### Something looks wrong.

Open an Issue on this page with the run ID. We reply within one business day and fix confirmed bugs in the next build.

### How URLs are compared

- The fragment (`#section`) is dropped; scheme and host are lower-cased; default ports are dropped.
- `/page` and `/page/` are treated as two different addresses, as a server treats them. If one redirects to the other, they become one page row.
- The query string is kept as written, so `?page=2` is a different address from `?page=3`.
- A host with and without `www.` is the same site. Other subdomains are separate sites: add them as their own start address.

### Limits

- No JavaScript rendering: the tool reads the HTML the server sends. Content added by scripts is not seen.
- No page speed measurement, no accessibility scoring and no judgement of rich-result eligibility. `responseMs` is the time one request took from the run, not a speed score.
- At most 20 sites per run, 2,000 pages per site and 500 external links per site.
- robots.txt is followed, so disallowed pages are skipped and listed as skipped. A site that asks for more than 60 seconds between requests is not audited.
- One page is read up to 5 MB, and one sitemap file up to 20 MB after decompression. At most 50 sitemap files and 50,000 sitemap addresses are read per site; beyond that `sitemapTruncated` is `true`.
- File links (PDF, images and similar) are status-checked, at most 200 per site, and are never audited as pages.
- Duplicate titles and descriptions are compared among the pages audited in this run.
- Requests to one host are sent one at a time, so large sites take longer by design.
- A site's rows are saved when the audit of that site is finished. In our tests a 2,000-page audit took 25 to 33 minutes with **Maximum requests per second** set to 5; the default of 2 was not timed at this size. Sites are audited one after another, so keep the run timeout long enough for all sites in the run: if a run is aborted or times out in the middle of a site, the rows of that site and of the sites after it are not saved and not charged.
- Sitemap files are read before the first page, one at a time, even when **Start from the sitemap too** is off (they are needed for the sitemap checks). A site with slow or very large sitemap files can add several minutes to a run.
- `brokenLinks` lists at most 100 links per page. Titles and meta descriptions are stored up to 1,000 characters; `titleLength` and `metaDescriptionLength` give the full length. `jsonLdTypes` lists at most 50 types.

# Changelog

This Actor's version history is a separate document: https://apify.com/gapforge/site-audit/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

One address per site, for example https://example.com. The audit starts at this address and stays on its host (with or without www). Two addresses on the same host are audited once. Public sites only.

## `maxPagesPerSite` (type: `integer`):

The audit stops after this many pages on each site. You pay per page, so this number also caps what a run can cost. 25 pages is a good first try; raise it to audit a bigger site (up to 2,000). Left empty, it audits up to 100 pages.

## `useSitemap` (type: `boolean`):

On: pages listed in the site's sitemaps (from robots.txt and /sitemap.xml) are audited along with the pages found by following links. Off: only pages reached by links from the start address are audited. The sitemap itself is read and reported either way.

## `maxRequestsPerSecond` (type: `integer`):

How fast the audit may ask one host for pages. A Crawl-delay in the site's robots.txt that is slower than this setting is used instead.

## `checkExternalLinks` (type: `boolean`):

On: links that point to other sites are tested too (a light HEAD request, then GET if needed), up to the limit below, and dead ones are listed in brokenLinks. Off: links between pages of the audited site are still tested.

## `maxExternalLinksPerSite` (type: `integer`):

With "Check links to other sites" on: test at most this many different external links per site. The site row says how many were left untested.

## `checkSecurityHeaders` (type: `boolean`):

Check the http to https redirect, six security headers on the start page, and the TLS certificate (days until it expires, issuer).

## `checkAiReadiness` (type: `boolean`):

Check /llms.txt and /llms-full.txt, and report what robots.txt tells each of 12 AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) about the home page.

## `onlyChanges` (type: `boolean`):

Monitor mode. Deliver (and charge for) a page row only when the page has an issue that is new, or an issue that has gone, compared with the last run that used the same state key. The first run reports each current issue as new.

## `stateKey` (type: `string`):

Name of the memory used by monitor mode. Use different keys for different schedules, e.g. weekly-my-site. Lower-case letters and digits, with single dashes between them. The memory is a key-value store in your account named site-audit-state-KEY; delete it to start fresh.

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "maxPagesPerSite": 25,
  "useSitemap": true,
  "maxRequestsPerSecond": 2,
  "checkExternalLinks": false,
  "maxExternalLinksPerSite": 200,
  "checkSecurityHeaders": true,
  "checkAiReadiness": true,
  "onlyChanges": false,
  "stateKey": "default"
}
```

# Actor output Schema

## `overview` (type: `string`):

Each site's summary row, then its pages: pages audited, errors, warnings, llms.txt, AI crawlers blocked, missing security headers, TLS days left.

## `pages` (type: `string`):

One line per row: address, status, title, number of issues and broken links.

## `issues` (type: `string`):

One line per issue: address, code, severity, message and fix.

## `allFields` (type: `string`):

Every field of every row in the dataset, as defined by the dataset schema.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/"
    ],
    "maxPagesPerSite": 25
};

// Run the Actor and wait for it to finish
const run = await client.actor("gapforge/site-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://books.toscrape.com/"],
    "maxPagesPerSite": 25,
}

# Run the Actor and wait for it to finish
run = client.actor("gapforge/site-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "maxPagesPerSite": 25
}' |
apify call gapforge/site-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gapforge/site-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qRmlSoQuf9wsZ4gjw/builds/WOj5x50dcUH3Ogz4B/openapi.json
