# Indexability and Canonical Audit Crawler (`kayhermes/indexability-and-canonical-audit-crawler`) Actor

Deterministic local indexability + canonical audit of explicit public HTTP(S) URLs from static HTML: robots.txt enforcement, meta/X-Robots-Tag noindex, HTTP status, redirect chains, and declared canonical analysis.

- **URL**: https://apify.com/kayhermes/indexability-and-canonical-audit-crawler.md
- **Developed by:** [Khoa Nguyen](https://apify.com/kayhermes) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 completed audit results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Indexability and Canonical Audit Crawler

Actor #197 audits the **initial static HTML** of explicit public HTTP(S) URLs and writes one
**terminal record per requested URL**. For each URL it checks deterministic local signals:
robots.txt enforcement, HTTP status + redirect chain, meta robots / X-Robots-Tag `noindex`,
HTML structure, and declared canonical analysis. It then emits a local indexability verdict
(`INDEXABLE` / `NOT_INDEXABLE` / `INDETERMINATE`) plus bounded findings with versioned rule ids.

This is a **deterministic local diagnostic tool**. It does **not** predict, rank, or guarantee
Google/search-engine indexing or SEO outcomes.

### What it is for

SEO teams, content agencies, and AI-search analytics vendors can use the dataset to audit
obvious, locally observable indexability/canonical conditions across a list of URLs: pages that
declare `noindex`, pages blocked by robots.txt, non-200 pages, redirect chains, missing/self/
different/conflicting/cross-domain canonicals, and scheme/host canonical variants.

Each run input is **one or more `{ "url": ... }` objects**; each accepted URL is audited
independently and produces exactly one dataset item (result or error). Input order is preserved;
duplicate entries are each processed independently.

### Input

`startUrls` is required and must contain at least one entry: an array of objects, each with a
single string `url`.

```json
{
  "startUrls": [
    { "url": "https://example.com/page-a" },
    { "url": "https://example.org/page-b" }
  ]
}
```

The request-list schema default, prefill, and `sample_input.json` use
`{ "url": "https://example.com/" }`, the same object shape required for every entry. URL strings
are rejected. **v1 is object-only and single-URL-per-entry**: extra per-entry keys (`title`,
`method`, `headers`, `depth`, …) are rejected as input errors — the implementation only honors a
`url` key. There is no crawl tree, no sitemap parsing, no proxy/auth/cookie/JS-render
configuration.

#### Accepted URL shape

- Absolute URL with scheme `http:` or `https:` only.
- No URL credentials.
- Non-empty host; literal IP hosts are allowed but must resolve to a **public** address.
- Optional query string and fragment (fragment is stripped for fetch and comparison).

#### Rejected inputs (deterministic error record, no page fetch)

Bare URL strings, missing `url`, extra per-entry keys, non-HTTP(S) schemes, malformed URLs, and
empty URLs each produce one `indexability-and-canonical-audit-crawler-error` record
(`kind: input_rejected` / `input_invalid_url`).

### Output

The default dataset contains one terminal record per input entry. Two record types:

| Record type | Meaning |
|---|---|
| `indexability-and-canonical-audit-crawler-result` | Accepted URL processed to a terminal result (including robots-blocked and non-200 outcomes). |
| `indexability-and-canonical-audit-crawler-error` | Input rejection or fetch/parse failure (e.g. SSRF veto, redirect cap, oversized body, transport/5xx-after-retry). |

#### Result record core fields

| Field | Meaning |
|---|---|
| `inputUrl`, `finalUrl` | Original supplied URL and final post-redirect URL. |
| `requestId` | v1 always `"auto"`. |
| `http` | Final-response evidence: `statusCode`, `statusText`, `finalUrl`, `redirectChain[]`, `contentType`, bounded `headers`, `bytesRead`. |
| `robots` | robots.txt evidence for the origin: `url`, `fetchStatus`, `allowed`, `matchedRule`, `parseNote`. |
| `indexability` | `verdict` (`INDEXABLE` / `NOT_INDEXABLE` / `INDETERMINATE`), `reasons[]` (rule ids + keys), `metaRobots`, `xRobotsTag`, `effectiveRule`. |
| `canonical` | Declared-canonical audit: `state` (`SELF` / `DIFFERENT` / `CROSS_DOMAIN` / `MISSING` / `CONFLICT` / `INVALID`), `fromHtml`, `fromHeader`, `declaredTargets[]`. |
| `findings[]` | Deterministic findings (`v1-index-*` / `v1-canon-*`) with severity, message, and evidence; capped at 20 per page (`findingsTruncated`). |
| `auditedAt`, `contractVersion`, `resultType` | Timestamp, `1.0.0`, record type. |

#### Error record core fields

`inputUrl`, `requestId`, `error` (`kind`, `code`, `message`, `ruleId`, `detail`),
`contractVersion`, `resultType`.

A 404/410 page is a **result** with `verdict: NOT_INDEXABLE` (the fetch succeeded; the status is
durable page evidence). A 500-after-retry or a transport failure is an **error record** (the
fetch never produced a durable page signal).

#### Example result

```json
{
  "inputUrl": "https://example.com/a",
  "finalUrl": "https://example.com/a",
  "requestId": "auto",
  "http": {
    "statusCode": 200,
    "statusText": "OK",
    "finalUrl": "https://example.com/a",
    "redirectChain": [],
    "contentType": "text/html; charset=utf-8",
    "headers": { "content-type": ["text/html; charset=utf-8"] },
    "bytesRead": 512
  },
  "robots": { "url": "https://example.com/robots.txt", "fetchStatus": 200, "allowed": true, "matchedRule": null, "parseNote": null },
  "indexability": { "verdict": "INDEXABLE", "reasons": [], "metaRobots": { "present": false, "rawValues": [], "unrecognized": [] }, "xRobotsTag": { "present": false, "rawValues": [], "userAgentScoped": [] }, "effectiveRule": "indexable" },
  "canonical": { "state": "MISSING", "fromHtml": { "present": false, "rawHref": null, "resolvedHref": null, "ignoredAttrs": [] }, "fromHeader": { "present": false, "rawHref": null }, "declaredTargets": [] },
  "findings": [ { "findingId": "v1-canon-001", "severity": "info", "message": "Canonical is missing (no HTML canonical, no Link header).", "evidence": {} } ],
  "findingsTruncated": false,
  "auditedAt": "2026-09-05T00:00:00.000Z",
  "contractVersion": "1.0.0",
  "resultType": "indexability-and-canonical-audit-crawler-result"
}
```

(Field values are illustrative; exact `findings` messages and header subsets come from the actual
implementation. Fetch it live and inspect the real record.)

#### Example error

```json
{
  "inputUrl": "https://[::1]/page",
  "requestId": "auto",
  "error": { "kind": "fetch_ssrf_blocked", "code": "ERR_FETCH_SSRF_BLOCKED", "message": "URL resolves to a non-public address.", "ruleId": null, "detail": {} },
  "contractVersion": "1.0.0",
  "resultType": "indexability-and-canonical-audit-crawler-error"
}
```

### Deterministic local rules

Findings use versioned, deterministic rule ids documented in the frozen v1 contract
(`.actor` schemas and the source). Indexability verdict resolution order:
`v1-index-001` (HTTP status) → `v1-index-002` (robots.txt gate) → `v1-index-003`
(non-HTML content type) → `v1-index-004/005` (meta / X-Robots-Tag noindex) →
`v1-index-007` (unparseable body) → `v1-index-008` (canonical self/declared note).

User-agent scoping: a `noindex` directive applies to the actor's crawler token only when it is
unqualified (`noindex`, `none`) or explicitly scoped to that token (`actor-197-indexability-crawler: noindex`)
or to `*`. Directives scoped to a different product token — `X-Robots-Tag: googlebot: noindex`,
`otherbot: nofollow`, or `<meta name="googlebot" content="noindex">` — are recorded in the
evidence (`indexability.xRobotsTag.rawValues` / `userAgentScoped`, `indexability.metaRobots.rawValues`)
but do NOT flip the actor's verdict (`<meta name="robots">` remains global).

Canonical findings: `v1-canon-001` (missing) … `v1-canon-014` (meta-refresh present), covering
self/different/cross-domain/relative/fragment/conflict/redirect-to-noncanonical/canonical-on-
non-indexable/scheme-host-variant/ignored-attrs/invalid/unverifiable-target conditions.

robots.txt is enforced per origin for the crawler token
(`User-Agent: actor-197-indexability-crawler/1.0 …`). robots-blocked URLs are NOT fetched and
produce a `NOT_INDEXABLE` result (`robots.allowed: false`). robots.txt 404 = no restrictions;
robots.txt 429/5xx/unreachable after retries = deterministic `unavailable` policy.

### Limits (v1)

- Static HTML only. No browser rendering, no JavaScript execution, no remote validator.
- Single URL per input entry. No crawl tree / depth / same-site expansion.
- Response body capped at **1 MiB**; robots.txt parse capped at **500 KiB** (truncated + flagged).
- At most **5 redirect hops**, each re-vetted for SSRF safety.
- Retry only transport errors + HTTP 408/429/5xx, max 2 retries, bounded exponential backoff
  (1s/2s + jitter). Never retry other 4xx. SSRF veto is never retried.
- SSRF-safe public-address fetch: pinned sockets, `agent: false`, per-hop redirect re-vetting.
  Loopback/private/link-local/cloud-metadata/documentation/NAT64 ranges are blocked.
- Findings capped at **20 per page** (`findingsTruncated: true` beyond).
- Canonical targets are recorded as evidence but **never fetched** in v1.

### Deliberate v1 non-goals

- **No SEO ranking / position / Google-indexing guarantees.** Findings are deterministic local
  diagnostics, never a promise that a search engine will or will not index a URL.
- No content duplication similarity scoring, soft-404 detection, hreflang cluster validation,
  AMP/pagination handling, or JS-rendered signals.
- No proxy lists, auth/cookies/custom headers, output-format selection, or billing events.
- No Cloud deployment / Store publication / billing activation is performed by this repository.

### Local development

```sh
npm ci
npm test
npm run run-local     # deterministic fixture runner (no network) -> local-out/
node --check src/*.js test/*.test.js
npx --yes apify-cli@latest validate-schema < /dev/null
```

`src/main.js` uses the normal Apify input/dataset interfaces: the platform injects `INPUT` in the
default Key-Value Store and records are pushed to the default dataset. `npm start` is therefore
the Apify platform entrypoint (or a local run that initializes those standard storages). The
deterministic unit suite exercises the contract with an injectable transport and **never touches
the network**.

The Docker image is pinned to `apify/actor-node:22`, installs production dependencies with
`npm ci --omit=dev`, and starts `src/main.js` through `npm start`.

### Store-positioning recommendation (not activated)

The roadmap records a future primary result event named
`indexability-and-canonical-audit-crawler-result` and launch guidance of $1.50 per 1,000 raw
results. This source does **not** emit a billing event and no current pricing configuration was
verified or activated. Do not make a pay-per-result Store claim until a separately reviewed event
adapter, current Apify pricing verification, an exact-default Cloud run, and a non-empty
dataset/output check are complete.

# Actor input Schema

## `startUrls` (type: `array`):

One or more ordered public HTTP(S) page URL request objects, each containing only a url string. v1 audits each URL independently (single-URL scope; no crawl tree). Extra per-entry keys (title, method, headers, etc.) are rejected as input errors.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

Normalized per-input terminal records: indexability verdict + canonical audit + bounded findings, or a stable error.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("kayhermes/indexability-and-canonical-audit-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("kayhermes/indexability-and-canonical-audit-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}' |
apify call kayhermes/indexability-and-canonical-audit-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kayhermes/indexability-and-canonical-audit-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ZhSb3CVgfLyYm289L/builds/pAWfIfB1h9gd8GwyN/openapi.json