Indexability and Canonical Audit Crawler

Actor #197 audits the initial static HTML of explicit public HTTP(S) URLs and writes one terminal record per requested URL. For each URL it checks deterministic local signals: robots.txt enforcement, HTTP status + redirect chain, meta robots / X-Robots-Tag noindex, HTML structure, and declared canonical analysis. It then emits a local indexability verdict (INDEXABLE / NOT_INDEXABLE / INDETERMINATE) plus bounded findings with versioned rule ids.

This is a deterministic local diagnostic tool. It does not predict, rank, or guarantee Google/search-engine indexing or SEO outcomes.

What it is for

SEO teams, content agencies, and AI-search analytics vendors can use the dataset to audit obvious, locally observable indexability/canonical conditions across a list of URLs: pages that declare noindex, pages blocked by robots.txt, non-200 pages, redirect chains, missing/self/ different/conflicting/cross-domain canonicals, and scheme/host canonical variants.

Each run input is one or more { "url": ... } objects; each accepted URL is audited independently and produces exactly one dataset item (result or error). Input order is preserved; duplicate entries are each processed independently.

Input

startUrls is required and must contain at least one entry: an array of objects, each with a single string url.

{
  "startUrls": [
    { "url": "https://example.com/page-a" },
    { "url": "https://example.org/page-b" }
  ]
}

The request-list schema default, prefill, and sample_input.json use { "url": "https://example.com/" }, the same object shape required for every entry. URL strings are rejected. v1 is object-only and single-URL-per-entry: extra per-entry keys (title, method, headers, depth, …) are rejected as input errors — the implementation only honors a url key. There is no crawl tree, no sitemap parsing, no proxy/auth/cookie/JS-render configuration.

Accepted URL shape

• Absolute URL with scheme http: or https: only.
• No URL credentials.
• Non-empty host; literal IP hosts are allowed but must resolve to a public address.
• Optional query string and fragment (fragment is stripped for fetch and comparison).

Rejected inputs (deterministic error record, no page fetch)

Bare URL strings, missing url, extra per-entry keys, non-HTTP(S) schemes, malformed URLs, and empty URLs each produce one indexability-and-canonical-audit-crawler-error record (kind: input_rejected / input_invalid_url).

Output

The default dataset contains one terminal record per input entry. Two record types:

Record type	Meaning
indexability-and-canonical-audit-crawler-result	Accepted URL processed to a terminal result (including robots-blocked and non-200 outcomes).
indexability-and-canonical-audit-crawler-error	Input rejection or fetch/parse failure (e.g. SSRF veto, redirect cap, oversized body, transport/5xx-after-retry).

Result record core fields

Field	Meaning
inputUrl, finalUrl	Original supplied URL and final post-redirect URL.
requestId	v1 always "auto".
http	Final-response evidence: statusCode, statusText, finalUrl, redirectChain[], contentType, bounded headers, bytesRead.
robots	robots.txt evidence for the origin: url, fetchStatus, allowed, matchedRule, parseNote.
indexability	verdict (INDEXABLE / NOT_INDEXABLE / INDETERMINATE), reasons[] (rule ids + keys), metaRobots, xRobotsTag, effectiveRule.
canonical	Declared-canonical audit: state (SELF / DIFFERENT / CROSS_DOMAIN / MISSING / CONFLICT / INVALID), fromHtml, fromHeader, declaredTargets[].
findings[]	Deterministic findings (v1-index-* / v1-canon-*) with severity, message, and evidence; capped at 20 per page (findingsTruncated).
auditedAt, contractVersion, resultType	Timestamp, 1.0.0, record type.

Error record core fields

inputUrl, requestId, error (kind, code, message, ruleId, detail), contractVersion, resultType.

A 404/410 page is a result with verdict: NOT_INDEXABLE (the fetch succeeded; the status is durable page evidence). A 500-after-retry or a transport failure is an error record (the fetch never produced a durable page signal).

Example result

{
  "inputUrl": "https://example.com/a",
  "finalUrl": "https://example.com/a",
  "requestId": "auto",
  "http": {
    "statusCode": 200,
    "statusText": "OK",
    "finalUrl": "https://example.com/a",
    "redirectChain": [],
    "contentType": "text/html; charset=utf-8",
    "headers": { "content-type": ["text/html; charset=utf-8"] },
    "bytesRead": 512
  },
  "robots": { "url": "https://example.com/robots.txt", "fetchStatus": 200, "allowed": true, "matchedRule": null, "parseNote": null },
  "indexability": { "verdict": "INDEXABLE", "reasons": [], "metaRobots": { "present": false, "rawValues": [], "unrecognized": [] }, "xRobotsTag": { "present": false, "rawValues": [], "userAgentScoped": [] }, "effectiveRule": "indexable" },
  "canonical": { "state": "MISSING", "fromHtml": { "present": false, "rawHref": null, "resolvedHref": null, "ignoredAttrs": [] }, "fromHeader": { "present": false, "rawHref": null }, "declaredTargets": [] },
  "findings": [ { "findingId": "v1-canon-001", "severity": "info", "message": "Canonical is missing (no HTML canonical, no Link header).", "evidence": {} } ],
  "findingsTruncated": false,
  "auditedAt": "2026-09-05T00:00:00.000Z",
  "contractVersion": "1.0.0",
  "resultType": "indexability-and-canonical-audit-crawler-result"
}

(Field values are illustrative; exact findings messages and header subsets come from the actual implementation. Fetch it live and inspect the real record.)

Example error

{
  "inputUrl": "https://[::1]/page",
  "requestId": "auto",
  "error": { "kind": "fetch_ssrf_blocked", "code": "ERR_FETCH_SSRF_BLOCKED", "message": "URL resolves to a non-public address.", "ruleId": null, "detail": {} },
  "contractVersion": "1.0.0",
  "resultType": "indexability-and-canonical-audit-crawler-error"
}

Deterministic local rules

Findings use versioned, deterministic rule ids documented in the frozen v1 contract (.actor schemas and the source). Indexability verdict resolution order: v1-index-001 (HTTP status) → v1-index-002 (robots.txt gate) → v1-index-003 (non-HTML content type) → v1-index-004/005 (meta / X-Robots-Tag noindex) → v1-index-007 (unparseable body) → v1-index-008 (canonical self/declared note).

User-agent scoping: a noindex directive applies to the actor's crawler token only when it is unqualified (noindex, none) or explicitly scoped to that token (actor-197-indexability-crawler: noindex) or to *. Directives scoped to a different product token — X-Robots-Tag: googlebot: noindex, otherbot: nofollow, or <meta name="googlebot" content="noindex"> — are recorded in the evidence (indexability.xRobotsTag.rawValues / userAgentScoped, indexability.metaRobots.rawValues) but do NOT flip the actor's verdict (<meta name="robots"> remains global).

Canonical findings: v1-canon-001 (missing) … v1-canon-014 (meta-refresh present), covering self/different/cross-domain/relative/fragment/conflict/redirect-to-noncanonical/canonical-on- non-indexable/scheme-host-variant/ignored-attrs/invalid/unverifiable-target conditions.

robots.txt is enforced per origin for the crawler token (User-Agent: actor-197-indexability-crawler/1.0 …). robots-blocked URLs are NOT fetched and produce a NOT_INDEXABLE result (robots.allowed: false). robots.txt 404 = no restrictions; robots.txt 429/5xx/unreachable after retries = deterministic unavailable policy.

Limits (v1)

• Static HTML only. No browser rendering, no JavaScript execution, no remote validator.
• Single URL per input entry. No crawl tree / depth / same-site expansion.
• Response body capped at 1 MiB; robots.txt parse capped at 500 KiB (truncated + flagged).
• At most 5 redirect hops, each re-vetted for SSRF safety.
• Retry only transport errors + HTTP 408/429/5xx, max 2 retries, bounded exponential backoff (1s/2s + jitter). Never retry other 4xx. SSRF veto is never retried.
• SSRF-safe public-address fetch: pinned sockets, agent: false, per-hop redirect re-vetting. Loopback/private/link-local/cloud-metadata/documentation/NAT64 ranges are blocked.
• Findings capped at 20 per page (findingsTruncated: true beyond).
• Canonical targets are recorded as evidence but never fetched in v1.

Deliberate v1 non-goals

• No SEO ranking / position / Google-indexing guarantees. Findings are deterministic local diagnostics, never a promise that a search engine will or will not index a URL.
• No content duplication similarity scoring, soft-404 detection, hreflang cluster validation, AMP/pagination handling, or JS-rendered signals.
• No proxy lists, auth/cookies/custom headers, output-format selection, or billing events.
• No Cloud deployment / Store publication / billing activation is performed by this repository.

Local development

npm ci
npm test
npm run run-local     # deterministic fixture runner (no network) -> local-out/
node --check src/*.js test/*.test.js
npx --yes apify-cli@latest validate-schema < /dev/null

src/main.js uses the normal Apify input/dataset interfaces: the platform injects INPUT in the default Key-Value Store and records are pushed to the default dataset. npm start is therefore the Apify platform entrypoint (or a local run that initializes those standard storages). The deterministic unit suite exercises the contract with an injectable transport and never touches the network.

The Docker image is pinned to apify/actor-node:22, installs production dependencies with npm ci --omit=dev, and starts src/main.js through npm start.

Store-positioning recommendation (not activated)

The roadmap records a future primary result event named indexability-and-canonical-audit-crawler-result and launch guidance of $1.50 per 1,000 raw results. This source does not emit a billing event and no current pricing configuration was verified or activated. Do not make a pay-per-result Store claim until a separately reviewed event adapter, current Apify pricing verification, an exact-default Cloud run, and a non-empty dataset/output check are complete.