AI Crawlability Preflight

A small deterministic Apify Actor that checks obvious crawl/indexability blockers, canonical signals, and JSON-LD parsing state for 1–100 public HTTP/HTTPS page URLs.

It is designed as a machine-readable preflight utility for workflows. It does not call an LLM or any paid/external API.

What it returns

One default-dataset row per completed URL preflight, including:

• input URL, final URL, redirect chain, final HTTP status, fetch state, Content-Type, and bounded-body state;
• wildcard (User-agent: *) robots.txt path allowance plus robots fetch evidence;
• page-level <meta name="robots"> directives and noindex detection;
• canonical URL plus self|different|missing|invalid status;
• JSON-LD block count, parseable block count, extracted @type values, and stable parse-error details;
• deterministic PASS|WARN|BLOCK classification with stable reason codes.

Observed facts are stored separately from the derived preflight classification so downstream consumers can use the raw evidence directly.

Input

{
  "urls": [
    "https://example.com/",
    "https://example.com/product"
  ],
  "reportFormat": "none"
}

• urls — required array, 1–100 absolute public http/https URLs.
• reportFormat — none or markdown; default none.

When reportFormat is markdown, the Actor writes a deterministic REPORT.md to the run's default Key-Value Store. It contains counts and reason-code tables only; it contains no LLM recommendations.

Output example

{
  "url": "https://example.com/product",
  "finalUrl": "https://example.com/product",
  "httpStatus": 200,
  "fetchStatus": "ok",
  "contentType": "text/html; charset=utf-8",
  "redirectChain": ["https://example.com/product"],
  "bodyTruncated": false,
  "fetchError": null,
  "robots": {
    "robotsUrl": "https://example.com/robots.txt",
    "robotsHttpStatus": 200,
    "robotsFetchStatus": "ok",
    "robotsTxtFound": true,
    "wildcardAllowed": true,
    "robotsFetchError": null,
    "metaRobots": ["follow", "index"],
    "noindex": false,
    "htmlAnalyzed": true
  },
  "canonical": {
    "url": "https://example.com/product",
    "raw": "https://example.com/product",
    "status": "self",
    "count": 1,
    "analyzed": true
  },
  "jsonLd": {
    "blockCount": 1,
    "parseableBlockCount": 1,
    "types": ["Product"],
    "parseErrors": [],
    "missing": false,
    "analyzed": true
  },
  "preflight": {
    "status": "PASS",
    "blockers": [],
    "warnings": []
  }
}

Classification and reason codes

BLOCK is produced if any blocker exists. Otherwise WARN is produced if any warning exists. Otherwise the row is PASS.

Blockers:

• FETCH_ERROR
• HTTP_NON_2XX
• ROBOTS_DISALLOW
• META_NOINDEX

Warnings:

• CANONICAL_DIFFERENT
• CANONICAL_MISSING
• CANONICAL_INVALID
• JSONLD_MISSING
• JSONLD_PARSE_ERROR
• ROBOTS_FETCH_ERROR
• NON_HTML_CONTENT
• BODY_TRUNCATED

The work-order-required MVP reason codes are therefore all covered: HTTP_NON_2XX, ROBOTS_DISALLOW, META_NOINDEX, CANONICAL_DIFFERENT, CANONICAL_MISSING, JSONLD_MISSING, JSONLD_PARSE_ERROR, and FETCH_ERROR.

Fetch-state semantics

• ok — an HTTP response was obtained (including non-2xx responses other than the explicit blocked statuses below).
• blocked — the final page response was HTTP 401, 403, or 429.
• error — no final HTTP response was obtained because of a transport, DNS, URL-safety, timeout, or redirect failure.

HTTP_NON_2XX remains the deterministic blocker for all final non-2xx HTTP responses.

Public-target safety

Before each page request and each redirect target, the Actor:

• requires absolute HTTP/HTTPS;
• rejects credential-bearing URLs;
• resolves the hostname and rejects any non-global/private/loopback/link-local target.

Every page, redirect, and robots.txt fetch is also enforced at TCP connection time. The custom HTTPX/httpcore backend resolves immediately before opening the socket, rejects the entire DNS answer set if any address is non-global, and connects to the exact validated public IP. The original hostname remains the HTTP origin and TLS SNI/certificate-validation name, closing the prior DNS-validation/connection TOCTOU gap without weakening TLS validation.

Deliberate non-claims

This Actor does not:

• measure visibility, citations, rankings, or share of voice in ChatGPT, Gemini, Perplexity, or any other AI engine;
• produce a GEO/SEO score or predict citation probability;
• provide consulting, manual audits, content writing, or recommendations;
• validate Google rich-result eligibility or the semantic correctness of every Schema.org property;
• crawl sitemaps or discover site-wide URLs;
• execute JavaScript or render pages in a browser;
• authenticate to private pages or use external proxy/API services in this MVP.

It reports deterministic evidence from direct HTTP responses only.

Pay per Event

This Actor uses the custom paid event page-preflight at USD 0.005 for each charge-eligible completed page-preflight result.

Charge eligibility is deterministic:

• fetched HTTP results that produce a bounded page-preflight row emit exactly one page-preflight;
• BODY_TRUNCATED remains chargeable because the bounded fetched result is delivered;
• URL-safety, DNS, connection, redirect, network, and timeout failures are still returned as diagnostic evidence but do not emit the custom page-preflight event.

Apify manages the Actor Start event separately. The Actor code does not manually charge that event. Current Store pricing should be treated as authoritative for the final amount shown before a run.