AI Readiness Auditor

Audit public website pages for AI discoverability, crawlability, structured data, metadata, content quality, accessibility signals, and technical delivery. The Actor gives growth, SEO, content, and engineering teams a prioritized, machine-readable readiness report for every successfully audited page.

What the Actor checks

The auditor uses direct HTTP retrieval first and performs deterministic checks—no API key or LLM subscription is required. Each page receives a score from 0 to 100, a letter grade, seven category scores, and stable-code findings with evidence and recommended fixes.

Output	What it means
overallScore, grade	Weighted AI readiness score and A–F grade
categoryScores	Crawlability, AI discoverability, metadata, structured data, content, accessibility, and performance
robotsAllowed, indexable	Generic robots.txt access and robots-meta indexability
aiBotAccess	Explicit root policy for GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, and PerplexityBot
llmsTxtStatus, sitemapStatus	Site-wide discovery-file availability
structuredDataTypes	schema.org @type values parsed from valid JSON-LD
findings	Prioritized critical, high, medium, low, and informational actions
discoveredUrls	Bounded same-origin links available for crawl continuation

The OUTPUT key-value-store record contains the run outcome, delivered and charged counts, warnings, charge-cap status, and a concise summary of robots.txt, sitemap, llms.txt, and named AI-crawler policy for every origin.

When to use it

Use this Actor to establish a website baseline before an AI visibility program, check whether important landing pages expose clear machine-readable entities, generate an implementation backlog, compare templates across a site, or monitor readiness after a redesign.

Do not use it as proof that a specific AI model will cite or rank a page. The score measures observable technical and content signals, not proprietary model behavior. It is also not a browser-rendering, accessibility-conformance, Core Web Vitals, security, or legal-compliance test. JavaScript-only content may appear thin because the Actor intentionally audits the server-delivered HTML that crawlers can retrieve cheaply and consistently.

Input

Field	Default	Description
startUrls	Apify homepage	One or more public HTTP/HTTPS pages; private, local, reserved-IP, and authenticated URLs are rejected
maxPages	5	Maximum billable page reports across all origins, from 1 to 100
maxDepth	1	Same-origin link depth from 0 to 3
respectRobotsTxt	true	Skip generically disallowed URLs without producing billable page records
includeSitemapUrls	true	Fill the bounded queue with same-origin URLs found in XML sitemaps
requestTimeoutSecs	20	Per-request timeout for pages and discovery files
proxyConfiguration	direct	Optional Apify or custom proxy settings for permitted public access

Minimal one-page audit:

{
  "startUrls": [{ "url": "https://example.com/" }],
  "maxPages": 1,
  "maxDepth": 0,
  "includeSitemapUrls": false
}

Bounded site sample:

{
  "startUrls": [{ "url": "https://example.com/" }],
  "maxPages": 25,
  "maxDepth": 2,
  "respectRobotsTxt": true,
  "includeSitemapUrls": true,
  "requestTimeoutSecs": 20,
  "proxyConfiguration": { "useApifyProxy": false }
}

API clients can start the Actor with the same JSON input, then read page records from the default dataset and the aggregate diagnostic from OUTPUT in the default key-value store. The output schema exposes direct links to both resources on the run page.

Example result

{
  "url": "https://example.com/guide",
  "finalUrl": "https://example.com/guide",
  "outcome": "data",
  "statusCode": 200,
  "overallScore": 82,
  "grade": "B",
  "categoryScores": {
    "crawlability": 100,
    "aiDiscoverability": 80,
    "metadata": 90,
    "structuredData": 80,
    "content": 75,
    "accessibility": 85,
    "performance": 90
  },
  "llmsTxtStatus": "present",
  "sitemapStatus": "present",
  "structuredDataTypes": ["Article", "BreadcrumbList"],
  "findings": [
    {
      "code": "FRESHNESS_SIGNAL_MISSING",
      "category": "content",
      "severity": "info",
      "message": "No machine-readable published or modified date was detected.",
      "recommendation": "Add time[datetime] or article date metadata when the content is time-sensitive.",
      "evidence": null
    }
  ]
}

Pricing

The Actor uses transparent pay per event pricing. Platform usage is intended to be included rather than passed through separately.

Event	Price
Audit run start	$0.0001 per run
AI readiness page audit	$0.004 per delivered page report

A one-page run costs about $0.0041 in Actor events. A successful 10-page audit costs about $0.0401. Blocked, robots-disallowed, unreachable, non-HTML, and rejected pages do not create dataset records and therefore do not trigger page-audit events. Apify's maximum-charge setting is checked before page work, and the Actor stops cleanly when no further page event fits the budget.

Reliability and limitations

The Actor retrieves server-delivered HTML plus /robots.txt, declared or conventional XML sitemaps, and /llms.txt. Requests use realistic headers, safe redirect validation, bounded retries for rate limits and transient server errors, a 5 MB response ceiling, and a per-request timeout. Public redirects are revalidated to prevent access to local or private infrastructure. One failed site does not discard valid reports from other URLs; diagnostics remain in OUTPUT.

Scoring is intentionally transparent and deterministic, but it is opinionated. Missing llms.txt is a low-severity opportunity, not a guarantee of poor AI visibility. Named crawler rules are reported as allowed, blocked, or unspecified without asserting that every vendor still uses the same crawler identity. Response time is origin-observation latency, not a field measurement. The Actor does not solve CAPTCHAs, bypass authentication or paywalls, or defeat access controls.

Responsible use

Audit only websites you are authorized to assess and follow their terms, robots policy, and applicable law. Page content and metadata can contain personal or sensitive information; configure Apify dataset retention and access controls appropriately. The Actor does not publish, republish, or train on retrieved content—it extracts a limited set of technical signals and short evidence values for the requested audit.

For support, include the run ID, a non-sensitive example URL, and the unexpected finding code. Never include passwords, tokens, cookies, or private proxy URLs in an issue.