AI Readiness Checker for Websites

Audit public website pages for AI crawler access, machine-readable structure, answer-ready content, trust signals, and technical discoverability. The Actor returns one transparent page report per dataset item, with a 0–100 score, letter grade, category breakdown, and prioritized recommendations.

The checker uses direct public HTML and site-owned files. It does not send content to an LLM, invent rankings, claim that a page will be cited, or require an external AI API key.

What the AI readiness checker audits

Area	Signals checked
AI crawler access	robots.txt policies for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended
Discovery	Page noindex, sitemap presence, canonical URLs, and llms.txt presence
Machine readability	JSON-LD types, content-specific schema, organization schema, semantic main or article markup, and page language
Answer-ready content	Titles, descriptions, H1 usage, content depth, heading and paragraph structure, question-style headings, lists, and tables
Trust and citation	Author and date signals, links to About/Contact/Privacy/Terms/editorial pages, and external supporting links
Technical delivery	HTTP status, content type, initial HTML response time, redirects, and indexability

Each result includes these useful fields:

• score and grade: the overall 0–100 assessment and A–F grade.
• scoreBreakdown: access (25), machine readability (20), content (25), trust (15), and technical delivery (15).
• aiBotAccess: the effective root-path robots policy for six named AI crawlers.
• structuredDataTypes: the unique schema.org @type values found in JSON-LD.
• issues: stable issue codes, severity, plain-language evidence, remediation, and score impact.
• Content and link counts for comparison across a site or over time.

When to use it

Use this Actor for a fast baseline before an AEO/GEO program, technical content audit, site migration, structured-data project, or AI visibility investigation. Agencies can export the dataset for client reports. Product, content, and SEO teams can schedule the Actor and compare scores over time. AI agents can use the stable output fields to route remediation tasks.

This Actor is not a substitute for analytics, server-log analysis, Search Console, a live AI citation monitor, JavaScript rendering, accessibility testing, or legal advice. It audits public initial HTML. Content rendered only after client-side JavaScript may appear missing. A high score indicates that the checked signals are present; it does not guarantee indexing, retrieval, ranking, or citation by any AI system.

Input

Field	Default	Description
startUrls	https://example.com/	One or more public HTTP(S) pages or sites to audit
maxPages	5	Hard cap from 1–50 delivered page audits and result events
crawlLinks	true	Follow same-site HTML links until the page cap
respectRobotsTxt	true	Skip discovered links disallowed for the wildcard crawler; starting URLs are still audited
maxConcurrency	3	Simultaneous page requests from 1–10
requestTimeoutSecs	30	Per-request timeout from 5–60 seconds
proxyConfiguration	direct connection	Optional Apify or authorized proxy configuration

Default input:

{
  "startUrls": [{ "url": "https://example.com/" }],
  "maxPages": 5,
  "crawlLinks": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 30
}

For a single-page audit, set crawlLinks to false and maxPages to 1. For a small site sample, keep discovery enabled and raise maxPages. The Actor deduplicates normalized URLs and only follows HTTP(S) links on the same hostname.

API example

curl -X POST "https://api.apify.com/v2/acts/YOUR_ACTOR_ID/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://www.example.com/guide"}],
    "maxPages": 10,
    "crawlLinks": true
  }'

Do not put real tokens in shared scripts, logs, or public repositories. Prefer an authorization header or an environment secret in production.

Output

The default dataset contains one homogeneous record per successfully audited page. Diagnostics and aggregates live in the OUTPUT and SUMMARY key-value records, so warnings never pollute the page dataset.

Compact result example:

{
  "finalUrl": "https://example.com/guide",
  "indexable": true,
  "aiCrawlable": true,
  "robotsTxtFound": true,
  "sitemapFound": true,
  "llmsTxtFound": false,
  "aiBotAccess": {
    "gptBot": "allowed",
    "oaiSearchBot": "allowed",
    "chatGptUser": "allowed",
    "claudeBot": "allowed",
    "perplexityBot": "allowed",
    "googleExtended": "allowed"
  },
  "structuredDataTypes": ["Article", "Organization"],
  "score": 86,
  "grade": "B",
  "scoreBreakdown": {
    "access": 25,
    "machineReadability": 18,
    "content": 23,
    "trust": 12,
    "technical": 8
  },
  "issues": [
    {
      "code": "LLMS_TXT_MISSING",
      "category": "machineReadability",
      "severity": "info",
      "message": "No /llms.txt file was found.",
      "recommendation": "Consider publishing /llms.txt with a concise site summary and links to canonical documentation.",
      "scoreImpact": 2
    }
  ]
}

OUTPUT adds the aggregate averageScore, overall grade, repeated topIssues, delivered record count, failed request count, robots skips, warnings, and estimated event charges. Invalid input produces a truthful REJECTED summary and zero dataset items. If all page requests fail, the run fails without fabricating results.

Pay-per-event pricing

Event	Price	Trigger
Actor start	$0.00005	One automatic start event for up to 1 GB memory
Page readiness audit	$0.006	One automatic event for each page record written to the default dataset

A five-page run produces five page-audit events, estimated at $0.030, plus the start event. A one-page run produces one page-audit event, estimated at $0.006, plus the start event. Platform usage is intended to be included in event prices. The Actor never manually charges either synthetic event, and it does not charge a result event for a failed, skipped, or rejected page.

Your Apify maximum-charge setting remains the final spending control. The Actor also caps work with maxPages, bounds retries, limits concurrency, and reports the number of delivered dataset items in its terminal status and summary.

Scoring and interpretation

The score is a deterministic checklist, not an opaque model judgment. Issues deduct from their category only, and category scores never fall below zero. Grades are A (90–100), B (80–89), C (70–79), D (60–69), and F (below 60).

Treat recommendations as a prioritized engineering backlog. Some signals are contextual: a deliberate crawler block is not inherently wrong, an author is not required on every transactional page, and llms.txt is an emerging convention rather than a universal requirement. The Actor therefore labels contextual findings as info and applies smaller deductions than hard blockers such as noindex, HTTP errors, or blocking every checked AI crawler.

Reliability and limits

The Actor starts with low-cost direct HTML requests, retries transient page failures twice, honors a per-request timeout, and can use an authorized proxy when supplied. It does not defeat CAPTCHAs, authentication, paywalls, or other access controls. A site that serves challenge HTML or requires browser rendering may receive an incomplete audit; inspect the response fields and warnings before acting on the score.

robots.txt, /sitemap.xml, and /llms.txt are checked per origin and cached during the run. The crawler follows only same-host links, filters common document and media extensions, and respects wildcard robots rules for discovered links by default. Starting URLs are fetched because examining the requested public page is the core audit operation.

Use the results only for sites you are permitted to access. Keep crawl limits conservative, respect site policies, avoid personal or sensitive pages, and independently review changes before deployment.