SEO & AI Search Readiness Audit (robots.txt, llms.txt, schema)

Audit any web page for classic SEO and AI-search readiness in one pass. You get a 0-100 SEO score, a 0-100 AI-readiness score, and a list of issues, each with a stable code, a severity and a one-line fix.

It also checks the things AI assistants care about and most SEO tools skip:

• Which AI crawlers your robots.txt blocks: AI search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot …) and AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended …), kept apart so you can tell a policy choice from a mistake.
• llms.txt: whether /llms.txt exists and follows the llmstxt.org format.
• JavaScript-only pages: most AI crawlers don't run JavaScript. If a page's HTML is an empty app shell, you'll see JS_ONLY_CONTENT.
• Structured data: JSON-LD types found, broken JSON-LD, missing @context, missing article dates.

Plain HTTP, no browser, no API keys. It's fast and cheap, and it was built to be called by AI agents.

What you get

One item per page (type: "page"):

Field	What it is
seo_score, ai_score	0-100. Each category starts at 100 and loses 15 per high, 7 per medium and 3 per low issue
issues	[{code, category, severity, message, fix, details}], e.g. TITLE_TOO_LONG, NOINDEX, JS_ONLY_CONTENT
title, meta_description, h1, headings, word_count, lang	On-page basics
canonical, meta_robots, x_robots_tag, indexable, hreflang	Indexing signals
open_graph, twitter_card, structured_data_types, jsonld_blocks	Social previews and schema.org
images, images_missing_alt, links_internal, links_external, broken_links	Content and links
http_status, response_ms, html_bytes, redirects	Transport

One item per site (type: "site", when Site checks is on): robots_txt, ai_crawlers (each crawler's purpose and whether it may fetch the homepage), ai_search_bots_blocked, ai_training_bots_blocked, sitemap (found, valid, entry count), llms_txt (found, valid, title, sections, links, problems) and site-level issues.

A readable Markdown report is saved as REPORT in the key-value store, and a run summary as OUTPUT (averages, most common issues, error codes).

Quick start

In the Console: paste one or more URLs and click Start.

API (Python):

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("errandkit/seo-ai-readiness-audit").call(run_input={
    "urls": ["https://example.com/", "https://example.com/pricing"],
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["type"] == "site":
        print("AI search bots blocked:", item["ai_search_bots_blocked"], "llms.txt:", item["llms_txt"]["found"])
    else:
        print(item["url"], item["seo_score"], item["ai_score"], [i["code"] for i in item["issues"]])

Crawl part of a site and check links (follows internal links up to 50 pages, checks up to 100 links per page):

{
  "urls": ["https://example.com/"],
  "maxPagesPerSite": 50,
  "checkLinks": true
}

Example page item (trimmed):

{
  "type": "page",
  "url": "https://example.com/",
  "status": "ok",
  "seo_score": 70,
  "ai_score": 93,
  "title": "Example Domain",
  "word_count": 26,
  "issues": [
    {"code": "META_DESCRIPTION_MISSING", "category": "seo", "severity": "medium",
     "message": "No meta description.", "fix": "Add a meta description of about 70-160 characters."},
    {"code": "STRUCTURED_DATA_MISSING", "category": "ai", "severity": "medium",
     "message": "No JSON-LD or microdata found.", "fix": "Add JSON-LD (schema.org) describing the page: ..."}
  ]
}

Use it from an AI agent (MCP)

This Actor works as a tool in Claude, Cursor, ChatGPT connectors and other MCP clients through Apify's MCP server:

https://mcp.apify.com/?actors=errandkit/seo-ai-readiness-audit

Ask things like "Audit example.com and tell me the three most important fixes" or "Can ChatGPT search and Perplexity crawl my site?". Results are JSON with stable codes, so the agent doesn't have to guess.

Input options

Option	Default	Notes
urls	–	Pages to audit. example.com is read as https://example.com/
maxPagesPerSite	1	1 = only the URLs you give. Higher = also follow internal links on the same site, up to this many pages per site
checkSite	true	Once per site: robots.txt, AI crawler access, XML sitemap, llms.txt
checkLinks	false	Request each link on audited pages (HEAD, then GET if needed) and report broken ones
maxLinksPerPage	100	Cap for link checking (max 200)
saveReport	true	Save the Markdown REPORT

Pricing

Pay per event. Failed and skipped pages are free.

Event	Price	Per 1,000
Page audited (HTML page fetched and analyzed)	$0.004	$4.00
Site checked (robots.txt, AI crawlers, sitemap, llms.txt; once per site)	$0.005	$5.00
Link checked (only with checkLinks; each URL once per run)	$0.0005	$0.50
Actor start (Apify standard)	$0.00005	$0.05

Examples: one page + its site check costs $0.009. A 50-page crawl of one site costs $0.205; with link checking of ~2,000 unique links, add $1.00. Pages that time out, return 4xx/5xx, aren't HTML or are blocked by robots.txt are not charged. Set Maximum cost per run and the Actor stops before going over it, keeping everything already finished.

Error codes (per page)

Code	Meaning
HTTP_ERROR	The page returned 4xx or 5xx (status in http_status)
NOT_HTML	The URL is a PDF, image, JSON … not an HTML page
ROBOTS_DISALLOWED	robots.txt doesn't allow our crawler there, so it wasn't fetched (status: "skipped")
FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS	Network problems
URL_NOT_ALLOWED	Not http(s), or a private network address
MAX_COST_REACHED	Your maximum cost per run was reached

How it crawls (good-citizen rules)

• It always obeys robots.txt for its own user agent ErrandkitAuditBot (and *), including Crawl-delay up to 5 s. If robots.txt returns a server error, it crawls nothing on that site, as Google does.
• At most one request every 0.5 s per site, at most 5 sites at a time.
• It only fetches the pages you list and, if you ask, internal links on the same site. It never logs in, never submits forms and doesn't collect personal data (no emails or phone numbers).

Good to know

• No JavaScript is run. That's deliberate: it shows what search and AI crawlers that don't render JavaScript see. Core Web Vitals and Lighthouse scores aren't measured.
• Scores are a simple, transparent sum of issue weights, not a guess at rankings. Use the issues list to decide what to fix.
• Link checking requests each external link once per run (HEAD, falling back to GET for servers that refuse HEAD). 429 Too Many Requests is not counted as broken.

Privacy

Pages are fetched and analyzed inside your own Apify run. Nothing is sent to any third-party service or AI model, and nothing is kept by the developer.

Feedback

Found a false positive? Open an issue on the Actor's Issues tab with the URL.

Part of the Errandkit catalog of data tools, APIs and MCP servers for AI agents: https://errandkit.com