SEO Audit & AI Search Readiness Checker

A technical SEO audit of every page, plus the new question: can ChatGPT, Claude, Perplexity and other AI search engines see this site? Give it one site or a thousand. For each site you get one row with its AI crawler access (robots.txt rules for 18 AI bots, llms.txt, Content-Signal), and for each page one row with the on-page SEO basics and a list of issue codes.

USD 2 per 1,000 pages, USD 3 per 1,000 sites. Pages that fail to load are free.

What people use it for

• Client audits. Crawl a client's site and hand over a sortable list of pages with missing titles, long meta descriptions, missing H1s, wrong canonicals, noindex and missing alt text.
• Release checks. Run it after every deploy and catch the page that suddenly got noindex or lost its title.
• AI visibility checks. See at a glance which AI bots a site's robots.txt blocks, whether that is on purpose, and whether the site publishes an llms.txt.
• Bulk AI-access check over a domain list. Turn on AI checks only and paste a list of domains: one site row per domain, no page crawl, USD 3 per 1,000 domains.

Sample output

Real rows from a run on 2026-10-05.

Site rows:

host	AI bots blocked	allowed	partial	llms.txt	llms-full.txt	robots.txt
crawlee.dev	0	18	0	200 (30,542 bytes)	200 (2,886,889 bytes)	200
apify.com	0	18	0	200 (16,837 bytes)	404	200 (Content-Signal: search=yes, ai-input=yes, ai-train=yes)
www.nytimes.com (AI checks only)	16	0	2 (Amazonbot, MistralAI-User)	404	404	200 (8,547 bytes)

Page rows:

url	status	title	words	ms	issues
https://crawlee.dev/	200	Crawlee · Build reliable crawlers. Fast.	282	67	no-jsonld
https://crawlee.dev/python	200	Crawlee for Python · Fast, reliable Python web crawlers.	677	15	canonical-other, no-jsonld
https://crawlee.dev/blog	200	Crawlee Blog - learn how to build better scrapers | Crawlee for JavaScript · Build reliable crawlers. Fast.	1,597	15	title-too-long, no-h1
https://apify.com/actors	200	Automate anything with Apify Actors · Apify	737	15	description-too-long

One site row in full (the aiBots object is cut to three of its 18 bots here):

{
  "type": "site",
  "host": "crawlee.dev",
  "robotsTxtStatus": 200,
  "robotsTxtBytes": 103,
  "sitemaps": ["https://crawlee.dev/sitemap.xml", "https://crawlee.dev/python/sitemap.xml"],
  "contentSignals": null,
  "crawlDelaySecs": null,
  "aiBots": {
    "GPTBot": { "verdict": "allowed", "matchedGroup": "*" },
    "ClaudeBot": { "verdict": "allowed", "matchedGroup": "*" },
    "PerplexityBot": { "verdict": "allowed", "matchedGroup": "*" }
  },
  "aiBlockedCount": 0,
  "aiAllowedCount": 18,
  "aiPartialCount": 0,
  "llmsTxtStatus": 200,
  "llmsTxtBytes": 30542,
  "llmsTxtTitle": "Crawlee for Python · Fast, reliable Python web crawlers.",
  "llmsTxtLinkCount": 361,
  "llmsFullTxtStatus": 200,
  "llmsFullTxtBytes": 2886889,
  "checkedAt": "2026-10-05T16:39:19.598Z",
  "error": null
}

One page row in full:

{
  "type": "page",
  "url": "https://crawlee.dev/",
  "finalUrl": "https://crawlee.dev/",
  "status": 200,
  "responseMs": 67,
  "redirects": 0,
  "bytes": 53556,
  "title": "Crawlee · Build reliable crawlers. Fast.",
  "titleLength": 40,
  "metaDescription": "Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.",
  "metaDescriptionLength": 146,
  "h1Count": 1,
  "h1": "Build reliable web scrapers. Fast.",
  "canonical": "https://crawlee.dev/",
  "canonicalStatus": "self",
  "metaRobots": null,
  "xRobotsTag": null,
  "noindex": false,
  "aiDirectives": [],
  "lang": "en",
  "hreflangCount": 2,
  "ogTitle": "Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "ogImage": true,
  "jsonLdTypes": [],
  "jsonLdErrors": 0,
  "imageCount": 14,
  "imagesMissingAlt": 0,
  "wordCount": 282,
  "internalLinks": 17,
  "externalLinks": 10,
  "issues": ["no-jsonld"],
  "checkedAt": "2026-10-05T16:39:19.100Z",
  "error": null
}

Input

Field	What it does
startUrls	Sites or pages to audit. Each host is audited once for AI access. In the default crawl mode, if the first start URL of a host redirects to another host (for example to www.), that host is the one audited.
maxPagesPerSite	Pages fetched per site (default 50, up to 10,000). Pages blocked by robots.txt and non-HTML files do not count.
useSitemap	Take page URLs from the sitemaps listed in robots.txt (or /sitemap.xml when none is listed) instead of following links.
sameHostOnly	On by default: follow links only on the site's own host. Off: also follow links to other subdomains of the same domain, sharing the same page limit. Links to other domains are never followed.
aiChecksOnly	Only the site row, no page crawl. The cheapest mode for long domain lists.
maxConcurrency	Requests in flight for the whole run (default 5). Per host the limits below always apply.
outputIssuesOnly	Only page rows that have at least one issue (or an error). Clean pages are left out and not charged.

Site row fields

Field	Meaning
host	The host audited.
robotsTxtStatus, robotsTxtBytes	HTTP status of /robots.txt (after up to 5 redirects) and its size.
sitemaps	Sitemap: lines from robots.txt.
contentSignals	Content-Signal: lines from robots.txt, if any (for example {"search": "yes", "ai-train": "no"}).
crawlDelaySecs	The Crawl-delay that applies to this crawler, if any.
aiBots	For each AI bot: verdict is allowed, blocked (the whole site), partial (some paths blocked, or only some allowed) or unknown (robots.txt answered 5xx); matchedGroup is the bot's own name when robots.txt has a group for it, * when the catch-all group decides, or null when there is no robots.txt or no group applies.
aiBlockedCount, aiAllowedCount, aiPartialCount	Counts over the 18 bots.
llmsTxtStatus, llmsTxtBytes, llmsTxtTitle, llmsTxtLinkCount	/llms.txt: HTTP status, size, first # heading and number of Markdown links. Status is soft-404 when the site answers with an HTML page instead of a text file, and skipped-by-robots when robots.txt keeps this crawler out.
llmsFullTxtStatus, llmsFullTxtBytes	The same for /llms-full.txt (size counted up to 50 MB).
error	Why the site could not be checked (DNS failure, private address, robots.txt 5xx).

When robots.txt answers 404 (or another 4xx), every bot is allowed, as Google treats it. When it answers 5xx or 429, the verdicts are unknown and no pages are crawled.

Page row fields and issue codes

Each page row has url, finalUrl, status, responseMs (time to response headers, summed over redirects), redirects, bytes, title, metaDescription, h1Count, h1, canonical, canonicalStatus (self, other or missing), metaRobots, xRobotsTag, noindex, aiDirectives (noai, noimageai found in meta tags or X-Robots-Tag), lang, hreflangCount, ogTitle, ogImage, jsonLdTypes, jsonLdErrors, imageCount, imagesMissingAlt, wordCount (visible text), internalLinks, externalLinks, issues and error.

Issue code	When
missing-title	No <title> or an empty one.
title-too-long	Title longer than 60 characters.
missing-description	No meta description.
description-too-long	Meta description longer than 160 characters.
no-h1 / multiple-h1	No <h1>, or more than one.
noindex	noindex or none in meta robots or the X-Robots-Tag header.
canonical-other	The canonical URL points to a different URL.
missing-canonical	No canonical link.
no-jsonld	No JSON-LD structured data block that parses.
missing-alt	At least one <img> without an alt attribute (alt="" counts as present).
thin-content	Fewer than 200 words of visible text.
possibly-js-rendered	Fewer than 50 words and 5 or more scripts: the content is probably built by JavaScript, which this actor does not run.
slow	More than 1,500 ms to the response headers.
http-error	Status 400 or higher (the only issue reported for such pages).

AI bots checked (list updated 2026-10-05)

Bot	Owner	Purpose
GPTBot	OpenAI	training
OAI-SearchBot	OpenAI	search
ChatGPT-User	OpenAI	user-triggered
ClaudeBot	Anthropic	training
Claude-SearchBot	Anthropic	search
Claude-User	Anthropic	user-triggered
anthropic-ai	Anthropic (older token)	training
PerplexityBot	Perplexity	search
Perplexity-User	Perplexity	user-triggered
Google-Extended	Google (Gemini training and grounding control)	training
Applebot-Extended	Apple (Apple Intelligence training control)	training
CCBot	Common Crawl	training
Bytespider	ByteDance	training
meta-externalagent	Meta	training
Amazonbot	Amazon	search
cohere-ai	Cohere	training
DuckAssistBot	DuckDuckGo	search
MistralAI-User	Mistral	user-triggered

Training bots collect pages to train models. Search bots build an AI search or answer index. User-triggered fetchers load a page when a person asks an AI assistant about it. Blocking training bots while allowing search bots is a common, deliberate choice; blocking search bots means the site is unlikely to be cited in those AI answers.

Pricing

What	Price
Page audited (any status under 500, including 404s: that is the answer)	USD 0.002 (USD 2 per 1,000)
Site AI-access check (robots.txt answered with a status under 500, other than 429)	USD 0.003 (USD 3 per 1,000)
Pages that fail to load (DNS, timeout, TLS), 5xx pages, invalid URLs, unreachable sites, pages blocked by robots.txt, clean pages left out by outputIssuesOnly	Free

Examples on the free plan:

• The default input (2 sites, 10 pages each): 2 x 0.003 + 20 x 0.002 = USD 0.046.
• One site, 500 pages: 0.003 + 500 x 0.002 = USD 1.003.
• 1,000 domains with AI checks only: 1,000 x 0.003 = USD 3.

Higher Apify plans get a discount. Set a maximum cost per run in the run options; the actor stops cleanly when it is reached and keeps everything written so far.

Schedule it weekly

Save the input as a task and schedule it weekly. Compare runs to see which issues appeared or were fixed, and whether a site changed its AI bot rules. Export to Google Sheets, Excel or a webhook.

How it crawls (politeness)

• Every request identifies the actor: Mozilla/5.0 (compatible; seo-ai-readiness-audit/1.0; +https://apify.com/JTPalms). It never pretends to be a browser or an AI bot. The AI bot verdicts are read from robots.txt, never tested by sending a bot's name.
• It obeys robots.txt for its own names (seo-ai-readiness-audit and jtpalms-seo-audit, or the * group when neither is named): disallowed pages are skipped, free, and counted as skippedByRobots in the run summary. If robots.txt keeps it out of /llms.txt, that file is reported as skipped-by-robots instead of fetched.
• At most 2 requests at a time to one host, at least 250 ms between them, and a robots.txt Crawl-delay is honoured up to 10 seconds.
• 20 seconds timeout per request; network errors and 5xx answers are retried twice, 4xx answers are not.
• HTTP only: no browser, no proxies. Some sites block data-center traffic or bots; you see their real answer (often 403), which is charged as an http-error page because it is the page's status.
• Private and internal addresses (localhost, 10.x, 169.254.x and similar) are refused.
• Crawling a site may be restricted by that site's terms; only audit sites you own or are allowed to audit.

What this does not do

• No JavaScript rendering. Pages are read as the server sends them. Sites built entirely in the browser show thin content and get possibly-js-rendered.
• No Core Web Vitals or Lighthouse scores. responseMs is server response time only.
• No backlink or authority data. For that, see Domain Authority Checker.
• No broken-link report for every link. It audits the pages it crawls; for every broken link on a site use Broken Link Checker, and to list every URL in a sitemap use Sitemap URL Extractor.
• It does not check whether AI bots actually visit the site (that needs the site's server logs), only what robots.txt and the pages tell them.

Related actors

• Broken Link Checker: Find Dead Links on Any Website: Crawl a website and find every broken link, image, script and stylesheet: 404s, 5xx, DNS and TLS errors, redirect loops and redirected...
• URL Status Checker: Bulk HTTP Status Code & Redirect Checker: Bulk URL status and HTTP status code checker.
• Sitemap URL Extractor: All Pages of Any Website: Get every URL of a website from its XML sitemaps.
• Domain Authority Checker: Backlink Rank from Open Data: Bulk domain authority checker built on open data: website authority and backlink strength from the Majestic Million (referring subnets...

FAQ

A page works in my browser but shows 403 here. Why? The site blocks automated requests, and this actor says plainly that it is one. That 403 is what search and AI crawlers from data centers may see too.

Why is llmsTxtStatus "soft-404"? The site answered /llms.txt with an ordinary HTML page (often its home page or a "not found" page with status 200) instead of a text file. AI tools looking for llms.txt get nothing useful.

Does allowed mean AI search engines will show the site? No. It means robots.txt does not keep that bot out. Whether an AI answer cites the site depends on the engine.

Something missing or wrong? Open an issue on the actor page with the URL.