AI Search Visibility Check – AI Bots, llms.txt, JS Text

Can ChatGPT, Claude, Perplexity and Google's AI features actually read your pages? This Actor checks a list of your own URLs and tells you, per page:

1. How much of the main text is visible without JavaScript – the headline number. Most AI crawlers download the HTML and do not run JavaScript, so a page that draws its content with React/Vue/Next.js on the client can look almost empty to them. The Actor loads each page twice (plain HTML, and in headless Chromium) and reports the share of the rendered main text that already exists in the plain HTML.
2. Which AI crawlers your robots.txt allows or blocks for that page – GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot-Extended, Amazonbot, meta-externalagent, CCBot, Bytespider, DuckAssistBot and MistralAI-User – each labelled as training, search or user-triggered, so you can block training while staying visible in AI search.
3. noindex / nosnippet / noai directives in meta tags (generic and bot-specific, e.g. <meta name="GPTBot">) and the X-Robots-Tag header.
4. Whether /llms.txt exists – and if not, a ready-to-edit llms.txt draft built from your page title, description and site links.

Built for site owners, marketers, SEO/GEO consultants and developers checking sites they own or manage.

Polite and honest by design

• Only the URLs you list are checked. Links are not followed.
• robots.txt is always obeyed for this tool (AiVisibilityCheck, falling back to *). Disallowed pages are reported as skipped and never requested. If robots.txt returns a server error, the site is left alone (RFC 9309).
• We never pretend to be an AI crawler. Every request uses the tool's own User-Agent: Mozilla/5.0 (compatible; AiVisibilityCheck/1.0; +https://apify.com/petipeti/ai-search-visibility-check). The AI-crawler verdicts come from reading your robots.txt, not from impersonating the bots. (Consequence: if your server or CDN treats real AI crawlers differently by User-Agent or IP, that is not visible here.)
• At least 2 seconds between requests to the same site (default 3 s). A longer Crawl-delay is honoured. One page at a time.
• The browser skips images, video and fonts. Nothing behind a login is accessed and no personal data is collected.

Input

{
  "startUrls": [{ "url": "https://www.example.com/" }, { "url": "https://www.example.com/pricing" }],
  "renderJs": true,
  "delaySeconds": 3
}

Put your home page first: the llms.txt draft is built from the first checked page of each site.

Output (one row per URL)

{
  "url": "https://www.example.com/pricing",
  "status": 200,
  "noJsTextShare": 0.18,
  "textWithoutJsChars": 210,
  "textWithJsChars": 4120,
  "blockedAiCrawlers": ["GPTBot", "CCBot"],
  "aiCrawlers": [
    { "bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "allowed": false, "rule": "User-agent: GPTBot" },
    { "bot": "OAI-SearchBot", "operator": "OpenAI", "purpose": "search", "allowed": true, "rule": "User-agent: *" }
  ],
  "directives": { "robotsMeta": "index, follow", "xRobotsTag": null, "noindex": false, "nosnippet": false, "noai": false },
  "llmsTxt": { "present": false, "status": 404 },
  "llmsTxtDraft": "# Example\n\n> ...\n\n## Main pages\n\n- [Pricing](https://www.example.com/pricing)\n...",
  "issues": [
    "Only 18% of the main text is in the HTML without JavaScript – most AI crawlers do not run JavaScript, so they see little of this page",
    "Opted out of AI training for: GPTBot, CCBot (intentional for many sites)",
    "No /llms.txt – a starting draft is in llmsTxtDraft (first row of this site)"
  ]
}

The llms.txt draft is also saved as a llms-txt-<host> record in the key-value store, ready to download.

How the "visible without JavaScript" share is measured

The main content of the rendered page (<main>, the longest <article>, or the body without header/nav/footer) is split into words – character pairs for Japanese, Chinese and Korean – and the Actor counts how many of them also appear in the plain HTML (scripts, styles and <noscript> excluded). 1.0 means everything is server-rendered; values under 0.5 mean AI crawlers that skip JavaScript miss most of the page. Turn renderJs off for a quick robots/llms.txt-only run (the share is then null).

Pricing

Pay per event, platform usage included (headless browser time is covered):

Event	Price
Actor start	$0.01 per run
url-checked – one URL checked successfully	$4.00 per 1,000 URLs ($0.004 each)

Example: checking 50 pages costs $0.01 + $0.20 = $0.21. URLs skipped because of robots.txt or errors are not charged. A run without URLs only returns a setup reminder and charges nothing beyond the start.

Limits

• What AI products actually do with a page also depends on their own policies, your CDN/bot-protection rules and IP-based decisions. This Actor reports what your robots.txt, directives and HTML say.
• noai / noimageai are not official standards and are respected only by some tools.
• llms.txt is a proposed convention (llmstxt.org); support by AI products varies.

日本語での説明

自分のサイトのページが AI 検索（ChatGPT・Claude・Perplexity・Google の AI 機能など）からどう見えるかを調べます。指定した URL だけを調べ、robots.txt に必ず従い、同じサイトへのアクセスは最低2秒あけます。AI クローラーのふりはせず、このツール自身の User-Agent で取得します。

• JavaScript なしで見える本文の割合（多くの AI クローラーは JavaScript を実行しないため、画面をあとから描くサイトはほぼ空に見えることがあります）。日本語は2文字ずつに区切って比べます
• robots.txt で GPTBot・ClaudeBot・PerplexityBot・Google-Extended など18種が許可されているか（学習用・検索用・ユーザー操作用の区別つき）
• noindex・nosnippet・noai（meta タグと X-Robots-Tag）
• /llms.txt があるか。ない場合はページのタイトル・説明・リンクから下書きを作ります

料金: 1回の実行ごとに $0.01、1 URL あたり $0.004（1,000 URL で $4）。robots.txt で止められた URL やエラーの URL は課金しません。