# AI Search Visibility Check – AI Bots, llms.txt, JS Text (`petipeti/ai-search-visibility-check`) Actor

See your pages the way AI search does: robots.txt access for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 14 more, noindex/noai directives, llms.txt with a generated draft, and how much main text is visible without JavaScript.

- **URL**: https://apify.com/petipeti/ai-search-visibility-check.md
- **Developed by:** [peti peti](https://apify.com/petipeti) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Visibility Check – AI Bots, llms.txt, JS Text

**Can ChatGPT, Claude, Perplexity and Google's AI features actually read your pages?** This Actor checks a list of your own URLs and tells you, per page:

1. **How much of the main text is visible without JavaScript** – the headline number. Most AI crawlers download the HTML and do not run JavaScript, so a page that draws its content with React/Vue/Next.js on the client can look almost empty to them. The Actor loads each page twice (plain HTML, and in headless Chromium) and reports the share of the rendered main text that already exists in the plain HTML.
2. **Which AI crawlers your robots.txt allows or blocks for that page** – GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot-Extended, Amazonbot, meta-externalagent, CCBot, Bytespider, DuckAssistBot and MistralAI-User – each labelled as *training*, *search* or *user-triggered*, so you can block training while staying visible in AI search.
3. **noindex / nosnippet / noai directives** in meta tags (generic and bot-specific, e.g. `<meta name="GPTBot">`) and the `X-Robots-Tag` header.
4. **Whether `/llms.txt` exists** – and if not, a **ready-to-edit llms.txt draft** built from your page title, description and site links.

Built for **site owners, marketers, SEO/GEO consultants and developers** checking sites they own or manage.

### Polite and honest by design

- **Only the URLs you list are checked.** Links are not followed.
- **robots.txt is always obeyed** for this tool (`AiVisibilityCheck`, falling back to `*`). Disallowed pages are reported as skipped and never requested. If robots.txt returns a server error, the site is left alone (RFC 9309).
- **We never pretend to be an AI crawler.** Every request uses the tool's own User-Agent: `Mozilla/5.0 (compatible; AiVisibilityCheck/1.0; +https://apify.com/petipeti/ai-search-visibility-check)`. The AI-crawler verdicts come from reading your robots.txt, not from impersonating the bots. (Consequence: if your server or CDN treats real AI crawlers differently by User-Agent or IP, that is not visible here.)
- At least 2 seconds between requests to the same site (default 3 s). A longer `Crawl-delay` is honoured. One page at a time.
- The browser skips images, video and fonts. Nothing behind a login is accessed and no personal data is collected.

### Input

```json
{
  "startUrls": [{ "url": "https://www.example.com/" }, { "url": "https://www.example.com/pricing" }],
  "renderJs": true,
  "delaySeconds": 3
}
```

Put your home page first: the llms.txt draft is built from the first checked page of each site.

### Output (one row per URL)

```json
{
  "url": "https://www.example.com/pricing",
  "status": 200,
  "noJsTextShare": 0.18,
  "textWithoutJsChars": 210,
  "textWithJsChars": 4120,
  "blockedAiCrawlers": ["GPTBot", "CCBot"],
  "aiCrawlers": [
    { "bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "allowed": false, "rule": "User-agent: GPTBot" },
    { "bot": "OAI-SearchBot", "operator": "OpenAI", "purpose": "search", "allowed": true, "rule": "User-agent: *" }
  ],
  "directives": { "robotsMeta": "index, follow", "xRobotsTag": null, "noindex": false, "nosnippet": false, "noai": false },
  "llmsTxt": { "present": false, "status": 404 },
  "llmsTxtDraft": "# Example\n\n> ...\n\n## Main pages\n\n- [Pricing](https://www.example.com/pricing)\n...",
  "issues": [
    "Only 18% of the main text is in the HTML without JavaScript – most AI crawlers do not run JavaScript, so they see little of this page",
    "Opted out of AI training for: GPTBot, CCBot (intentional for many sites)",
    "No /llms.txt – a starting draft is in llmsTxtDraft (first row of this site)"
  ]
}
```

The llms.txt draft is also saved as a `llms-txt-<host>` record in the key-value store, ready to download.

#### How the "visible without JavaScript" share is measured

The main content of the rendered page (`<main>`, the longest `<article>`, or the body without header/nav/footer) is split into words – character pairs for Japanese, Chinese and Korean – and the Actor counts how many of them also appear in the plain HTML (scripts, styles and `<noscript>` excluded). 1.0 means everything is server-rendered; values under 0.5 mean AI crawlers that skip JavaScript miss most of the page. Turn `renderJs` off for a quick robots/llms.txt-only run (the share is then `null`).

### Pricing

Pay per event, platform usage included (headless browser time is covered):

| Event | Price |
|---|---|
| Actor start | $0.01 per run |
| `url-checked` – one URL checked successfully | $4.00 per 1,000 URLs ($0.004 each) |

Example: checking 50 pages costs $0.01 + $0.20 = $0.21. URLs skipped because of robots.txt or errors are not charged. A run without URLs only returns a setup reminder and charges nothing beyond the start.

### Limits

- What AI products actually do with a page also depends on their own policies, your CDN/bot-protection rules and IP-based decisions. This Actor reports what your robots.txt, directives and HTML say.
- `noai` / `noimageai` are not official standards and are respected only by some tools.
- llms.txt is a proposed convention (llmstxt.org); support by AI products varies.

### 日本語での説明

自分のサイトのページが AI 検索（ChatGPT・Claude・Perplexity・Google の AI 機能など）からどう見えるかを調べます。指定した URL だけを調べ、robots.txt に必ず従い、同じサイトへのアクセスは最低2秒あけます。AI クローラーのふりはせず、このツール自身の User-Agent で取得します。

- JavaScript なしで見える本文の割合（多くの AI クローラーは JavaScript を実行しないため、画面をあとから描くサイトはほぼ空に見えることがあります）。日本語は2文字ずつに区切って比べます
- robots.txt で GPTBot・ClaudeBot・PerplexityBot・Google-Extended など18種が許可されているか（学習用・検索用・ユーザー操作用の区別つき）
- noindex・nosnippet・noai（meta タグと X-Robots-Tag）
- /llms.txt があるか。ない場合はページのタイトル・説明・リンクから下書きを作ります

料金: 1回の実行ごとに $0.01、1 URL あたり $0.004（1,000 URL で $4）。robots.txt で止められた URL やエラーの URL は課金しません。

# Actor input Schema

## `startUrls` (type: `array`):

Pages of websites you own or manage. Only these URLs are checked; links are not followed. Put the home page first to get the best llms.txt draft.

## `renderJs` (type: `boolean`):

Loads each page once more in headless Chromium and measures how much of the main text exists only after JavaScript runs. Turn off for a faster robots/llms.txt-only check.

## `maxUrls` (type: `integer`):

Upper limit of URLs checked in one run.

## `delaySeconds` (type: `integer`):

At least 2 seconds. A longer Crawl-delay in robots.txt is used instead.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "renderJs": true,
  "maxUrls": 50,
  "delaySeconds": 3
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL: share of main text visible without JavaScript, AI crawler access, directives, llms.txt and findings.

## `llmsTxtDrafts` (type: `string`):

Key-value store with a llms-txt-<host> record for each site without /llms.txt.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("petipeti/ai-search-visibility-check").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("petipeti/ai-search-visibility-check").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}' |
apify call petipeti/ai-search-visibility-check --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,petipeti/ai-search-visibility-check"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3MLLmGh1Np6u34EQi/builds/4jvZDaqN6JVw5bzVn/openapi.json