# AI Crawler Checker & GEO / AEO Audit (llms.txt, robots.txt) (`chivalrous_skyline_vtd/ai-search-readiness-audit`) Actor Check if ChatGPT, Perplexity, Gemini and Claude can crawl and cite your site: AI-crawler rules in robots.txt, llms.txt, sitemap, JSON-LD. Returns a 0-100 GEO score and a fix list per site. - **URL**: https://apify.com/chivalrous_skyline_vtd/ai-search-readiness-audit.md - **Developed by:** [HexaX](https://apify.com/chivalrous_skyline_vtd) (community) - **Categories:** AI, SEO tools - **Security:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $10.00 / 1,000 site auditeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## AI Search Readiness Audit (GEO / AEO) **AI Crawler Checker & GEO / AEO Audit**: check whether ChatGPT, Perplexity, Gemini and Claude can crawl and cite your website. For each site you get a 0-100 GEO score, an A-F grade and a plain list of fixes. ### What you get - Which AI search crawlers and AI training crawlers your `robots.txt` blocks. - Whether `llms.txt`, a sitemap and JSON-LD structured data are present and valid. - Page-level basics that answer engines rely on: title, meta description, canonical, headings, Open Graph, noindex. - A prioritised fix list, one row per site, as JSON, CSV or Excel. ### Use cases - **Agency monthly audits**: run the same client list every month and compare scores. - **Competitor comparison**: audit your site and five competitors in one run. - **Pre-launch check**: confirm a new site is not blocking AI crawlers by mistake. - **GEO / AEO reporting**: add a repeatable readiness score to SEO reports. ### Quick start Open the Input tab, keep the example sites (apify.com, github.com, wikipedia.org) or enter your own, and press Start. The Output tab shows one row per site. ### What is checked - **robots.txt**: exists; does not block everything; which AI crawlers are blocked. AI *search* bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Google-Extended, Applebot-Extended) lower the score when blocked. AI *training* bots (GPTBot, anthropic-ai, CCBot, Bytespider, Meta-ExternalAgent) are reported but never scored: opting out of training is a legitimate choice. - **llms.txt**: present, starts with an H1, number of links. - **sitemap**: from robots.txt `Sitemap:` lines, or `/sitemap.xml`; URL count. - **The given page**: title, meta description, canonical, `lang`, single `

`, JSON-LD types (and invalid blocks), Open Graph, noindex (meta or `X-Robots-Tag`), amount of text in the raw HTML. ### Input ```json { "urls": ["https://apify.com", "https://github.com", "https://www.wikipedia.org"], "maxSites": 3 } ``` One audit per site (origin); duplicates are merged. ### Output (one row per site) `origin`, `status` (`ok` / `unreachable`), `score`, `grade`, `robotsTxt`, `llmsTxt`, `sitemap`, `page`, `issues` (list of fixes), `error`. ### Pricing Pay per event: one `site-audited` event per site with `status: ok`. Unreachable sites are not charged. ### Use with AI agents (API) Call the Actor through the Apify API, the Apify client libraries or an MCP client and read the dataset rows as JSON. Each row has a numeric `score` and an `issues` list, which an agent can turn into a to-do list. ### FAQ **Does blocking GPTBot hurt my score?** No. AI training bots are reported but never scored; opting out of training is a legitimate choice. Only AI search bots (OAI-SearchBot, PerplexityBot, ClaudeBot and similar) lower the score when blocked. **Is the score a ranking or citation measurement?** No. It is a checklist with fixed weights, not a measurement of how often AI tools cite you. **Why does my JavaScript-only content count as missing?** The HTML is not rendered, which is also how many AI crawlers see a page. **Am I charged for unreachable sites?** No. Only sites with `status: ok` are charged. ### Behaviour and limits - Reads only public files and the one page you give. robots.txt is respected: if it disallows this Actor's user agent for your page, the page checks are skipped and said so. - 1 second pause between sites. Pages are read up to 2 MB. HTML is not rendered: content that only appears after JavaScript runs counts as missing, which is also how many AI crawlers see it. - The score is a checklist with fixed weights, not a measurement of rankings or citations. ### Run the tests `python -m unittest discover -s tests -t .` (no network access needed). ### Related actors - [Video Frame Extractor](https://apify.com/chivalrous_skyline_vtd/video-frame-extractor): frames, scene changes and thumbnails from video file URLs. - [Stable Diffusion Prompt Extractor](https://apify.com/chivalrous_skyline_vtd/sd-prompt-extractor): read prompts and settings embedded in AI images. - [Japanese to Romaji, Hiragana & Furigana Converter](https://apify.com/chivalrous_skyline_vtd/japanese-furigana-romaji): readings for Japanese text. - [Japan Diet Parliament Debates & Speeches Search](https://apify.com/chivalrous_skyline_vtd/japan-diet-minutes): speeches from Japan's National Diet since 1947. # Actor input Schema ## `urls` (type: `array`): Home page (or any page) of each site to audit. One audit is run per site (origin). Only public pages are read; robots.txt is respected. ## `maxSites` (type: `integer`): Safety limit on the number of sites audited in one run (1-500). ## Actor input object example ```json { "urls": [ "https://apify.com", "https://github.com", "https://www.wikipedia.org" ], "maxSites": 3 } ``` # Actor output Schema ## `results` (type: `string`): Results in the run's default dataset. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://apify.com", "https://github.com", "https://www.wikipedia.org" ], "maxSites": 3 }; // Run the Actor and wait for it to finish const run = await client.actor("chivalrous_skyline_vtd/ai-search-readiness-audit").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": [ "https://apify.com", "https://github.com", "https://www.wikipedia.org", ], "maxSites": 3, } # Run the Actor and wait for it to finish run = client.actor("chivalrous_skyline_vtd/ai-search-readiness-audit").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://apify.com", "https://github.com", "https://www.wikipedia.org" ], "maxSites": 3 }' | apify call chivalrous_skyline_vtd/ai-search-readiness-audit --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,chivalrous_skyline_vtd/ai-search-readiness-audit" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/OI1vKNsaFP16g7Vbg/builds/zrbZYgUh0s6fdjsTS/openapi.json