Pricing

from $10.00 / 1,000 results

AI Crawler Access Checker — robots.txt & llms.txt Audit

Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended & more) can access your website. Bulk audit of robots.txt rules, llms.txt presence and sitemap for AI search visibility (GEO/AEO).

Pricing

from $10.00 / 1,000 results

Rating

0.0

(0)

Developer

Tim Zinin

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

SEO tools

Automation

You can access the AI Crawler Access Checker — robots.txt & llms.txt Audit programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "websites": [
9        "apify.com",
10        "openai.com",
11        "anthropic.com",
12    ] }
13
14# Run the Actor and wait for it to finish
15run = client.actor("zinin/ai-crawler-access-checker").call(run_input=run_input)
16
17# Fetch and print Actor results from the run's dataset (if there are any)
18print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
19for item in client.dataset(run["defaultDatasetId"]).iterate_items():
20    print(item)
21
22# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Crawler Access Checker — robots.txt & llms.txt Audit API in Python

The Apify API client for Python is the official library that allows you to use AI Crawler Access Checker — robots.txt & llms.txt Audit API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Crawler Access Checker — robots.txt & llms.txt Audit API in JavaScript

AI Crawler Access Checker — robots.txt & llms.txt Audit API through CLI

AI Crawler Access Checker — robots.txt & llms.txt Audit OpenAPI definition

AI Crawler Access Checker — robots.txt & llms.txt Audit API

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

(1)

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

llms.txt Generator - Make Any Website AI-Readable

apifmcpfactory/llms-txt-generator

Generates a spec-compliant llms.txt file for any website from the site's own pages and metadata (no AI invention) - the fix when an AI-readiness audit finds llms.txt missing; $0.02 per site.

Fred Jones

AI Crawler Robots.txt Audit

davidbenittah/ai-crawler-robots-audit

Check whether ChatGPT, Claude, Perplexity, Google AI and other major AI crawlers can access each website, then separate search visibility risks from training protection.

David

llms.txt Checker - AI Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Fred Jones

EU VAT Validator (VIES)

apifmcpfactory/eu-vat-validator

Validates EU VAT numbers against the EU's official VIES registry, returning validity, business name and address, and an official consultation number as audit evidence - use before zero-rating an intra-EU B2B invoice or to clean a customer VAT list; $0.005 per number checked.

Fred Jones

Website Technology Detector - Tech Stack Scanner

apifmcpfactory/tech-stack-detector

Detects the technologies a website is built with (CMS, ecommerce, analytics, frameworks; 7,500+ signatures) - use when you need a site's tech stack for lead qualification or competitor research; $0.005 per site checked.

Fred Jones

SEO, GEO & AEO Audit — AI Search Readiness Checker

fayoussef/seo-geo-aeo-audit

Audit any website for Google SEO and AI search visibility in one run. Get 0–100 SEO, GEO & AEO scores per page, AI-crawler & llms.txt checks, schema validation, and a prioritized fix list with a shareable HTML report. No API keys needed — just enter a URL.

youssef farhan

GEO & llms.txt Readiness Auditor

gp005/geo-llms-auditor

Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.

Geo

llms.txt Generator & Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-suite

Generate, audit and monitor your llms.txt so ChatGPT and AI agents can read your site. Get an AI-readiness score and a ready-to-publish file. Works on public pages only. We store nothing personal.

Fred Jones

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.