Pricing

Pay per event

Try for free

Go to Apify Store

AI Crawler Access & llms.txt Checker

Try for free

See which AI crawlers a website allows or blocks. Evaluates robots.txt for 27 AI and search bots (training, AI search and assistant fetchers), checks llms.txt and llms-full.txt, content signals and noai meta tags, and summarizes each site's AI policy. Bulk, pay per site.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Ian McCann

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

Categories

SEO tools

Developer tools

Security

You can access the AI Crawler Access & llms.txt Checker programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "websites": [
9        "nytimes.com",
10        "apify.com",
11        "wordpress.org",
12    ] }
13
14# Run the Actor and wait for it to finish
15run = client.actor("vex_workflow/ai-crawler-access-checker").call(run_input=run_input)
16
17# Fetch and print Actor results from the run's dataset (if there are any)
18print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
19for item in client.dataset(run.default_dataset_id).iterate_items():
20    print(item)
21
22# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Bot Blocker & llms.txt Checker (GEO / AI Visibility) API in Python

The Apify API client for Python is the official library that allows you to use AI Crawler Access & llms.txt Checker API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Crawler Access & llms.txt Checker API in JavaScript

AI Crawler Access & llms.txt Checker API through CLI

AI Crawler Access & llms.txt Checker OpenAPI definition

AI Crawler Access & llms.txt Checker API

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

Tim Zinin

5.0

(9)

AI Search Readiness Checker

epistemedeus/ai-search-readiness-checker

Audit public website signals that affect machine readability: HTML retrieval, metadata, JSON-LD, robots.txt, sitemap, and llms.txt. Get a disclosed technical score, evidence URLs, concrete fixes, and explicit limits. No ranking or citation claims.

Lucian Constantinescu

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

AI Website Spell Checker

louisdeconinck/ai-website-spell-checker

Discover and correct spelling errors across entire websites effortlessly with our AI-powered crawler. Start from any URL, and receive a comprehensive report detailing each error, suggested corrections, and severity levels. Enhance your site's professionalism and user experience today!

Louis Deconinck

5.0

(2)

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

AI Crawler Access Audit | Robots.txt & User-Agent Checks

a.klimm67/ai-crawler-reality-check

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Andrii Klymenko

AI Crawler Checker - robots.txt Checker for GPTBot & AI Bots

tidytools/ai-crawler-access-checker

Check if GPTBot, ClaudeBot, PerplexityBot and 26 more AI crawlers are blocked on your site: bulk robots.txt check of up to 10,000 domains, AI search score, fix snippet, Content Signals, change alerts. $2/1k sites.

Yukai Lin

Website Technology & Tracking Detector

vex_workflow/website-tech-stack-detector

Find out what any website runs on: CMS, e-commerce platform, analytics, ad pixels, marketing automation, chat, payments, reviews, consent tools, frameworks, hosting and CDN. Also extracts tracking IDs (GA4, GTM, Google Ads, Meta, TikTok and more). Bulk, fast, about 140 technologies.

Ian McCann

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Techionik

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.