You can access the AI Crawler Access Auditor programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "sites": [
10        "https://apify.com",
11        "https://www.anthropic.com",
12    ],
13    "paths": [
14        "/",
15        "/blog/",
16    ],
17}
18
19# Run the Actor and wait for it to finish
20run = client.actor("axelhu/ai-crawler-access-auditor").call(run_input=run_input)
21
22# Fetch and print Actor results from the run's dataset (if there are any)
23print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
24for item in client.dataset(run.default_dataset_id).iterate_items():
25    print(item)
26
27# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Crawler Robots.txt & llms.txt Access Auditor API in Python

The Apify API client for Python is the official library that allows you to use AI Crawler Access Auditor API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Crawler Access Auditor API in JavaScript

AI Crawler Access Auditor API through CLI

AI Crawler Access Auditor OpenAPI definition

AI Crawler Access Auditor API

Website SEO, Accessibility & AI Readiness Auditor

feeng/website-seo-ai-readiness-auditor

🔎 Crawl websites and turn technical SEO, accessibility, security, performance, broken-link, structured-data, robots.txt, llms.txt, and AI crawler signals into prioritized 0–100 audit scores and client-ready reports.

Daniel S.

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Web Accessibility Checker

louisdeconinck/web-accessibility-checker

Audit your entire website for WCAG 2.1 & 2.2 compliance using the industry-standard axe-core engine. Automatically crawl pages, detect accessibility violations, and generate detailed actionable reports to ensure legal compliance and improve SEO.

Louis Deconinck

Web Accessibility WCAG Auditor ($0.25/run)

openfrontier_ai/Web-Accessibility-WCAG-Auditor

A11yBot automates WCAG 2.2 audits by "seeing" your site like a human. While standard tools only scan code, our visual AI catches barriers like poor contrast and missing focus, that HTML scanners miss. Its "Triple-Check" trait cross-references the DOM, ARIA tree, and screenshots for total precision.

OpenFrontier AI

5.0

(1)

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

Ushba Khan

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.

Ushba Khan

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

Website SEO Audit Crawler – Broken Links, Schema & llms.txt

openharvest/website-seo-audit

Website SEO audit crawler: on-page issues (title, meta description, H1, canonical, hreflang, noindex), broken internal and external links, schema.org JSON-LD errors, sitemap and robots.txt checks, AI-crawler rules (GPTBot, ClaudeBot…) and llms.txt. One row per page plus a site summary.