Pricing

from $20.00 / 1,000 domain auditeds

Try for free

Go to Apify Store

Crawlability & AI Crawler Access Auditor

Try for free

AI crawler access audit and robots.txt checker for any domain: rules per crawler (GPTBot, ClaudeBot, CCBot, Google-Extended and 25 more), Content-Signal and Content-Usage, sitemaps with URL counts, llms.txt, meta robots on sample pages, TDM opt-out detection and the anti-bot service in front.

Pricing

from $20.00 / 1,000 domain auditeds

Rating

0.0

(0)

Developer

Leyten Data

Actor stats

Bookmarked

Total users

Monthly active users

5 hours ago

Last modified

Categories

Developer tools

SEO tools

You can access the Crawlability & AI Crawler Access Auditor programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "domains": [
9        "nos.nl",
10        "rijksoverheid.nl",
11        "apify.com",
12        "anthropic.com",
13    ] }
14
15# Run the Actor and wait for it to finish
16run = client.actor("lwsdjfls/crawlability-ai-access-auditor").call(run_input=run_input)
17
18# Fetch and print Actor results from the run's dataset (if there are any)
19print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
20for item in client.dataset(run.default_dataset_id).iterate_items():
21    print(item)
22
23# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Crawlability & AI Crawler Access Audit (llms.txt) API in Python

The Apify API client for Python is the official library that allows you to use Crawlability & AI Crawler Access Auditor API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Crawlability & AI Crawler Access Auditor API in JavaScript

Crawlability & AI Crawler Access Auditor API through CLI

Crawlability & AI Crawler Access Auditor OpenAPI definition

Crawlability & AI Crawler Access Auditor API

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

GEO Auditor — GEO Readiness & AI Visibility Audit

yizheng/geo-auditor

Technical site audit + live LLM citation check: ChatGPT, Claude, Gemini. 6-axis SEO audit with scoring & radar charts. $0.03/run.

yizheng zhou

GEO & AEO Site Audit — llms.txt, AI Crawlers & Schema Fix List

wheaten_steelpan/geo-site-auditor

Audit your website for Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO). Crawls pages, tests robots.txt/llms.txt/schema.org, evaluates AI answerability, and generates ranked fixes + visual report.

Vanja V

AI Readiness Auditor

muhammadafzal/ai-readiness-auditor

Audit website crawlability, AI bot access, llms.txt, sitemaps, metadata, JSON-LD, content, and accessibility with prioritized page-level fixes.

Muhammad Afzal

Similarweb Traffic Intelligence Scraper

automation-lab/similarweb-traffic-intelligence-scraper

Extract public Similarweb traffic estimates, engagement, rankings, channels, countries, and keywords for domain cohorts. No Similarweb account required.

Automation Lab

SEO Auditor & Rank Tracker — BrightEdge Alternative

khadinakbar/brightedge-alternative

Audit a public website and optionally check Google keyword ranks. Get one report with technical issues, page evidence, prioritized fixes, robots and sitemap checks, and keyword observations for an SEO repair plan.

Khadin Akbar

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

GEO Auditor — AI Search Readiness & Citability Audit

foxlabs/geo-auditor

Audit how ready your site is to be found, read & cited by AI search (ChatGPT, Perplexity, Gemini, Claude). Checks AI-crawler access, structured data, content extractability, speed & trust — scored 0-100 with a prioritized fix list. GEO / AEO technical audit.

Berkan Kaplan

SEO Audit: On-Page, Redirects & AI Crawler Access

aiqlabs/seo-audit-tool

Give it a list of URLs and it audits each page for on-page and technical SEO, plus the four checks general auditors skip: AI crawler rules, /llms.txt, redirects that throw the path away, and noindex via X-Robots-Tag. One row per URL - to crawl a whole site instead, use SEO & AI Visibility Auditor.

Ai-Q Labs

Website SEO, Accessibility & AI Readiness Auditor

feeng/website-seo-ai-readiness-auditor

🔎 Crawl websites and turn technical SEO, accessibility, security, performance, broken-link, structured-data, robots.txt, llms.txt, and AI crawler signals into prioritized 0–100 audit scores and client-ready reports.