Pricing

from $5.00 / 1,000 results

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

Crawls a site from one starting URL and reports how visible it is to AI: which AI crawlers robots.txt admits, whether llms.txt exists, and what structured data and metadata each page offers. Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead.

Pricing

from $5.00 / 1,000 results

Rating

0.0

(0)

Developer

Ai-Q Labs

Actor stats

Bookmarked

Total users

Monthly active users

9 days ago

Last modified

Categories

SEO tools

Developer tools

You can access the AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "startUrls": [{ "url": "https://apify.com" }] }
9
10# Run the Actor and wait for it to finish
11run = client.actor("aiqlabs/seo-ai-visibility-auditor").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI SEO Audit Tool: robots.txt, llms.txt & AI Crawler Checker API in Python

The Apify API client for Python is the official library that allows you to use AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers API in JavaScript

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers API through CLI

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers OpenAPI definition

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers API

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

Agent Readiness Audit

ondrejulehla/agent-readiness-audit

Check how ready a website is for AI agents: robots.txt policy for AI crawlers, llms.txt, sitemap, structured data, MCP server card, and JavaScript dependence.

Ondřej Úlehla

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

The Howlers

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Howth Technology Factory

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.

Ushba Khan

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Evidence-based audit of your own website: which AI crawlers robots.txt lets in (RFC 9309, 21 user-agent and control tokens, checked against the operators' own docs), which pages are discoverable, and each page's technical readiness — a ticket-grade fix plan with honest coverage disclosure.

Ken Yao

AI SEO Tracker - Technical & AI Search Audit

khadinakbar/ai-seo-tracker

Audit a public site for technical SEO and AI-search readiness. Returns source-linked checks for robots.txt, sitemap, llms.txt, indexing directives, metadata, canonical URLs and structured data. Use for site audits and repeatable monitoring; not for AI-answer brand visibility.