Pricing

$1.00 / 1,000 row returneds

Try for free

Go to Apify Store

AI Crawler Access — robots.txt, llms.txt and sitemap per domain

Try for free

For every domain: whether AI crawlers are allowed, whether llms.txt exists, and whether the sitemap is reachable. No browser needed. Blocked or unreachable files are reported as unknown, never as absent.

Pricing

$1.00 / 1,000 row returneds

Rating

0.0

(0)

Developer

The Artifact Machine

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Categories

SEO tools

You can access the AI Crawler Access — robots.txt, llms.txt and sitemap per domain programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "domains": [
9        "basecamp.com",
10        "linear.app",
11    ] }
12
13# Run the Actor and wait for it to finish
14run = client.actor("artifact-machine/ai-crawler-access-enrichment").call(run_input=run_input)
15
16# Fetch and print Actor results from the run's dataset (if there are any)
17print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
18for item in client.dataset(run.default_dataset_id).iterate_items():
19    print(item)
20
21# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Crawler Access API in Python

The Apify API client for Python is the official library that allows you to use AI Crawler Access — robots.txt, llms.txt and sitemap per domain API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Crawler Access — robots.txt, llms.txt and sitemap per domain API in JavaScript

AI Crawler Access — robots.txt, llms.txt and sitemap per domain API through CLI

AI Crawler Access — robots.txt, llms.txt and sitemap per domain OpenAPI definition

AI Crawler Access — robots.txt, llms.txt and sitemap per domain API

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.

Ushba Khan

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

aiqlabs/seo-ai-visibility-auditor

Crawls a site from one starting URL and reports how visible it is to AI: which AI crawlers robots.txt admits, whether llms.txt exists, and what structured data and metadata each page offers. Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead.

Ai-Q Labs

AI Crawler Policy Lookup: robots.txt & llms.txt Checker

accountable_eel/ai-crawler-policy-lookup

Check what a website's robots.txt says about AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more) and whether it publishes an llms.txt. Reads the site's own public files directly: no API key, no third-party service. Charged only when an explicit AI policy or llms.txt is found.

Adrian Voss

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

Howth Technology Factory

AI Crawler Access and llms.txt Visibility Panel

lavender_whistle/ai-visibility-panel

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Cathal Judge

AI Crawler & llms.txt Policy Auditor

quanmatrix/ai-crawler-llmstxt-policy-auditor

Audit robots.txt policies for major AI crawlers and inspect llms.txt discovery files for public websites with structured machine-readable results.

Rafael Barreto Haddad

AI Crawler Policy Audit API | 🤖 $1/1K

webdata_labs/ai-crawler-policy-audit-api

[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.

WebData Labs

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.