Pricing

$9.90 / 1,000 website checks

Try for free

Go to Apify Store

AI Crawler Access Audit | Robots.txt & User-Agent Checks

Try for free

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Pricing

$9.90 / 1,000 website checks

Rating

0.0

(0)

Developer

Andrii Klymenko

Actor stats

Bookmarked

Total users

Monthly active users

14 days ago

Last modified

Categories

SEO tools

Developer tools

Security

You can access the AI Crawler Access Audit | Robots.txt & User-Agent Checks programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "urls": [{ "url": "https://example.com/" }] }
9
10# Run the Actor and wait for it to finish
11run = client.actor("a.klimm67/ai-crawler-reality-check").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Crawler Access Audit: Robots.txt & User-Agent Tests API in Python

The Apify API client for Python is the official library that allows you to use AI Crawler Access Audit | Robots.txt & User-Agent Checks API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Crawler Access Audit | Robots.txt & User-Agent Checks API in JavaScript

AI Crawler Access Audit | Robots.txt & User-Agent Checks API through CLI

AI Crawler Access Audit | Robots.txt & User-Agent Checks OpenAPI definition

AI Crawler Access Audit | Robots.txt & User-Agent Checks API

Legacy PhantomJS Crawler

apify/legacy-phantomjs-crawler

Replacement for the legacy Apify Crawler product with a backward-compatible interface. The Actor uses PhantomJS headless browser to recursively crawl websites and extract data from them using a piece of front-end JavaScript code.

Apify

900

5.0

(8)

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

Similarweb Traffic Crawler

yeekal/similarweb-traffic-crawler

Collect public Similarweb traffic estimates in bulk with fast HTTP requests, including rankings, engagement, traffic sources, countries, keywords, competitors, and AI referrals.

yeekal

4.0

(1)

Website Content Crawler

novashieldai/website-content-crawler

Universal website crawler that extracts clean text/markdown content, metadata, links, and images from any URL. Features sitemap parsing, robots.txt respect, and multi-page BFS crawling with depth control.

Ali haydar Karadaş

Sitemap Indexability & AI Crawler Audit API

dobus/sitemap-indexability-ai-crawler-audit

Audit sitemap URLs, robots.txt, HTTP status, canonical/noindex signals, and AI crawler access as structured JSON for technical SEO and agents.

Afonso Fernandes

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...