Pricing

$9.90 / 1,000 website checks

Try for free

Go to Apify Store

AI Crawler Access Audit | Robots.txt & User-Agent Checks

Try for free

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Pricing

$9.90 / 1,000 website checks

Rating

0.0

(0)

Developer

Andrii Klymenko

Actor stats

Bookmarked

Total users

Monthly active users

14 days ago

Last modified

AI Crawler Access Audit: Robots.txt & User-Agent Tests API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Access Audit | Robots.txt & User-Agent Checks locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Access Audit | Robots.txt & User-Agent Checks API in Python

AI Crawler Access Audit | Robots.txt & User-Agent Checks API in JavaScript

AI Crawler Access Audit | Robots.txt & User-Agent Checks OpenAPI definition

AI Crawler Access Audit | Robots.txt & User-Agent Checks API

Legacy PhantomJS Crawler

apify/legacy-phantomjs-crawler

Replacement for the legacy Apify Crawler product with a backward-compatible interface. The Actor uses PhantomJS headless browser to recursively crawl websites and extract data from them using a piece of front-end JavaScript code.

Apify

900

5.0

(8)

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

Similarweb Traffic Crawler

yeekal/similarweb-traffic-crawler

Collect public Similarweb traffic estimates in bulk with fast HTTP requests, including rankings, engagement, traffic sources, countries, keywords, competitors, and AI referrals.

yeekal

4.0

(1)

Website Content Crawler

novashieldai/website-content-crawler

Universal website crawler that extracts clean text/markdown content, metadata, links, and images from any URL. Features sitemap parsing, robots.txt respect, and multi-page BFS crawling with depth control.

Ali haydar Karadaş

Sitemap Indexability & AI Crawler Audit API

dobus/sitemap-indexability-ai-crawler-audit

Audit sitemap URLs, robots.txt, HTTP status, canonical/noindex signals, and AI crawler access as structured JSON for technical SEO and agents.

Afonso Fernandes

AI Crawler Access Auditor

kingii98/ai-crawler-access-auditor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

kingii98

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...