Pricing

Pay per event

Try for free

Go to Apify Store

robots.txt & llms.txt Monitor: AI Crawler Block Checker

Try for free

Bulk-check which AI crawlers (GPTBot, ClaudeBot, etc.) each site blocks in robots.txt, plus llms.txt presence. No login. Export JSON/CSV/Excel. Pay only per result.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Jeremy Yarbrough

Actor stats

Bookmarked

Total users

Monthly active users

15 days ago

Last modified

robots.txt & llms.txt Monitor: AI Crawler Block Checker API through CLI

The Apify CLI is the official tool that allows you to use robots.txt & llms.txt Monitor: AI Crawler Block Checker locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

robots.txt & llms.txt Monitor: AI Crawler Block Checker API in Python

robots.txt & llms.txt Monitor: AI Crawler Block Checker API in JavaScript

robots.txt & llms.txt Monitor: AI Crawler Block Checker OpenAPI definition

robots.txt & llms.txt Monitor: AI Crawler Block Checker API

LLM Visibility Monitor: AI Search Brand Audit

constructive_calm/llm-visibility-monitor

See whether AI models recommend your brand. Asks real buyer questions (without naming you) to the API models behind ChatGPT, Claude, Gemini, Perplexity and Grok, with or without live web search, and reports mentions, cited sources, competitor share and changes since your last run.

Omar Eldeeb

5.0

(1)

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

Sitemap URL Extractor & New Page Monitor

mediapro-tools/sitemap-url-change-monitor

Pull every URL, lastmod and priority from any site's XML sitemaps (robots.txt discovery, indexes, .gz). No login. Export JSON/CSV/Excel. Pay only per result.

Jeremy Yarbrough

Bulk SSL Certificate Expiry Checker & Monitor

mediapro-tools/ssl-certificate-expiry-monitor

Check SSL/TLS certificates for thousands of domains: days to expiry, issuer, SANs, hostname match, trust chain, TLS version and cipher. No login. Export JSON/CSV/Excel. Pay only per result.

Jeremy Yarbrough

llms.txt Generator & Checker (plus llms-full.txt)

mediapro-tools/llms-txt-generator-checker

Make any website AI-ready. No login. Export JSON/CSV/Excel. Pay only per result.

Jeremy Yarbrough

ClinicalTrials.gov Search & Monitor (Official API v2)

mediapro-tools/clinical-trials-monitor

Search and monitor ClinicalTrials.gov through its official API. No login. Export JSON/CSV/Excel. Pay only per result.

Jeremy Yarbrough

llms.txt Generator: Make Any Website LLM-Ready

f0rty7even/llms-txt-generator

Crawl a website and generate an llms.txt file (the emerging standard that helps LLMs and AI agents understand your site). Optionally build llms-full.txt with full page content. Output saved to the key-value store.