Pricing

from $0.60 / 1,000 saved observations

Try for free

Go to Apify Store

AI Crawler Policy Monitor - Robots.txt Evidence

Try for free

Compare robots.txt policies for named AI crawlers and paths. Export allow/disallow evaluations, source rules and saved policy changes.

Pricing

from $0.60 / 1,000 saved observations

Rating

0.0

(0)

Developer

Brandt May

Actor stats

Bookmarked

Total users

Monthly active users

13 days ago

Last modified

AI Crawler Policy Monitor - Robots.txt Evidence API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Policy Monitor - Robots.txt Evidence locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Policy Monitor - Robots.txt Evidence API in Python

AI Crawler Policy Monitor - Robots.txt Evidence API in JavaScript

AI Crawler Policy Monitor - Robots.txt Evidence OpenAPI definition

AI Crawler Policy Monitor - Robots.txt Evidence API

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Crawler Access Audit | Robots.txt & User-Agent Checks

a.klimm67/ai-crawler-reality-check

Compare robots.txt policy with live HTTPS responses for 11 named User-Agent strings, including GPTBot and ClaudeBot. Review possible blocks, llms.txt, and page extractability. Tests run from Apify, not verified crawler IPs; results cannot prove actual crawler access.

Andrii Klymenko

AI Crawler Policy Drift Monitor

checksmithcats/ai-crawler-policy-drift-monitor

Tracks changes in public robots.txt, noindex, sitemap, and llms.txt signals against a stored baseline. Public crawler-facing signals only.

Checksmith Cats

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Techionik

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

Ushba Khan

AI Toolchain Release Monitor - GitHub Release Changes

maydit/ai-toolchain-release-monitor

Monitor GitHub releases for AI toolchains: tags, prereleases, dates, asset metadata and edited notes. Compare saved releases without downloading binaries.

Brandt May

AI Model Price & Capability Monitor - Models.dev

maydit/ai-model-price-capability-monitor

Monitor Models.dev model pricing, context limits and capabilities by provider. Compare catalog observations with source attribution and structured exports.

Brandt May