Pricing

Pay per event

Public AI Crawler Policy Signal Agent

Analyze public robots.txt and llms.txt files for AI crawler allow/block policy evidence, LLM guidance files, stable hashes, and useful-result pricing.

Pricing

Pay per event

Rating

0.0

(0)

Developer

jack su

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

Business

Status

status

Required

Type:string

Input URL

inputUrl

Required

Type:string

Site origin

siteOriginUrl

Optional

Type:string

robots.txt

robotsUrl

Optional

Type:string

Robots status

robotsStatus

Optional

Type:string

llms.txt

llmsTxtUrl

Optional

Type:string

llms-full.txt

llmsFullTxtUrl

Optional

Type:string

LLMS status

llmsTxtStatus

Optional

Type:string

AI policy

aiPolicyStatus

Optional

Type:string

Policy type

policyType

Optional

Type:string

AI crawler rules

aiCrawlerPolicySignals

Optional

Type:object[]

AI rule count

aiCrawlerSignalCount

Optional

Type:integer

AI providers

knownAiProviders

Optional

Type:string[]

AI user agents

knownAiBotUserAgents

Optional

Type:string[]

Wildcard policy

wildcardRobotsPolicy

Optional

Type:object

LLMS files

llmsTxtSignals

Optional

Type:object[]

LLMS count

llmsTxtSignalCount

Optional

Type:integer

Attempts

attemptedUrls

Optional

Type:object[]

Evidence snippets

evidenceSnippets

Optional

Type:object[]

Evidence URLs

evidenceUrls

Optional

Type:string[]

Labels

riskLabels

Optional

Type:string[]

Diagnostics

diagnostics

Optional

Type:string[]

Readiness

readinessScore

Optional

Type:number

Confidence

confidenceScore

Optional

Type:number

Completeness

completenessScore

Optional

Type:number

Next action

recommendedNextAction

Optional

Type:string

Hash

aiCrawlerPolicyHash

Optional

Type:string

Previous hash

previousAiCrawlerPolicyHash

Optional

Type:string

Change

changeStatus

Optional

Type:string

Billable event

billableEventName

Optional

Type:string

Error

error

Optional

Type:string

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

robots.txt parser that audits AI crawler block rules (GPTBot, ClaudeBot, anthropic-ai, PerplexityBot) across thousands of websites in one run. Returns per-bot allow/disallow disposition and crawl-delay.

naoki anzai

Public Security.txt Readiness Agent

jacksu/public-security-txt-readiness-agent

Check public security.txt readiness: contact, expiry, canonical, policy, encryption, risk labels, hashes, and useful-result pricing.

jack su

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

The Howlers

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

Site QA Indexability AI Crawler Report Scraper

taroyamada/site-qa-indexability-ai-crawler-report-scraper

Unofficially audit user-supplied public pages, robots.txt, and llms.txt signals for AI crawler indexability issues and source-linked report rows.

naoki anzai

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

llms.txt + llms-full.txt Generator for Any Docs Site

ianymu/llms-txt-converter

llms-txt-converter is an Apify Actor that crawls any documentation site and outputs the two GEO-standard files (llms.txt and llms-full.txt) used by ChatGPT, Claude, Perplexity, Gemini, and Bing Copilot to find and cite your docs.

Yanlong Mu

LLMs.txt Generator

onescales/the-llms-txt-generator

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs