Pricing

Pay per event

Public AI Crawler Policy Signal Agent

Analyze public robots.txt and llms.txt files for AI crawler allow/block policy evidence, LLM guidance files, stable hashes, and useful-result pricing.

Pricing

Pay per event

Rating

0.0

(0)

Developer

jack su

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

Business

Site or policy URLs

siteUrls

Required

Public HTTP/HTTPS site origins or direct robots.txt, llms.txt, or llms-full.txt URLs. Query parameters, fragments, credentials, path params, private-network hosts, sensitive account paths, and token-like path segments are rejected.

Type:string[]

Min. items:1

Max. items:20

Default:

[
  "https://openai.com/"
]

Previous AI crawler policy records

previousAiCrawlerPolicyRecords

Optional

Optional previous records from this Actor. If aiCrawlerPolicyHash is unchanged for the same site, the result is written without charging the useful event.

Type:object[]

Default:

[]

Check llms-full.txt

includeLlmsFullTxt

Optional

Also check /llms-full.txt. Disable this if you only want robots.txt and /llms.txt.

Type:boolean

Default:true

Request timeout

requestTimeoutSecs

Optional

Maximum time in seconds to wait for each public policy-file response.

Type:integer

Minimum:5

Maximum:30

Default:15

robots.txt Parser & AI Crawler Block Checker

taroyamada/robotstxt-ai-checker

robots.txt parser that audits AI crawler block rules (GPTBot, ClaudeBot, anthropic-ai, PerplexityBot) across thousands of websites in one run. Returns per-bot allow/disallow disposition and crawl-delay.

naoki anzai

Public Security.txt Readiness Agent

jacksu/public-security-txt-readiness-agent

Check public security.txt readiness: contact, expiry, canonical, policy, encryption, risk labels, hashes, and useful-result pricing.

jack su

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

The Howlers

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

Site QA Indexability AI Crawler Report Scraper

taroyamada/site-qa-indexability-ai-crawler-report-scraper

Unofficially audit user-supplied public pages, robots.txt, and llms.txt signals for AI crawler indexability issues and source-linked report rows.

naoki anzai

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

llms.txt + llms-full.txt Generator for Any Docs Site

ianymu/llms-txt-converter

llms-txt-converter is an Apify Actor that crawls any documentation site and outputs the two GEO-standard files (llms.txt and llms-full.txt) used by ChatGPT, Claude, Perplexity, Gemini, and Bing Copilot to find and cite your docs.

Yanlong Mu

LLMs.txt Generator

onescales/the-llms-txt-generator

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs