Deprecated

Pricing

from $15.00 / 1,000 ai crawler permission changes

See alternative Actors

Go to Apify Store

AI Crawler Checker - robots.txt, llms.txt & Content Signals

Deprecated

See alternative Actors

Monitor any domain's robots.txt, Cloudflare Content-Signal line and llms.txt for AI-crawler permission changes across tracked bot tokens (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more). Delta-only structured JSON; the baseline audit is free and you pay per detected change.

Pricing

from $15.00 / 1,000 ai crawler permission changes

Rating

0.0

(0)

Developer

Stefano Seggio

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

AI Crawler Checker: robots.txt, llms.txt, Content-Signal API through CLI

The Apify CLI is the official tool that allows you to use AI Crawler Checker - robots.txt, llms.txt & Content Signals locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

AI Crawler Checker - robots.txt, llms.txt & Content Signals API in Python

AI Crawler Checker - robots.txt, llms.txt & Content Signals API in JavaScript

AI Crawler Checker - robots.txt, llms.txt & Content Signals OpenAPI definition

AI Crawler Checker - robots.txt, llms.txt & Content Signals API

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

Mikolabs

5.0

(1)

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

1.1K

3.0

(1)

Deep URL Content Crawler

6sigmag/deep-url-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple URLs simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

118

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

halam

575

3.7

(3)

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

Website Content Crawler | Markdown for RAG

xtech/news-source-crawler

Turn public news, blog, and documentation pages into clean Markdown or plain text. Discover pages through sitemaps and feeds, or extract known URLs directly. Export content, metadata, and change-tracking fields for RAG and monitoring, while respecting robots.txt and website boundaries.

Xtech

445

4.8

(3)

AI Brand Visibility Monitor - ChatGPT, Perplexity, Claude

chazee/ai-visibility-monitor

Do AI assistants recommend your brand? Measure mention rates, share of voice, sentiment, and which domains ChatGPT, Perplexity, Gemini & Google AI Overviews cite for your category. No API keys needed.

Chaz Eden

AI Content Detector 🔍

easyapi/ai-content-detector

🤖 Analyze text content to determine if it's AI-generated with high accuracy. Get detailed probability analysis and authoritative conclusions about content authenticity. Perfect for content verification, academic integrity, and digital publishing quality control.

EasyApi

Website Content Crawler Fast

timelody/website-content-crawler-fast

Scraping data from every single web page.

timelody

5.0

(1)

Website Content Crawler

worshipful_knife/website-content-crawler

Deep crawl websites and extract clean text, Markdown, or HTML for LLMs, RAG, and AI apps. Removes navigation, ads, cookie banners. Supports headless browser & HTTP. Sitemap discovery, URL scoping, file downloads. Feed ChatGPT, LangChain, LlamaIndex, Pinecone. The cheapest content crawler on Apify.