Pricing

Pay per usage

AI Crawler Log Analyzer — GPTBot, Claude & Perplexity

Analyze Nginx, Apache, Cloudflare, and JSON server logs for AI search, user-fetch, and training crawlers. Get bot traffic, errors, bandwidth, top pages, and actionable recommendations without exposing raw lines in dataset output.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Luigy Gabriel

Actor stats

Bookmarked

Total users

Monthly active users

7 hours ago

Last modified

AI Crawler Log Analyzer

Turn Nginx, Apache, Cloudflare, or JSON-line server logs into an AI-crawler visibility report. The Actor separates search/discovery bots from user-triggered fetchers and training/dataset crawlers, then reports response health, bandwidth, top pages, and recommendations.

What it detects

OpenAI: OAI-SearchBot, ChatGPT-User, and GPTBot
Anthropic: Claude-SearchBot, Claude-User, and ClaudeBot
Perplexity: PerplexityBot and Perplexity-User
Meta, Amazonbot, and Common Crawl tokens

Privacy

The raw logText input is processed inside the Actor run. Raw lines, IP addresses, and full user-agent strings are not written to the output dataset. Only aggregate metrics are returned.

Input

Paste combined-format access logs or one JSON object per line. For Cloudflare exports, the Actor recognizes ClientRequestUserAgent, ClientRequestURI, ClientRequestMethod, EdgeResponseStatus, and EdgeResponseBytes.

Output

Dataset item with aggregate metrics
REPORT key-value record with the complete JSON analysis
Bot breakdown, top paths, status codes, bandwidth, and prioritized recommendations

The Actor does not infer robots.txt permissions from traffic alone. Use the report to identify which policies and response paths need manual review.

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

AI Search Readiness Audit (AEO/GEO)

burly_bat/ai-search-readiness-audit

Bulk-audit URLs for AI search readiness (AEO/GEO). Checks 13 AI crawlers against robots.txt with quoted evidence, llms.txt, JSON-LD structured data, no-JS renderability and citation signals. PASS/WARN/FAIL + 0-100 score per URL and a client-ready HTML report. Input: URLs, sitemap or dataset.

Burly Bat

GEO Competitive Benchmark — AI Search Readiness vs Competitors

foxlabs/geo-benchmark

Benchmark your site vs up to 5 competitors for AI-search readiness in one run. See where rivals out-rank you for ChatGPT, Perplexity, Gemini & Claude citation — AI-crawler access, schema, extractability, speed, trust — as a ranked scoreboard + per-signal gap list. Deterministic GEO/AEO.

Berkan Kaplan

GEO Auditor — AI Search Readiness & Citability Audit

foxlabs/geo-auditor

Audit how ready your site is to be found, read & cited by AI search (ChatGPT, Perplexity, Gemini, Claude). Checks AI-crawler access, structured data, content extractability, speed & trust — scored 0-100 with a prioritized fix list. GEO / AEO technical audit.

Berkan Kaplan

LinkedIn Scraper: Posts, Reactions, Comments & Engagement

scrapio/linkedin-company-page-scraper

Scrapes full data from any public LinkedIn company page, including About info, employees, posts, insights, specialties, locations, size, and industry. Ideal for lead enrichment, competitor analysis, market research, and automated company intelligence at scale

Scrapio

100

5.0

(1)

Facebook Comments Scraper: Sentiment & Toxicity Analysis

simpleapi/facebook-comments-scraper

💬 Facebook Comments Scraper scrapes comments & replies from public posts — author names, text, timestamps, reactions & permalinks. 🔍 Clean, dedupe, and export to CSV/JSON. 📊 Ideal for social listening, sentiment, marketing, and research. 🚀 Fast, reliable, API-ready.

SimpleAPI

Twitter X Reply Scraper: Reply Sentiment & Tone Breakdown

scrapio/twitter-x-reply-scraper

💬 Twitter/X Reply Scraper extracts tweet replies at scale—usernames, text, timestamps, IDs & thread context. 🔎 Perfect for social listening, sentiment, competitor research & engagement tracking. ⚡ CSV/JSON export for dashboards, BI & automation. 🚀

Scrapio

Google Maps Scraper — Reviews & Sentiment Analysis

scraper-engine/google-maps-scraper

Scrape business listings from Google Maps to collect names, addresses, phone numbers, websites, and more. Perfect for generating local B2B leads or enriching your CRM with location-based contact data.

Scraper Engine

278

5.0

(4)

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.

Andy Page

AI Visibility Checker API

dev00/visibility-checker-api

Analyze domain visibility, brand authority, and search visibility across search engines and AI crawlers.

dev00

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.