Pricing

from $3.50 / 1,000 url checkeds

Try for free

Go to Apify Store

AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked?

Try for free

Shows if GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Meta-ExternalAgent are allowed or blocked on each site in a dataset, CSV or Sheet: robots.txt, a real fetch per bot, llms.txt, plus change monitoring. Inputs: datasetId, fileUrl, urls. $0.005 per URL. Agent-ready: x402, MCP.

Pricing

from $3.50 / 1,000 url checkeds

Rating

0.0

(0)

Developer

Adam Pearce

Actor stats

Bookmarked

Total users

Monthly active users

7 hours ago

Last modified

AI Crawler Access Checker: GPTBot, ClaudeBot Blocked? API

Below, you can find a list of relevant HTTP API endpoints for calling the AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? Actor. For this, you’ll need an Apify account. Replace <YOUR_API_TOKEN> in the URLs with your Apify API token, which you can find under API & Integrations in Apify Console. For details, see the API reference.

Run Actor

POST
https://api.apify.com/v2/actors/nerolabs~ai-crawler-access-checker/runs?token=<YOUR_API_TOKEN>

Note: By adding the method=POST query parameter, this API endpoint can be called using a GET request and thus used in third-party webhooks. Please refer to our Run Actor API documentation.

Run Actor synchronously and get dataset items

POST
https://api.apify.com/v2/actors/nerolabs~ai-crawler-access-checker/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>

Note: This endpoint supports both POST and GET request methods. However, only the POST method allows you to pass input data. For more information, please refer to our Run Actor synchronously and get dataset items API documentation.

Get Actor

GET
https://api.apify.com/v2/actors/nerolabs~ai-crawler-access-checker?token=<YOUR_API_TOKEN>

For more information, please refer to our Get Actor API documentation.

Actors can be used to scrape web pages, extract data, or automate browser tasks. Use the AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? API programmatically via the Apify API.

You can choose from:

AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? API in Python

AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? API in JavaScript

AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? API through CLI

AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? OpenAPI definition

You can start AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? with the Apify API by sending an HTTP POST request to the Run Actorendpoint. An Actor’s input and its content type can be passed as a payload of the POST request, and additional options can be specified using URL query parameters. The AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? is identified within the API by its ID, which is the creator’s username and the name of the Actor.

When the AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? run finishes you can list the data from its default dataset(storage) via the API or you can preview the data directly on Apify Console.

Dataset Filter & Transform

nerolabs/dataset-filter-transform

Returns rows of any Apify dataset, CSV/Excel/JSON file URL or Google Sheet that pass filter rules (25 operators, AND/OR, date ranges that fix a scraper's ignored date filter) after transforms (rename, cast, compute, regex), then sort, dedupe, export, webhook. Agent-ready: pay per event (x402, MCP).

Adam Pearce

157

Bulk OCR: Image & Scanned PDF to Text from CSV or Google Sheet

nerolabs/bulk-ocr-image-pdf-to-text

OCR every image and scanned PDF linked in an Apify dataset, CSV, Excel or Google Sheet, keeping your columns. Inputs: datasetId or fileUrl or fileUrls, urlField, languages. Adds text, confidence and page counts. Tesseract inside the run, no outside service. Charged per page. Agent-ready: x402, MCP.

Adam Pearce

Dataset Aggregate, Group By & Pivot

nerolabs/dataset-aggregate-pivot

Returns GROUP BY and pivot tables for any Apify dataset, file or Google Sheet by URL, or JSON array: count, sum, average, min, max, median per group, date buckets, pivot columns. Exports CSV/Excel, appends to a named dataset, posts to a webhook. Agent-ready: pay per event (x402, MCP), per input row.

Adam Pearce

Web Accessibility Checker

louisdeconinck/web-accessibility-checker

Audit your entire website for WCAG 2.1 & 2.2 compliance using the industry-standard axe-core engine. Automatically crawl pages, detect accessibility violations, and generate detailed actionable reports to ensure legal compliance and improve SEO.

Louis Deconinck

SEO Checker

louisdeconinck/seo-checker

SEO Checker is an advanced Actor that performs comprehensive on-site SEO analysis for any website. It crawls web pages and extracts crucial SEO elements, providing detailed insights to help improve your website's search engine optimization.

Louis Deconinck

347

5.0

(2)

AI Brand Visibility Tracker — ChatGPT, LLM Search, Rank Monitor

wheaten_steelpan/ai-visibility-checker

Run brand prompts across ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews. Sample each prompt up to 3 times and report the spread, so you can tell a real mention from run-to-run noise. Returns mentions, cited sources, competitor share of voice and trends across runs.

Vanja V

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers

LLMs.txt Checker - AI Readiness Scanner

alizarin_refrigerator-owner/llms-txt-checker

Batch check websites for llms.txt files & identify AI optimization opportunities. A emerging standard that websites use to communicate with AI systems. It helps LLMs understand, What the company does, How to present information about the business, Contact information & key pages Brand guidelines

The Howlers

Plagiarism Checker with AI Precision

dev00/plagiarism-checker-with-ai-precision

Instantly check text for plagiarism and compare it against billions of web pages and academic sources. Returns comprehensive match percentages, semantic risk levels, and source links.

dev00

Monitoring Checker Run Status

apify/monitoring-checker-run-status

The monitoring checker run status is a part of the Apify Monitoring Suite (apify/monitoring). See its readme for more information and how to use this.