Pricing

$5.00 / 1,000 processed results

Go to Apify Store

AI Training Blocker Checker

Try for free

Check robots.txt, llms.txt, sitemap hints, and AI crawler blocking rules for websites and domains.

Pricing

$5.00 / 1,000 processed results

Rating

0.0

(0)

Developer

Daniel Christensen

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Why use this Actor?

This API is built for fast, low-cost business intelligence on public web data. It uses raw HTTP by default, avoids browser/proxy costs, and returns productized fields like score, grade, summary, signals, issues, and recommendations.

Best for

SEO agencies building prospect lists
B2B lead generation teams
Content and ecommerce researchers
AI/RAG and automation workflows needing clean structured data

Input

Provide up to 10 public URLs/domains or the listed niche inputs. Runs are intentionally bounded for predictable cost.

{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}

Output

Each result includes the processed URL, HTTP status, score, grade, concise summary, niche-specific extracted fields, and practical recommendations.

Pricing

Pay per successful result. Designed to be lightweight and inexpensive to run.

Responsible use

Process only public pages you are allowed to analyze. This Actor does not bypass logins, CAPTCHAs, paywalls, or access controls.

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

jaehan byun

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

Ken Yao

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit

The Howlers

Robots.txt Checker - CMS-Aware Analysis with AI Recommendations

alizarin_refrigerator-owner/robots-txt-checker

The Robots.txt Checker provides comprehensive analysis of your robots.txt file: Syntax Validation CMS Detection - Identify WordPress, Shopify, Drupal,& 6+ other CMS platforms Best Practice Check Companion File Checks - sitemap.xml, llms.txt, security.txt AI Recommendations - CMS-specific suggestions

The Howlers

llms.txt Generator — Make Any Website AI-Readable

darknezz/llms-txt-generator

Crawl any website and generate llms.txt and llms-full.txt files following the llms.txt standard. BFS crawler with URL filtering and markdown/plaintext output. For AI readiness, SEO for AI, documentation portals, and archiving.

Oaida Adrian

AI Readiness Checker - Website Scanner

alizarin_refrigerator-owner/ai-readiness-checker

Analyze any website for AI optimization readiness. Check robots.txt, llms.txt, structured data, meta tags & content quality. Get actionable recommendations to improve AI crawler accessibility.

The Howlers