Pricing

from $2.64 / 1,000 audit completeds

Try for free

Go to Apify Store

Robots.txt Checker

Try for free

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

Pricing

from $2.64 / 1,000 audit completeds

Rating

0.0

(0)

Developer

DevilScrapes

Actor stats

Bookmarked

Total users

Monthly active users

17 hours ago

Last modified

Robots.txt Checker API through CLI

The Apify CLI is the official tool that allows you to use Robots.txt Checker locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Robots.txt Checker API in Python

Robots.txt Checker API in JavaScript

Robots.txt Checker OpenAPI definition

Robots.txt Checker API

Robots.txt Checker - CMS-Aware Analysis with AI Recommendations

alizarin_refrigerator-owner/robots-txt-checker

The Robots.txt Checker provides comprehensive analysis of your robots.txt file: Syntax Validation CMS Detection - Identify WordPress, Shopify, Drupal,& 6+ other CMS platforms Best Practice Check Companion File Checks - sitemap.xml, llms.txt, security.txt AI Recommendations - CMS-specific suggestions

The Howlers

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Security.txt Auditor

phoenix2810/security-txt-auditor

Audit a public website's security.txt vulnerability disclosure metadata. Checks discovery paths, required fields, expiration, contact methods, and returns a readiness score with recommendations.

Sanskar Jaiswal

Robots.txt Tester & Validator

gifted_wagon/robots-txt-evidence-auditor

Parse robots.txt rules into normalized policy evidence, sitemap references, tested paths, and truthful failures.

Michael Olmos

Ads.txt Scraper — Authorized Sellers Extractor

q_services/ads-txt-scraper

Extract authorized digital sellers from ads.txt and app-ads.txt files for publisher and adops audits.

Q Services

Image Alt Text Checker | Accessibility & SEO Auditor

andok/alt-text-auditor

Bulk audit websites for missing image alt text. Improve image SEO rankings and ensure web accessibility compliance (ADA/WCAG) with automated scanning.

Andok

Google AI Overview Citation Scraper

devilscrapes/ai-overview-citations

Scrape which domains Google's AI Overview cites for your target queries — one row per (query × cited source) with position, snippet, and selector telemetry — export to JSON or CSV. AEO / generative-SEO data for 2026, with captcha-aware retry and Pydantic-validated output. AI Overview API + monitor.