Pricing

from $3.00 / 1,000 audit completeds

Try for free

Go to Apify Store

Robots.txt & Security.txt Compliance Auditor

Try for free

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

Pricing

from $3.00 / 1,000 audit completeds

Rating

0.0

(0)

Developer

DevilScrapes

Actor stats

Bookmarked

Total users

Monthly active users

21 days ago

Last modified

Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

arman-bd/website-tech-audit-scraper

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

Arman Hossain

Security.txt Checker — Security Contact Finder

q_services/security-txt-checker

Check websites for security.txt, extract security contacts, expiry, policy, hiring and compliance fields.

Q Services

Security.txt Checker

gifted_wagon/security-txt-evidence-auditor

Discover and audit RFC 9116 security.txt files with contact, expiry, canonical, policy, encryption, and line-level evidence.

Michael Olmos

security.txt Auditor

junipr/security-txt-auditor

Audit security.txt auditor inputs and return structured findings, evidence rows, and a buyer-ready report for SEO, developer, and operations teams.

junipr

AI Agent Readiness Audit — llms.txt, robots.txt & Schema

paoe/ai-agent-readiness-audit

Check whether AI agents can use a website: robots.txt AI-agent blocks, llms.txt, JSON-LD structured data, sitemap and summarisation signals.

Rashad Flet

robots.txt & llms.txt Monitor - Crawler & AI Access Rules

cuantic_data/robots-llms-txt-monitor

Fetches and parses a domain's robots.txt (RFC 9309) and llms.txt (the emerging AI-agent-friendly file), and flags what changed since the last run. Both files are public by design, meant to be read by automated clients - no login, no scraping of restricted content.

Cuantic Data

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

Security.txt Checker - Website Security Contacts

benthepythondev/security-txt-checker

Check websites for security.txt disclosure files and extract contacts, policies, encryption keys, expiry dates and raw security metadata.

Ben

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

AI Crawler Policy Checker: robots.txt & llms.txt

accountable_eel/ai-crawler-policy-lookup

Check what a website's robots.txt says about AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more) and whether it publishes an llms.txt. Reads the site's own public files directly: no API key, no third-party service. Charged only when an explicit AI policy or llms.txt is found.

Adrian Voss