Pricing

Pay per event

Try for free

Go to Apify Store

Robots.txt & Security.txt Compliance Auditor

Try for free

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

Pricing

Pay per event

Rating

0.0

(0)

Developer

DevilScrapes

Actor stats

Bookmarked

Total users

Monthly active users

21 hours ago

Last modified

Categories

Developer tools

You can access the Robots.txt & Security.txt Compliance Auditor programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "sites": [
12        "www.cloudflare.com",
13        "github.com"
14    ],
15    "proxyConfiguration": {
16        "useApifyProxy": true
17    }
18};
19
20// Run the Actor and wait for it to finish
21const run = await client.actor("devilscrapes/robots-txt-security-txt-auditor").call(input);
22
23// Fetch and print Actor results from the run's dataset (if any)
24console.log('Results from dataset');
25console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
26const { items } = await client.dataset(run.defaultDatasetId).listItems();
27items.forEach((item) => {
28    console.dir(item);
29});
30
31// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Robots.txt Checker & Security.txt Checker Compliance Auditor API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Robots.txt & Security.txt Compliance Auditor API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Robots.txt & Security.txt Compliance Auditor API in Python

Robots.txt & Security.txt Compliance Auditor API through CLI

Robots.txt & Security.txt Compliance Auditor OpenAPI definition

Robots.txt & Security.txt Compliance Auditor API

Federal Website Compliance Auditor — GSA Site Scanning API

malonestar/gsa-site-scanning-auditor

Audit 29,000+ federal .gov websites via the GSA Site Scanning API. Per-agency compliance scorecards (USWDS, DAP, HTTPS/HSTS, IPv6, mobile, required links, Section 508), flat site exports and gap lists of sites missing a technology — with explicit coverage and per-scan status on every row.

Kyle Maloney

GEO & llms.txt Readiness Auditor

gp005/geo-llms-auditor

Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.

Geo

SEO Auditor

scrapers-hub/seo-auditor

SEO Auditor crawls a website and audits meta tags, headings, images, links, content, performance, schema, accessibility and technical health - with per-page scores and a ranked issue list. 🔍 Full technical SEO reporting.

Scrapers Hub

Technical On-Page SEO & Meta Auditor

gp005/seo-onpage-auditor

Audit any website's on-page SEO: titles, meta descriptions, H1 tags, canonical URLs, hreflang, Open Graph, Twitter Cards, JSON-LD schema, broken links, image alt text, and robots compliance. Single polite crawl, first-party domains only.

Geo

Public Security.txt Readiness Agent

jacksu/public-security-txt-readiness-agent

Check public security.txt readiness: contact, expiry, canonical, policy, encryption, risk labels, hashes, and useful-result pricing.

jack su

Website Metadata Extractor

scrapers-hub/website-metadata-extractor

Website metadata extractor to extract titles, descriptions, keywords, and meta tags from any website 🌐📊 Perfect for SEO analysis, auditing, and research. Fast, accurate, and scalable extraction.

Scrapers Hub

Find Sitemap From Url

scrapers-hub/find-sitemap-from-url

🔎 Find Sitemap From Url tool instantly locates a website’s sitemap from its URL, helping you discover indexable pages faster. ⚡ Perfect for SEO audits, crawling, indexing analysis & link research. 🚀 Boost efficiency with accurate sitemap detection.

Scrapers Hub

SEO & Accessibility Auditor Scraper

scrapers-hub/seo-accessibility-auditor

SEO & Accessibility Auditor crawls your site and scores meta tags, headings, content, technical health and schema, surfacing top issues, category averages and per-page results. ♿ Combined SEO and a11y reporting in a single run.

Scrapers Hub

UK Find a Tender Contracts Scraper

parseforge/ukcontracts-tenders-scraper

Scrapes UK public sector tender notices from Find a Tender. Returns each notice as a flat row with title, description, buyer, value, deadline, and award status. Filter by keyword and export up to 1,000,000 tenders.