# Robots.txt & Security.txt Compliance Auditor (`devilscrapes/robots-txt-security-txt-auditor`) Actor

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

- **URL**: https://apify.com/devilscrapes/robots-txt-security-txt-auditor.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<div align="center">
  <img src=".actor/icon.svg" width="160" alt="Devil Scrapes mark" />

## Robots.txt & Security.txt Compliance Auditor

**💰 $3.20 / 1 000 results**  ·  pay only for results  ·  no credit card to try

*We do the dirty work so your dataset stays clean.* 😈

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

</div>

***

### 🎯 What this scrapes

Feed this Actor a list of sites and it fetches four well-known metadata files per site — `robots.txt`, `security.txt` (RFC 9116), `ads.txt`, and `humans.txt` — and parses them into one structured audit row: crawl-policy rules, named AI-crawler posture (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider), security-contact/expiry data, and ads.txt record counts. Buyers are SEO/compliance teams, ad-ops teams vetting ad-tech supply chains, and publishers checking their own AI-bot policy — a recurring audit teams re-run on a schedule. No API key needed anywhere; all four paths are plain-text, no-auth files.

### 🔥 What we handle for you

- 🔍 **Soft-404 detection** — a custom-branded "page not found" that still answers HTTP 200 doesn't fool us; we check `Content-Type` and the file's own directive syntax before calling it present.
- 🤖 **Named AI-crawler posture** — one line per site tells you exactly whether GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and Bytespider are blocked, allowed, or simply never mentioned.
- 🔁 **RFC 9116 well-known-then-legacy fallback** — we check `/.well-known/security.txt` first and fall back to the legacy `/security.txt` path automatically.
- 🧱 **Per-file fault isolation** — one file missing or erroring never sabotages the other three; you get a full row with clear markers instead of a failed run.
- 🧊 **Clean, typed dataset rows** — Pydantic-validated, ISO-8601 timestamps, ready for CSV/JSON/Excel export.
- 💰 **Pay-Per-Event pricing** — you only pay for sites where at least one file resolved. No data, no charge beyond the warm-up fee.

### 💡 Use cases

- **SEO / technical-SEO audits** — check crawl-policy and AI-crawler exposure across your whole site portfolio in one run.
- **Ad-tech / ad-ops compliance** — validate `ads.txt` supply-chain records before flagging fraud risk.
- **Security compliance** — track vulnerability-disclosure contact and expiry dates across your infrastructure.
- **Competitive / publisher research** — see which AI crawlers competitors block or allow.

### ⚙️ How to use it

1. Click **Try for free** at the top of the page.
2. Paste in your list of sites (bare domain or full URL — both work).
3. Click **Start**. Output streams into the run's dataset.
4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or fetch via the API.

### 📥 Input

| Field | Type | Required | Default | Notes |
|---|---|:--:|---|---|
| `sites` | `array` | **yes** | — | Sites to audit for robots.txt, security.txt, ads.txt, and humans.txt. Bare domain or full URL — normalized to a lowercased host, `www.` preserved verbatim. 1–500 entries. |
| `proxyConfiguration` | `object` | no | `{"useApifyProxy": true}` | Apify Proxy configuration, wired per house standard. These four targets are plain-text, no-auth files. |

#### Example input

```json
{
  "sites": [
    "www.cloudflare.com",
    "github.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

### 📤 Output

Every row is one dataset item — one audited site.

| Field | Type | Notes |
|---|---|---|
| `site` | `string` | Normalized host from input. |
| `audited_at` | `string` | ISO-8601 UTC timestamp this row was recorded. |
| `robots_txt` | `object` | Parsed robots.txt — `exists`, `fetch_status`, `user_agent_groups`, `sitemaps`, `ai_crawler_posture`. |
| `security_txt` | `object` | Parsed security.txt (RFC 9116) — `exists`, `location`, `fetch_status`, `contacts`, `expires`, `expires_parsed`, `expires_in_past`, `policy_urls`. |
| `ads_txt` | `object` | Parsed ads.txt — `exists`, `fetch_status`, `record_count`. |
| `humans_txt` | `object` | Parsed humans.txt — `exists`, `fetch_status`, `excerpt` (first 2000 chars). |
| `ai_crawler_summary` | `object` | Blocked / allowed / not-mentioned buckets + counts for the six named AI crawlers. |
| `security_txt_expired` | `boolean` | Mirrors `security_txt.expires_in_past`. |
| `has_wildcard_disallow` | `boolean` | True when `User-agent: *` disallows `/`. |
| `fetch_errors` | `array` | Subset of `robots_txt` / `security_txt` / `ads_txt` / `humans_txt` that hit a connection-level error. |

#### Example output

```json
{
  "site": "www.cloudflare.com",
  "audited_at": "2026-09-05T12:00:00Z",
  "robots_txt": {
    "exists": true,
    "fetch_status": "present",
    "user_agent_groups": [
      {"agent": "GPTBot", "allow": [], "disallow": ["/"], "crawl_delay": null}
    ],
    "sitemaps": ["https://www.cloudflare.com/sitemap.xml"],
    "ai_crawler_posture": {
      "GPTBot": "blocked",
      "ClaudeBot": "not_mentioned",
      "CCBot": "not_mentioned",
      "Google-Extended": "not_mentioned",
      "PerplexityBot": "not_mentioned",
      "Bytespider": "not_mentioned"
    }
  },
  "security_txt": {
    "exists": true,
    "location": "well_known",
    "fetch_status": "present",
    "contacts": ["mailto:security@cloudflare.com"],
    "expires": null,
    "expires_parsed": null,
    "expires_in_past": false,
    "policy_urls": []
  },
  "ads_txt": {"exists": false, "fetch_status": "absent", "record_count": 0},
  "humans_txt": {"exists": false, "fetch_status": "absent", "excerpt": null},
  "ai_crawler_summary": {
    "blocked": ["GPTBot"],
    "allowed": [],
    "not_mentioned": ["ClaudeBot", "CCBot", "Google-Extended", "PerplexityBot", "Bytespider"],
    "blocked_count": 1,
    "allowed_count": 0,
    "not_mentioned_count": 5
  },
  "security_txt_expired": false,
  "has_wildcard_disallow": false,
  "fetch_errors": []
}
```

### 💰 Pricing

Pay-Per-Event — you pay only when these events fire:

| Event | USD | What it is |
|---|---:|---|
| `actor-start` | $0.20 | One-off warm-up charge per run |
| `audit-completed` | $0.003 | Once per site audited — i.e. per `ResultRow` pushed to the dataset. Sites unreachable on all four files are never billed. |

Example: 1 000 sites audited ≈ **$3.20**. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

### 🚧 Limitations

- No path-level Allow/Disallow matching against arbitrary customer URLs — only the raw rule dump and named-crawler posture.
- No parsing of non-standard robots.txt directives (`Host:`, `Clean-param:`, `Noindex:`).
- No security.txt PGP `Encryption:` fetch or signature verification.
- No following of `Sitemap:`, `Policy:`, or ads.txt owner/manager URLs — recorded as strings, never crawled.
- No plain-HTTP fallback — a host reachable only on port 80/no-TLS is treated as unreachable.
- No historical diffing/change-alerting between runs — each run is a stateless snapshot; schedule your own recurring runs to track change.
- Up to 500 sites per run — larger audits split across runs.

### ❓ FAQ

**Is this legal?**

Yes — all four files are plain-text, no-auth resources every public web server is expected to publish (RFC 9309 for robots.txt, RFC 9116 for security.txt, the IAB Tech Lab spec for ads.txt, and the humans.txt community convention). We only fetch what the target already serves to anyone who asks.

**Why did a site come back with no fields populated?**

A clean "file not found" is not a failure — it's a real answer. `fetch_status` reads `"absent"` for a file that genuinely doesn't exist and `"present"` for one that does; the row still gets pushed as long as at least one of the four files resolved.

**What counts as "blocked" for an AI crawler?**

A crawler is `blocked` when its `User-agent` group contains `Disallow: /`, `allowed` when it has a group with no blanket disallow, and `not_mentioned` when robots.txt never names it at all.

**Does this follow the URLs it finds (sitemaps, policy pages, ads.txt owner domains)?**

No — those are recorded as plain strings in the output, never crawled. This Actor audits the four files themselves, not what they link to.

### 💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an
issue on the Actor's **Issues** tab on Apify Console — we ship
fixes weekly and we read every report.

***

<div align="center">

Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of
opinionated public-data Actors. Honest pricing, real engineering, zero fine print.

</div>

# Actor input Schema

## `sites` (type: `array`):

Sites to audit for robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt. Bare domain or full URL — normalized to a lowercased host, with any <code>www.</code> prefix preserved verbatim.

## `proxyConfiguration` (type: `object`):

Apify Proxy configuration, wired per house standard. These four targets are plain-text, no-auth files with no anti-bot defenses expected — the default (non-residential) tier is sufficient.

## Actor input object example

```json
{
  "sites": [
    "www.cloudflare.com",
    "github.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

All dataset items as JSON.

## `datasetItemsCsv` (type: `string`):

Same data exported to CSV.

## `datasetView` (type: `string`):

Open the run dataset in the Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "www.cloudflare.com",
        "github.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/robots-txt-security-txt-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": [
        "www.cloudflare.com",
        "github.com",
    ],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/robots-txt-security-txt-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "www.cloudflare.com",
    "github.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call devilscrapes/robots-txt-security-txt-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devilscrapes/robots-txt-security-txt-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/b7amnMas4ZC3CzFCu/builds/rKtSacFWA5YM8YdIK/openapi.json