Robots.txt & Security.txt Compliance Auditor

---

🎯 What this scrapes

Feed this Actor a list of sites and it fetches four well-known metadata files per site — robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt — and parses them into one structured audit row: crawl-policy rules, named AI-crawler posture (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider), security-contact/expiry data, and ads.txt record counts. Buyers are SEO/compliance teams, ad-ops teams vetting ad-tech supply chains, and publishers checking their own AI-bot policy — a recurring audit teams re-run on a schedule. No API key needed anywhere; all four paths are plain-text, no-auth files.

🔥 What we handle for you

• 🔍 Soft-404 detection — a custom-branded "page not found" that still answers HTTP 200 doesn't fool us; we check Content-Type and the file's own directive syntax before calling it present.
• 🤖 Named AI-crawler posture — one line per site tells you exactly whether GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and Bytespider are blocked, allowed, or simply never mentioned.
• 🔁 RFC 9116 well-known-then-legacy fallback — we check /.well-known/security.txt first and fall back to the legacy /security.txt path automatically.
• 🧱 Per-file fault isolation — one file missing or erroring never sabotages the other three; you get a full row with clear markers instead of a failed run.
• 🧊 Clean, typed dataset rows — Pydantic-validated, ISO-8601 timestamps, ready for CSV/JSON/Excel export.
• 💰 Pay-Per-Event pricing — you only pay for sites where at least one file resolved. No data, no charge beyond the warm-up fee.

💡 Use cases

• SEO / technical-SEO audits — check crawl-policy and AI-crawler exposure across your whole site portfolio in one run.
• Ad-tech / ad-ops compliance — validate ads.txt supply-chain records before flagging fraud risk.
• Security compliance — track vulnerability-disclosure contact and expiry dates across your infrastructure.
• Competitive / publisher research — see which AI crawlers competitors block or allow.

⚙️ How to use it

1. Click Try for free at the top of the page.
2. Paste in your list of sites (bare domain or full URL — both work).
3. Click Start. Output streams into the run's dataset.
4. Export from Storage → Dataset as JSON, CSV, or Excel — or fetch via the API.

📥 Input

Field	Type	Required	Default	Notes
sites	array	yes	—	Sites to audit for robots.txt, security.txt, ads.txt, and humans.txt. Bare domain or full URL — normalized to a lowercased host, www. preserved verbatim. 1–500 entries.
proxyConfiguration	object	no	{"useApifyProxy": true}	Apify Proxy configuration, wired per house standard. These four targets are plain-text, no-auth files.

Example input

{
  "sites": [
    "www.cloudflare.com",
    "github.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

📤 Output

Every row is one dataset item — one audited site.

Field	Type	Notes
site	string	Normalized host from input.
audited_at	string	ISO-8601 UTC timestamp this row was recorded.
robots_txt	object	Parsed robots.txt — exists, fetch_status, user_agent_groups, sitemaps, ai_crawler_posture.
security_txt	object	Parsed security.txt (RFC 9116) — exists, location, fetch_status, contacts, expires, expires_parsed, expires_in_past, policy_urls.
ads_txt	object	Parsed ads.txt — exists, fetch_status, record_count.
humans_txt	object	Parsed humans.txt — exists, fetch_status, excerpt (first 2000 chars).
ai_crawler_summary	object	Blocked / allowed / not-mentioned buckets + counts for the six named AI crawlers.
security_txt_expired	boolean	Mirrors security_txt.expires_in_past.
has_wildcard_disallow	boolean	True when User-agent: * disallows /.
fetch_errors	array	Subset of robots_txt / security_txt / ads_txt / humans_txt that hit a connection-level error.

Example output

{
  "site": "www.cloudflare.com",
  "audited_at": "2026-09-05T12:00:00Z",
  "robots_txt": {
    "exists": true,
    "fetch_status": "present",
    "user_agent_groups": [
      {"agent": "GPTBot", "allow": [], "disallow": ["/"], "crawl_delay": null}
    ],
    "sitemaps": ["https://www.cloudflare.com/sitemap.xml"],
    "ai_crawler_posture": {
      "GPTBot": "blocked",
      "ClaudeBot": "not_mentioned",
      "CCBot": "not_mentioned",
      "Google-Extended": "not_mentioned",
      "PerplexityBot": "not_mentioned",
      "Bytespider": "not_mentioned"
    }
  },
  "security_txt": {
    "exists": true,
    "location": "well_known",
    "fetch_status": "present",
    "contacts": ["mailto:security@cloudflare.com"],
    "expires": null,
    "expires_parsed": null,
    "expires_in_past": false,
    "policy_urls": []
  },
  "ads_txt": {"exists": false, "fetch_status": "absent", "record_count": 0},
  "humans_txt": {"exists": false, "fetch_status": "absent", "excerpt": null},
  "ai_crawler_summary": {
    "blocked": ["GPTBot"],
    "allowed": [],
    "not_mentioned": ["ClaudeBot", "CCBot", "Google-Extended", "PerplexityBot", "Bytespider"],
    "blocked_count": 1,
    "allowed_count": 0,
    "not_mentioned_count": 5
  },
  "security_txt_expired": false,
  "has_wildcard_disallow": false,
  "fetch_errors": []
}

💰 Pricing

Pay-Per-Event — you pay only when these events fire:

Event	USD	What it is
actor-start	$0.20	One-off warm-up charge per run
audit-completed	$0.003	Once per site audited — i.e. per ResultRow pushed to the dataset. Sites unreachable on all four files are never billed.

Example: 1 000 sites audited ≈ $3.20. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

🚧 Limitations

• No path-level Allow/Disallow matching against arbitrary customer URLs — only the raw rule dump and named-crawler posture.
• No parsing of non-standard robots.txt directives (Host:, Clean-param:, Noindex:).
• No security.txt PGP Encryption: fetch or signature verification.
• No following of Sitemap:, Policy:, or ads.txt owner/manager URLs — recorded as strings, never crawled.
• No plain-HTTP fallback — a host reachable only on port 80/no-TLS is treated as unreachable.
• No historical diffing/change-alerting between runs — each run is a stateless snapshot; schedule your own recurring runs to track change.
• Up to 500 sites per run — larger audits split across runs.

❓ FAQ

Is this legal?

Yes — all four files are plain-text, no-auth resources every public web server is expected to publish (RFC 9309 for robots.txt, RFC 9116 for security.txt, the IAB Tech Lab spec for ads.txt, and the humans.txt community convention). We only fetch what the target already serves to anyone who asks.

Why did a site come back with no fields populated?

A clean "file not found" is not a failure — it's a real answer. fetch_status reads "absent" for a file that genuinely doesn't exist and "present" for one that does; the row still gets pushed as long as at least one of the four files resolved.

What counts as "blocked" for an AI crawler?

A crawler is blocked when its User-agent group contains Disallow: /, allowed when it has a group with no blanket disallow, and not_mentioned when robots.txt never names it at all.

Does this follow the URLs it finds (sitemaps, policy pages, ads.txt owner domains)?

No — those are recorded as plain strings in the output, never crawled. This Actor audits the four files themselves, not what they link to.

💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an issue on the Actor's Issues tab on Apify Console — we ship fixes weekly and we read every report.

---