Pricing

from $3.50 / 1,000 site auditeds

Go to Apify Store

Robots.txt Evidence Auditor

Try for free

Parse robots.txt rules into normalized policy evidence, sitemap references, tested paths, and truthful failures.

Pricing

from $3.50 / 1,000 site auditeds

Rating

0.0

(0)

Developer

Michael Olmos

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Reliability and safety

Normalized user-agent groups.
Allow disallow crawl-delay and sitemap evidence.
Bounded path policy checks.
Safe redirect validation.
Explicit uncharged failures.
Inputs are deduplicated and bounded; the default memory is locked to 256 MB.

Charging

PAY_PER_EVENT uses a one-time Actor-start event plus site-audited. Only successful completed audit rows are charged. Invalid inputs, DNS/network/TLS failures, and the OUTPUT summary are not charged.

Example input

{
  "urls": [
    "https://www.google.com/"
  ],
  "userAgent": "*",
  "testPaths": [
    "/search",
    "/"
  ],
  "maxSites": 1,
  "timeoutSecs": 20
}

Output

The default dataset contains normalized evidence rows. The OUTPUT record reports completed, failed, duplicate, budget-limited, and input-limited counts without hiding partial failures.

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Tim Zinin

OAI AdsBot Landing Page Auditor

eternal_algorithms/oai-adsbot-landing-page-auditor

Audit public HTTPS landing-page reachability and declared robots policy for OAI-AdsBot, with structured evidence and explicit limitations. Unofficial independent tool; not affiliated with or endorsed by OpenAI.

Eternal Algorithms

AI SHOPPING READINESS AUDITOR — why agents skip your products

egeusta/ai-readiness-auditor

Audit product pages for AI shopping readiness. Score schema.org Product and Offer data, find missing fields with evidence, and see which AI crawlers robots.txt allows.

Ege

Sitemap Freshness & Schema Integrity Auditor

sebastian-actors/sitemap-freshness-schema-integrity-auditor

Audit sitemap lastmod, schema dates, and page freshness with static HTTP checks and deterministic issue codes.

Sebastián S

GEO Auditor — AI Search Readiness & Citability Audit

foxlabs/geo-auditor

Audit how ready your site is to be found, read & cited by AI search (ChatGPT, Perplexity, Gemini, Claude). Checks AI-crawler access, structured data, content extractability, speed & trust — scored 0-100 with a prioritized fix list. GEO / AEO technical audit.

Berkan Kaplan

RevenuePath Guard: Website Conversion Auditor

prolific_hoopoe/website-conversion-path-auditor

Detect candidate failures in public signup, demo, pricing, buy, and checkout paths. Get browser-rendered evidence without clicking or submitting forms.

Bassem Hakim

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

junipr

Website SEO, Health & Broken Link Auditor

fascinating_lentil/website-health-seo-broken-link-auditor

Audit authorized websites for technical SEO, broken links, redirects, sitemap and indexability issues, TLS, security headers, PageSpeed, and prior-run regressions. Returns per-page evidence plus HTML, PDF, JSON, and AI-ready summaries.

Md Jakaria Mirza

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.