Robots Policy Auditor
Pricing
from $10.00 / 1,000 robots policy results
Go to Apify Store
Robots Policy Auditor
Audit robots.txt and crawl policy signals for domains before scraping, crawling, SEO checks, or data collection.
Audit robots.txt, sitemap declarations, crawl rules, and AI crawler policy signals.
What it does
This Actor audits supplied inputs and returns structured operational signals, warnings, a score, and supporting metadata. It is designed for teams that need repeatable checks before outreach, vendor review, monitoring, or data import.
Input
Provide a list of URLs, domains, or feed URLs depending on the Actor.
Output
Each input produces one dataset item with:
- input
- url/domain
- statusCode where applicable
- score
- signals
- warnings
- data
- checkedAt
Monetization
Recommended event: robots-audit-result. Charge per audited input.
