Robots Txt Inspector
Pricing
$0.01 / actor start
Go to Apify Store
Robots Txt Inspector
Pricing
$0.01 / actor start
userAgent
Optional
The user agent.
[]allowRules
Optional
The allow rules.
[]disallowRules
Optional
The disallow rules.
[]crawlDelaySeconds
Optional
The crawl delay seconds.
sitemaps
Optional
The sitemaps.
[]sourceUrl
Optional
The source url.
httpStatus
Optional
The http status.
otherDirectives
Optional
The other directives.
[]issues
Optional
The issues.
[]status
Optional
The status.
note
Optional
The note.
error
Optional
The error.
name
Optional
The name.
value
Optional
The value.