Robots.txt Policy Auditor

Store Positioning

Store title: Robots.txt Policy Auditor

Short description: Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks.

SEO title: Robots.txt Policy Auditor — technical SEO, web, and domain audit

SEO description: Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks. Use it to find crawlability, indexability, security, metadata, and page-quality issues with evidence-backed rows and audit reports.

Categories: SEO_TOOLS

Keywords: robots, txt, policy, auditor, robots.txt, sitemap, web/domain audit

Fixed-Inclusive PPE Pricing

This actor uses pay-per-event pricing. Event prices include Apify platform usage; users are not expected to pay a separate platform-usage pass-through charge for the configured pricing model.

• Tier: W1 — Web/domain audit
• Primary event: robots-file-checked at $0.00490 base
• Default max charge: $10.00
• Store discounts: FREE/BRONZE base, SILVER discounted, GOLD deepest approved discount

Event set:

• actor-start: base $0.00500, GOLD $0.00400. Robots Txt Policy Auditor: charged when actor start is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• robots-file-checked: base $0.00490, GOLD $0.00392. Robots Txt Policy Auditor: charged when robots file checked is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• rule-parsed: base $0.00372, GOLD $0.00298. Robots Txt Policy Auditor: charged when rule parsed is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• finding-emitted: base $0.00372, GOLD $0.00298. Robots Txt Policy Auditor: charged when finding emitted is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• audit-report-generated: base $0.05000, GOLD $0.04000. Robots Txt Policy Auditor: charged when audit report generated is completed. The price includes Apify platform usage; no separate usage pass-through is intended.

Public Task Concepts

• Audit Robots.txt Policy controls on a capped public sample
• Find high-priority Robots.txt Policy issues before release
• Validate Robots.txt Policy evidence from supplied pages
• Prioritize Robots.txt Policy fixes with severity and proof
• Export Robots.txt Policy QA rows for client review

Fetch and audit robots.txt files, parse crawl directives, test important paths, and report sitemap and policy risks.

Inputs

• siteOrigins
• robotsUrls
• rawRobotsTxtInputs
• importantPaths
• userAgents
• sitemapUrls
• allowedDomains
• testImportantPaths
• includeSitemapDiscovery
• includeUnknownDirectives
• requestDelayMs
• timeoutMs

Output

Dataset rows include: sourceUrl, siteOrigin, robotsStatus, userAgent, directiveType, directiveValue, lineNumber, appliesToPath, pathAllowed, matchedRule, sitemapUrl, crawlDelay, issueCode, severity, recommendation, evidenceSnippet.

The checked-in output example is produced by the deterministic inline robots.txt input and is verified against the dataset schema in automated tests.