Robots.txt Auditor

Store Positioning

Store title: Robots.txt Auditor

Short description: Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

SEO title: Robots.txt Auditor — technical SEO, web, and domain audit

SEO description: Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks. Use it to find crawlability, indexability, security, metadata, and page-quality issues with evidence-backed rows and audit reports.

Categories: SEO_TOOLS

Keywords: robots, txt, auditor, robots.txt, sitemap, web/domain audit

Fixed-Inclusive PPE Pricing

This actor uses pay-per-event pricing. Event prices include Apify platform usage; users are not expected to pay a separate platform-usage pass-through charge for the configured pricing model.

• Tier: W1 — Web/domain audit
• Primary event: robots-file-checked at $0.00490 base
• Default max charge: $10.00
• Store discounts: FREE/BRONZE base, SILVER discounted, GOLD deepest approved discount

Event set:

• actor-start: base $0.00500, GOLD $0.00400. Robots Txt Auditor: charged when actor start is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• robots-file-checked: base $0.00490, GOLD $0.00392. Robots Txt Auditor: charged when robots file checked is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• rule-parsed: base $0.00372, GOLD $0.00298. Robots Txt Auditor: charged when rule parsed is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• finding-emitted: base $0.00372, GOLD $0.00298. Robots Txt Auditor: charged when finding emitted is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• audit-report-generated: base $0.05000, GOLD $0.04000. Robots Txt Auditor: charged when audit report generated is completed. The price includes Apify platform usage; no separate usage pass-through is intended.

Public Task Concepts

• Audit public URLs for Robots.txt Auditor issues
• Find Robots.txt Auditor problems across a website
• Generate a technical Robots.txt Auditor report
• Audit robots issues across supplied public inputs
• Extract evidence-backed robots rows for client QA

What It Does

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

This local package is part of the Junipr Apify actors 171-200 premium build for ChatGPT review.

What It Does Not Do

• It does not call live Apify APIs.
• It does not publish, upload Store assets, create public tasks, or configure live PPE.
• It does not claim publish readiness.
• It does not make unbounded network calls in the supplied examples path.

Input Fields

Seed shape: domains or URLs, maxDomains, timeoutMs, strictMode, includeSummary

The tiny fixture is available at examples/input.tiny.json and mirrored at fixtures/input.tiny.json for the local runner.

Output Fields

• url
• robotsUrl
• statusCode
• userAgent
• directiveType
• directiveValue
• issueType
• severity
• evidence
• recommendation

Public Task Examples

Audit robots.txt directives for a domain

• Search intent: audit robots.txt directives for a domain
• Specific input: Fixture input shaped for audit robots.txt directives for a domain.
• Expected output: Dataset rows with url, robotsUrl, statusCode, userAgent plus local KVS report evidence.
• Why run it: SEO teams, agencies, web teams would run this to get a bounded local proof before any live Apify review or production use.

Find site-wide disallow rules before launch

• Search intent: find site-wide disallow rules before launch
• Specific input: Fixture input shaped for find site-wide disallow rules before launch.
• Expected output: Dataset rows with url, robotsUrl, statusCode, userAgent plus local KVS report evidence.
• Why run it: SEO teams, agencies, web teams would run this to get a bounded local proof before any live Apify review or production use.

Extract sitemap directives from robots.txt

• Search intent: extract sitemap directives from robots.txt
• Specific input: Fixture input shaped for extract sitemap directives from robots.txt.
• Expected output: Dataset rows with url, robotsUrl, statusCode, userAgent plus local KVS report evidence.
• Why run it: SEO teams, agencies, web teams would run this to get a bounded local proof before any live Apify review or production use.

Review crawler access rules for a client site

• Search intent: review crawler access rules for a client site
• Specific input: Fixture input shaped for review crawler access rules for a client site.
• Expected output: Dataset rows with url, robotsUrl, statusCode, userAgent plus local KVS report evidence.
• Why run it: SEO teams, agencies, web teams would run this to get a bounded local proof before any live Apify review or production use.

Create robots governance rows for SEO QA

• Search intent: create robots governance rows for seo qa
• Specific input: Fixture input shaped for create robots governance rows for seo qa.
• Expected output: Dataset rows with url, robotsUrl, statusCode, userAgent plus local KVS report evidence.
• Why run it: SEO teams, agencies, web teams would run this to get a bounded local proof before any live Apify review or production use.

Tests

• Actor-specific fixture tests live in test/robots-txt-auditor.test.ts.
• Node-based local suites validate dataset schema conformance, output samples, local smoke runs, and PPE guard behavior.

FAQ

Is this live on Apify?

No. This is a local package for ChatGPT review only.

Is it publish-ready?

No. It needs later live tiny-run validation and console configuration before any publication decision.