Robots Meta X-Robots Auditor

Store Positioning

Store title: Robots Meta X-Robots Auditor

Short description: Audit page-level robots directives from meta tags and X-Robots-Tag headers and detect conflicting indexability signals.

SEO title: Robots Meta X-Robots Auditor — technical SEO, web, and domain audit

SEO description: Audit page-level robots directives from meta tags and X-Robots-Tag headers and detect conflicting indexability signals. Use it to find crawlability, indexability, security, metadata, and page-quality issues with evidence-backed rows and audit reports.

Categories: SEO_TOOLS

Keywords: robots, meta, xrobots, auditor, web/domain audit

Pay-Per-Event Pricing

This actor uses pay-per-event pricing. Event prices include Apify platform usage; users are not expected to pay a separate platform-usage pass-through charge for the configured pricing model.

• Tier: W1 — Web/domain audit
• Primary event: page-audited at $0.00490 base
• Default max charge: $10.00
• Store discounts: FREE/BRONZE base, SILVER discounted, GOLD deepest approved discount

Event set:

• actor-start: base $0.00500, GOLD $0.00400. Robots Meta Xrobots Auditor: charged when actor start is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• page-audited: base $0.00490, GOLD $0.00392. Robots Meta Xrobots Auditor: charged when page audited is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• rule-parsed: base $0.00372, GOLD $0.00298. Robots Meta Xrobots Auditor: charged when rule parsed is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• finding-emitted: base $0.00372, GOLD $0.00298. Robots Meta Xrobots Auditor: charged when finding emitted is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
• audit-report-generated: base $0.05000, GOLD $0.04000. Robots Meta Xrobots Auditor: charged when audit report generated is completed. The price includes Apify platform usage; no separate usage pass-through is intended.

Public Task Concepts

• Audit Robots Meta X-Robots controls on a capped public sample
• Find high-priority Robots Meta X-Robots issues before release
• Validate Robots Meta X-Robots evidence from supplied pages
• Prioritize Robots Meta X-Robots fixes with severity and proof
• Export Robots Meta X-Robots QA rows for client review

Audit page-level robots directives from meta tags and X-Robots-Tag headers and detect conflicting indexability signals.

Inputs

This actor accepts the locked fields for actor 256: startUrls, sitemapUrls, htmlInputs, headerInputs, allowedDomains, maxPages, maxDepth, includeBotSpecificDirectives, includeHeaderDirectives, detectConflicts, requestDelayMs, timeoutMs. URL and sitemap inputs use production fetch paths; supplied HTML, JSON, header, or XML records are offline inputs for controlled runs.

Output

Dataset rows use the approved fields: sourceUrl, finalUrl, httpStatus, metaRobots, xRobotsTag, botName, indexDirective, followDirective, snippetDirective, archiveDirective, imagePreviewDirective, unavailableAfter, conflictDetected, issueCode, severity, recommendation, evidenceSnippet. The actor also writes these local report artifacts: robots-meta-xrobots-audit.md, robots-directive-results.csv, noindex-pages.csv, robots-conflicts.json, bot-specific-directives.csv.

Local run

npm test
APIFY_LOCAL_STORAGE_DIR=storage CRAWLEE_STORAGE_DIR=storage npm run smoke

Billing and cost controls

This actor uses pay-per-event pay-per-event pricing. It charges actor-start, then page-audited before each dataset row, and audit-report-generated before report artifacts. maxChargeUsd is enforced locally and in Apify runs; uncharged rows and reports are withheld. Platform usage pass-through remains off.