Pricing

from $4.90 / 1,000 page auditeds

Go to Apify Store

AI Crawler Policy Auditor

Try for free

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

Pricing

from $4.90 / 1,000 page auditeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

Developer tools

Audit ID

auditId

Optional

Stable identifier for this AI crawler policy audit row.

Type:string

Source Type

sourceType

Optional

Whether the target was supplied inline or fetched from a URL.

Type:string

Source ID

sourceId

Optional

User-provided or generated target identifier.

Type:string

Source URL

sourceUrl

Optional

Source site URL when available.

Type:string | null

Robots URL

robotsUrl

Optional

Fetched robots.txt URL when available.

Type:string | null

Robots Status Code

robotsStatusCode

Optional

HTTP status code for optional robots.txt fetches.

Type:number | null

Target Path

targetPath

Optional

Path evaluated against robots allow/disallow rules.

Type:string

Status

status

Optional

Overall audit status: pass, warn, or fail.

Type:string

Severity

severity

Optional

Highest issue severity for this target.

Type:string

AI Crawler Readiness Score

aiCrawlerReadinessScore

Optional

0-100 score for alignment with the expected access policy.

Type:number

Expected Access

expectedAccess

Optional

Selected access goal: allow, block, or neutral.

Type:string

Policy Summary

policySummary

Optional

Concise human-readable policy summary.

Type:string

Audited Agents

auditedAgents

Optional

Crawler agents evaluated.

Type:array

Allowed Agents

allowedAgents

Optional

Audited agents allowed for the target path.

Type:array

Blocked Agents

blockedAgents

Optional

Audited agents blocked for the target path.

Type:array

Partially Blocked Agents

partiallyBlockedAgents

Optional

Audited agents allowed on the target path but blocked elsewhere.

Type:array

Unknown Agents

unknownAgents

Optional

Audited agents with no explicit or wildcard robots policy.

Type:array

Explicitly Configured Agents

explicitlyConfiguredAgents

Optional

Audited agents with explicit robots groups.

Type:array

Agent Policies

agentPolicies

Optional

Per-agent policy decisions and matched robots rules.

Type:array

Has Robots.txt

hasRobotsTxt

Optional

Whether robots.txt content was supplied or fetched.

Type:boolean

Robots.txt Bytes

robotsTxtBytes

Optional

UTF-8 byte length of robots.txt content.

Type:number

Robots Group Count

robotsGroupCount

Optional

Number of parsed robots user-agent groups.

Type:number

Has Sitemap

hasSitemap

Optional

Whether robots.txt advertises one or more sitemap URLs.

Type:boolean

Sitemap URLs

sitemapUrls

Optional

Sitemap URLs found in robots.txt.

Type:array

Has llms.txt

hasLlmsTxt

Optional

Whether llms.txt content was supplied or fetched.

Type:boolean

llms.txt Bytes

llmsTxtBytes

Optional

UTF-8 byte length of llms.txt content.

Type:number

X-Robots-Tag

xRobotsTag

Optional

X-Robots-Tag value when supplied.

Type:string | null

Meta Robots Directives

metaRobotsDirectives

Optional

Parsed meta robots directives from supplied HTML.

Type:array

Has Noindex Signal

hasNoindexSignal

Optional

Whether page-level robots directives include noindex or none.

Type:boolean

Has NoAI Signal

hasNoaiSignal

Optional

Whether page-level robots directives include noai or noimageai.

Type:boolean

Generic Allows All

genericAllowsAll

Optional

Whether wildcard robots policy allows the target path with no path blocks.

Type:boolean

Generic Blocks All

genericBlocksAll

Optional

Whether wildcard robots policy blocks the target path with Disallow: /.

Type:boolean

Blocks Common AI Crawlers

blocksCommonAiCrawlers

Optional

Whether common AI crawlers are blocked.

Type:boolean

Blocks Search Crawlers

blocksSearchCrawlers

Optional

Whether Googlebot or Bingbot are blocked.

Type:boolean

Mixed AI Crawler Policy

mixedAiCrawlerPolicy

Optional

Whether some audited crawlers are blocked while others are allowed.

Type:boolean

Mixed Policy Signals

mixedPolicySignals

Optional

Whether robots, page-level directives, and expected access conflict.

Type:boolean

Changed Robots Policy

changedRobotsPolicy

Optional

Whether blocked crawler agents changed versus previousRobotsTxt.

Type:boolean

Added Blocked Agents

addedBlockedAgents

Optional

Crawler agents newly blocked versus the previous robots snapshot.

Type:array

Removed Blocked Agents

removedBlockedAgents

Optional

Crawler agents no longer blocked versus the previous robots snapshot.

Type:array

Issue Count

issueCount

Optional

Number of issues found on this target.

Type:number

Issues

issues

Optional

Structured issue objects with code, severity, and message.

Type:array

Recommendation

recommendation

Optional

Recommended next action for this target.

Type:string

Checked At

checkedAt

Optional

ISO timestamp when this target was audited.

Type:string

Other properties may be included.

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

Sean Burke

LLMS.txt Auditor

junipr/llms-txt-auditor

Check for llms.txt / AI crawler policy guidance and produce structured readiness reports.

junipr

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

Ushba Khan

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.

Kunteper Koyu

Mexico RFC Validator | SAT Taxpayer ID Format Check

parseforge/mexico-rfc-scraper

Validate Mexican RFC taxpayer identifiers in bulk. Check format, embedded date, and homoclave checksum using the SAT modulo-11 algorithm for persona fisica and persona moral. Returns isValid plus per-check details. CSV, Excel, JSON, XML for KYC and CFDI invoice workflows.

ParseForge

Wayback Machine Scraper - Track Website Changes Over Time

ryanclinton/wayback-machine-search

Search the Internet Archive's Wayback Machine for historical snapshots of any website. Retrieve archived page metadata -- including timestamps, URLs, MIME types, HTTP status codes, and content hashes -- for up to 10,000 snapshots per run.

Ryan Clinton

107

SEC EDGAR Scraper

constructive_calm/sec-edgar-scraper

Fetches SEC EDGAR filings, normalized financial statements, XBRL facts, insider trades (Form 4), institutional holdings (13F), activist stakes (13D/G), full-text search hits, and parsed 8-K items via official SEC APIs. Zero anti-bot. Optional AI summaries.

Omar Eldeeb

5.0

(1)

LinkedIn Search Jobs Scraper

api-empire/linkedin-search-jobs-scraper

LinkedIn Search Jobs Scraper extracts job listings from any LinkedIn search query. Capture titles, companies, locations, descriptions, salaries, and posting dates. Ideal for market research, hiring, lead generation, and workflows needing structured LinkedIn job search data.

API Empire

Schema Markup Scraper & SEO Auditor

autofacts/metadata-scraper

Extract JSON-LD, Microdata, RDFa, Open Graph & Twitter Cards. Runs a 0-100 SEO audit — checks canonical, hreflang, headings, image alt, EEAT author signals. Detects 80+ schema.org types including LocalBusiness with NAP, geo coordinates, and Google Place IDs.

Richard Feng

158

5.0

(1)

Advanced SEO Audit Tool - AI-Powered Insights

ittechinnovators/advanced-seo-audit-tool---ai-powered-insights

Perform end-to-end SEO, performance, UX, social media, and payment gateway analysis. Generate actionable recommendations to enhance website visibility, user experience, and ranking. Optimized for all business types with AI-driven suggestions.