Pricing

from $10.00 / 1,000 domain audits

Try for free

Go to Apify Store

Robots.txt and AI Crawler Auditor

Try for free

Audit robots.txt, AI crawler access, sitemaps, llms.txt, security.txt and ads.txt for a list of domains.

Pricing

from $10.00 / 1,000 domain audits

Rating

0.0

(0)

Developer

Chris O'Byrne

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

SEO tools

Developer tools

Domains

domains

Required

Domains to audit, one per line, for example example.com. A full URL is accepted and reduced to its host. Up to 1,000 per run.

Type:array

Max. items:1000

Test path

testPath

Optional

The URL path that every bot is tested against, for example / for the whole site or /blog/my-post to test one page. A query string is allowed.

Type:string

Default:/

Crawl sitemaps in depth

deepSitemaps

Optional

Follow nested sitemap indexes and count URLs and the oldest and newest lastmod dates. Without it, the Actor only checks that each listed sitemap responds. Billed as a separate 'sitemap-deep' event per domain.

Type:boolean

Default:false

Maximum URLs counted per domain

maxSitemapUrls

Optional

Stop counting after this many sitemap URLs per domain. Applies only when deep sitemap crawling is on. The upper limit is 50,000.

Type:integer

Minimum:1

Maximum:50000

Default:50000

Maximum sitemap files per domain

maxSitemapFiles

Optional

Stop after reading this many sitemap files per domain. Applies only when deep sitemap crawling is on. The Actor makes at most one request per second to each host, so large numbers take longer.

Type:integer

Minimum:1

Maximum:200

Default:25

Extra user-agent tokens

customUserAgents

Optional

Optional. Additional robots.txt user-agent tokens to test, for example Applebot or Diffbot. Letters, digits, hyphens and underscores only. Up to 20.

Type:array

Max. items:20

Sitemap Detector

coder_zoro/sitemap-detector

Find sitemap URLs fast with our free Sitemap Finder tool. Instantly detect sitemaps from any website for SEO audits, indexing checks, and crawl planning. Improve visibility, site structure insights, and search engine performance in just seconds

Zoro

182

5.0

(3)

UI/UX Auditor

lenient_grove/ai-ux-auditor-apify

🚀 Ultimate AI based UI/UX Auditor for Website : Comprehensive analysis covering UX/UI, performance, accessibility, SEO, mobile, conversion & tech stack. AI-powered insights with 7 analysis types. Perfect for agencies & developers! #ai, #ux-analysis, seo, accessibility, #performance, #web-scraping

Tejas Rawool

1.9

(2)

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Locomint

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)

Benoit Eveillard

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

Ushba Khan

Robots.txt & AI Crawler Policy Auditor

merlin-hermes/robots-ai-crawler-policy-auditor

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.