Pricing

from $2.00 / 1,000 domain checkeds

Try for free

Go to Apify Store

AI Crawler Readiness: robots.txt & llms.txt

Try for free

See which AI crawlers a site's robots.txt allows or blocks (18 known tokens plus your own), with the deciding rule for each, and whether llms.txt exists and is well-formed. Identified checker, three requests per domain, refusals and errors are free.

Pricing

from $2.00 / 1,000 domain checkeds

Rating

0.0

(0)

Developer

kelp forest

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Sitemap Detector

coder_zoro/sitemap-detector

Find sitemap URLs fast with our free Sitemap Finder tool. Instantly detect sitemaps from any website for SEO audits, indexing checks, and crawl planning. Improve visibility, site structure insights, and search engine performance in just seconds

Zoro

182

5.0

(3)

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

(1)

AI Search Visibility Audit

responsible_bookmark/ai-search-visibility-audit

Audits websites for AI search readiness — scores how likely pages are to be cited by ChatGPT, Perplexity, Gemini, Claude, Copilot, Grok, and Google AI Overviews / AI Mode across 6 categories: crawlability, entity clarity, content structure, schema markup, authority signals, AI-specific optimisation.

Peter

Favicon & Brand Asset Checker

pillowy_travel/favicon-brand-asset-checker

Checks favicon presence and basic brand asset indicators on a website.

SAHIL KUMAR

Static SEO Audit - Score and Fixes for URL Lists

quillray/static-seo-audit

Audit a list of pages for 18 on-page SEO issues. Get a 0-100 score, a fix for every issue and what changed since your last audit. No browser, pay per page.

Quillray

Sitemap Generator

alizarin_refrigerator-owner/sitemap-generator

Generate XML sitemaps by crawling any website. Discover all pages, images, & videos with configurable crawl depth, URL filters, & multiple output formats. Full Site Crawling ,Image Sitemap, Video Sitemap, Multiple Output Formats, URL Filtering, Configurable Depth, Last Modified, Webhook Integration

The Howlers

SitePulse

upscale_audio/sitepulse

Instant SEO & technical audit for 100+ pages. Finds broken links, missing alts, slow pages + gives SEO score & fixes in JSON. Perfect for agencies & devs. Pay-per-use. Schedule weekly.

Astral Shard

llms.txt Generator & Checker - llms-full.txt, AI SEO (GEO)

tidytools/llms-txt-generator

Generate llms.txt and llms-full.txt for any website from its sitemap or a crawl, with sections and AI descriptions, or validate existing llms.txt files in bulk. $2 per 1,000 pages or sites.

Yukai Lin

Robots.txt Tester — Is a URL Allowed to Crawl?

hipersoft/robots-txt-checker

Test whether URLs are allowed or blocked by a site's robots.txt for any crawler user-agent. Returns allowed/blocked, the matched rule, crawl-delay and the site's sitemaps. Great for SEO audits and crawler compliance. One row per URL as JSON, CSV or Excel.