Pricing

Pay per usage

Store/Crawler Surface Preflight

Cross-check public robots.txt, sitemap, noindex, llms.txt, policy links, and contact signals from a storefront URL. Reads only public URLs; no account access.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Checksmith Cats

Actor stats

Bookmarked

Total users

Monthly active users

7 days ago

Last modified

Configure MCP server with Storefront Crawlability and Crawler Surface Preflight

Get a ready-to-use configuration for your MCP client with the Storefront Crawlability and Crawler Surface Preflight Actor preconfigured at mcp.apify.com?tools=checksmithcats/store-crawler-surface-preflight .

You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

LLMS.txt Auditor

junipr/llms-txt-auditor

Check for llms.txt / AI crawler policy guidance and produce structured readiness reports.

junipr

LLMs.txt Discovery

toronto_777/llms-txt-discovery

Find public llms.txt, llms-full.txt, and AI-readable documentation links for company domains.

Steven Feng

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

Ken Yao

AI Training Blocker Checker

bipbip-apis/ai-training-blocker-checker

Check robots.txt, llms.txt, sitemap hints, and AI crawler blocking rules for websites and domains.

Daniel Christensen

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

jaehan byun

Public AI Crawler Policy Signal Agent

jacksu/public-ai-crawler-policy-signal-agent

Analyze public robots.txt and llms.txt files for AI crawler allow/block policy evidence, LLM guidance files, stable hashes, and useful-result pricing.

jack su

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

LLMs.txt Checker — AI Readiness Monitor

q_services/llms-txt-checker

Check websites for llms.txt and llms-full.txt, extract sections and links for AI readiness and GEO audits.

Q Services

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.