Get started
Solutions
Back
MCP server configuration
Configure your Apify MCP server with Actors and tools for seamless integration with MCP clients.
Start building
Apify for
Enterprise
Startups
Universities
Nonprofits
Use cases
Data for generative AI
Data for AI agents
Lead generation
Market research
View more →
Consulting
Professional services
Apify Partners
Product
Start here!
Ready-to-run tools for your AI agents and apps. Just pick one and go.
Browse 127,734 Actors
Apify platform
Apify Store
Actors for any job on the web
Actors
Build and run tools for AI agents and automation
Integrations
Connect with apps and services
MCP
Give your AI access to Actors
Anti-blocking
Scrape without getting blocked
Proxy
Rotate scraper IP addresses
Open source
Crawlee
Web scraping and crawling library
Developers
Documentation
Full reference for the Apify platform
Actor templates
Python, JavaScript, and TypeScript
Web scraping academy
Courses for beginners and experts
Monetize your code
Publish your Actors and get paid
Learn
API reference
CLI
SDK
Earn from your code
$2M paid out last month. Many developers earn over $3k.
Start earning now
Resources
Help and support
Advice and answers about Apify
Actor ideas
Get inspired to build Actors
Changelog
See what’s new on Apify
Customer stories
Find out how others use Apify
Company
About Apify
Contact us
Blog
Live events
Doing Good
Jobs
We're hiring!
Join our Discord
Talk to other builders
Pricing
Contact sales
Robots Txt Inspector
$0.01 / actor start
eichan/robots-txt-inspector
Rating
0.0
(0)
Developer
Eichan Jolly
Actor stats
0
Bookmarked
2
Total users
1
Monthly active users
5 days ago
Last modified
Categories
Features
Share
apage/robots-txt-audit
Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.
Andy Page
14
visita/ip-whois-inspector
A fast, all-in-one tool to perform WHOIS, Reverse DNS, Geolocation, and Port Scanning for a list of IP addresses. It finds the ISP, network owner, hostname, geographic location, and open ports for any public IPv4 or IPv6 address.
Visita Intelligence
36
alizarin_refrigerator-owner/robots-txt-checker
The Robots.txt Checker provides comprehensive analysis of your robots.txt file: Syntax Validation CMS Detection - Identify WordPress, Shopify, Drupal,& 6+ other CMS platforms Best Practice Check Companion File Checks - sitemap.xml, llms.txt, security.txt AI Recommendations - CMS-specific suggestions
The Howlers
7
antonio_espresso/llms-txt-generator
Generates a clean and structured markdown format for your AI Agents LLMs.txt file. Helps define AI agent interactions, improve SEO, and ensure compatibility across platforms. Ready to integrate into your website for better transparency and control.
Antonio Blago
34
datawinder/robots-txt-monitor
Stateful robots.txt monitoring with baseline awareness and severity-classified alerts. Detects meaningful policy changes over time — not noisy diffs.
DatawinderLabs
3
perryay/robots-txt-sitemap-analyzer
Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.
Perry AY
bgfc97/robots-txt-sitemap-analyzer
Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.
Bruno
perryay/ssl-cert-inspector
Never get caught by an expired or misconfigured certificate. Inspect SSL/TLS certificates for any domain — issuer, subject, validity dates, SANs, key algorithm, chain length, and security ratings. Includes deep inspection for weak ciphers and outdated TLS. Batch analyze up to 20 domains.
eichan/seo-robots-txt-checker
eichan/sitemap-validator