Get started
Solutions
Back
MCP server configuration
Configure your Apify MCP server with Actors and tools for seamless integration with MCP clients.
Start building
Apify for
Enterprise
Startups
Universities
Nonprofits
Use cases
Data for generative AI
Data for AI agents
Lead generation
Market research
View more →
Consulting
Professional services
Apify Partners
Product
Start here!
Ready-to-run tools for your AI agents and apps. Just pick one and go.
Browse 127,734 Actors
Apify platform
Apify Store
Actors for any job on the web
Actors
Build and run tools for AI agents and automation
Integrations
Connect with apps and services
MCP
Give your AI access to Actors
Anti-blocking
Scrape without getting blocked
Proxy
Rotate scraper IP addresses
Open source
Crawlee
Web scraping and crawling library
Developers
Documentation
Full reference for the Apify platform
Actor templates
Python, JavaScript, and TypeScript
Web scraping academy
Courses for beginners and experts
Monetize your code
Publish your Actors and get paid
Learn
API reference
CLI
SDK
Earn from your code
$2M paid out last month. Many developers earn over $3k.
Start earning now
Resources
Help and support
Advice and answers about Apify
Actor ideas
Get inspired to build Actors
Changelog
See what’s new on Apify
Customer stories
Find out how others use Apify
Company
About Apify
Contact us
Blog
Live events
Doing Good
Jobs
We're hiring!
Join our Discord
Talk to other builders
Pricing
Contact sales
Seo Robots Txt Checker
$0.01 / actor start
eichan/seo-robots-txt-checker
Rating
0.0
(0)
Developer
Eichan Jolly
Actor stats
0
Bookmarked
2
Total users
1
Monthly active users
10 days ago
Last modified
Categories
Features
Share
apage/robots-txt-audit
Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.
Andy Page
14
khadinakbar/website-seo-spider
Crawl public website pages and export structured SEO data. Review page URLs, titles, descriptions, headings, canonicals, links, and response details for page-level audits, migration checks, or downstream reporting.
Khadin Akbar
4
pillowy_travel/favicon-brand-asset-checker
Checks favicon presence and basic brand asset indicators on a website.
SAHIL KUMAR
67
dev00/mangools-advanced-seo-historical-backlink-checker
Instantly extract years of historical Majestic backlink data, Facebook share metrics, and global ranking history for any domain
dev00
15
glowing_glove/sitemap-robots-index-auditor
Audit robots.txt, sitemap discovery, indexability signals, canonical tags, and SEO crawl readiness for business websites.
Ushba Khan
3
maximedupre/seobility-seo-checker
Run Seobility SEO checks for public websites and page URLs. Get score, issue count, category scores, hints, factor checks, wiki links, and message evidence for exports, API workflows, and scheduled audits.
Maxime Dupré
bgfc97/robots-txt-sitemap-analyzer
Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.
Bruno
automation-lab/robots-sitemap-analyzer
This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...
Automation Lab
21
eichan/seo-meta-tags-extractor
Extract HTML title, description, robots and language metadata from up to five supplied pages.
eichan/seo-internal-link-map
Description
JSON example
Website URL or domain
siteUrl
Required
Website root or domain; always reads /robots.txt.
Named crawler
namedBot
Optional
Inspect this crawler separately from the wildcard group.