Get started
Product
Back
Start here!
Get data with ready-made web scrapers for popular websites
Browse 22,669 Actors
Apify platform
Apify Store
Pre-built web scraping tools
Actors
Build and run serverless programs
Integrations
Connect with apps and services
MCP
Give your AI access to Actors
Anti-blocking
Scrape without getting blocked
Proxy
Rotate scraper IP addresses
Open source
Crawlee
Web scraping and crawling library
Solutions
MCP server configuration
Configure your Apify MCP server with Actors and tools for seamless integration with MCP clients.
Start building
Web data for
Enterprise
Startups
Universities
Nonprofits
Use cases
Data for generative AI
Data for AI agents
Lead generation
Market research
View more →
Consulting
Apify Professional Services
Apify Partners
Developers
Documentation
Full reference for the Apify platform
Code templates
Python, JavaScript, and TypeScript
Web scraping academy
Courses for beginners and experts
Monetize your code
Publish your scrapers and get paid
Learn
API reference
CLI
SDK
Earn from your code
$596k paid out in December. Many developers earn $3k+ every month.
Start earning now
Resources
Help and support
Advice and answers about Apify
Actor ideas
Get inspired to build Actors
Changelog
See what’s new on Apify
Customer stories
Find out how others use Apify
Company
About Apify
Contact us
Blog
Live events
Partners
Jobs
We're hiring!
Join our Discord
Talk to scraping experts
Pricing
Contact sales
Robots Txt Analyzer
from $10.00 / 1,000 results
urban_quidnunc/robots-txt-analyzer
Analyze robots.txt files for multiple domains
Rating
0.0
(0)
Developer
Donny
Actor stats
0
Bookmarked
1
Total users
Monthly active users
16 hours ago
Last modified
Categories
Other
Share
consummate_mandala/robots-txt-analyzer
Robots Txt Analyzer. Automated analysis with detailed reports and actionable insights. Fast, accurate, and scalable.
Donny Nguyen
webdatalabs/notion-marketplace-scraper
Scrape templates, categories, ratings, and creator profiles from Notion's official template marketplace. Perfect for competitive analysis, market research, creator monitoring, and discovering trending Notion templates.
WebDataLabs
15
5.0
(2)
codescraper/website-metadata-extractor
A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.
CodeScraper
71
(1)
sovereigntaylor/seo-analyzer-mcp
Ricardo Akiyoshi
lenient_grove/SiteScanner---Visual-Tech-Stack-Analyzer
Uncover any website's technology stack in seconds with a premium, enterprise-grade visual report.
Tejas Rawool
12
diao-bah-timbi/orphan-content-analyzer
Detect orphan and ghost pages that silently hurt your SEO. Crawl your website, analyze internal links, and instantly identify pages with zero inbound links. Ideal for SEO audits, content optimization, and large-scale website analysis.
Mamadou Diao Bah
2
quantifiable_bouquet/brand-dna
Extract structured brand identity from any website — colors, fonts, tone, positioning, and reusable marketing templates — using deterministic heuristics with no AI hallucinations.
Hayder Al-Khalissi
14
iamuendo/Bug-Bounty-Recon-Scanner
Find exposed admin panels, missing/weak security headers, sensitive file leaks, and HTTPS misconfigurations across target domains. Export prioritised risk scores and JSON reports. Run via API, schedule scans, or integrate with bug bounty tools.
Isaac Muendo
shahidirfan/Goodreads-Quotes
Goodreads Quotes Scraper: Effortlessly extract quotes, authors, and tags with this lightweight Goodreads Quotes Scraper. Designed for speed and reliability, it gathers wisdom from thousands of pages instantly. For the best performance, residential proxies are highly recommended.
Shahid Irfan
5
amernas/google-ads-transparency-analyzer
Google Ads Transparency Analyzer Apify actor that extracts competitor ad data from Google Ads Transparency Center. Features batch processing, FULL/LITE modes, date/region filtering, and proxy support. Automates competitive intelligence gathering for marketers and analysts. Outputs to Apify datasets.
Traffic Architect
166
1.6
Description
JSON example
Domains
domains
Required
List of domain names to analyze robots.txt for
Item