Get started
Solutions
Back
MCP server configuration
Configure your Apify MCP server with Actors and tools for seamless integration with MCP clients.
Start building
Apify for
Enterprise
Startups
Universities
Nonprofits
Use cases
Data for generative AI
Data for AI agents
Lead generation
Market research
View more →
Consulting
Professional services
Apify Partners
Product
Start here!
Ready-to-run tools for your AI agents and apps. Just pick one and go.
Browse 108,493 Actors
Apify platform
Apify Store
Actors for any job on the web
Actors
Build and run tools for AI agents and automation
Integrations
Connect with apps and services
MCP
Give your AI access to Actors
Anti-blocking
Scrape without getting blocked
Proxy
Rotate scraper IP addresses
Open source
Crawlee
Web scraping and crawling library
Developers
Documentation
Full reference for the Apify platform
Actor templates
Python, JavaScript, and TypeScript
Web scraping academy
Courses for beginners and experts
Monetize your code
Publish your Actors and get paid
Learn
API reference
CLI
SDK
Earn from your code
$1.7M paid out last month. Many developers earn over $3k.
Start earning now
Resources
Help and support
Advice and answers about Apify
Actor ideas
Get inspired to build Actors
Changelog
See what’s new on Apify
Customer stories
Find out how others use Apify
Company
About Apify
Contact us
Blog
Live events
Doing Good
Jobs
We're hiring!
Join our Discord
Talk to other builders
Pricing
Contact sales
Robots.txt Validator
Pay per usage
predictable_function/my-actor-3
List of website base URLs whose robots.txt files will be validated
Rating
5.0
(1)
Developer
riya rawat
Actor stats
0
Bookmarked
65
Total users
Monthly active users
9 months ago
Last modified
Categories
SEO tools
Share
apify/rag-web-browser
Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their content as clean Markdown for further processing by an LLM. Can also fetch individual URLs.
Apify
207K
4.7
(35)
predictable_function/my-actor-2
List of URLs whose HTTP redirect chains will be analyzed
71
compass/crawler-google-places
Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.
Compass
677K
(1,822)
predictable_function/website-uptime-response-monitor
Monitors website uptime and response time for given URLs.
66
(2)
gr_59017/website-title-heading-quality-checker
Analyzes website title tags and heading structure (H1–H6) to evaluate SEO quality, content hierarchy, and best practices. Detects issues like missing or multiple H1s, improper heading order, and suboptimal title length, and returns a quality score with suggestions.
Gautam Rana
73
eunit/sitemap-generator
Boost SEO with this automatic Sitemap Generator. Crawl any site to create XML, HTML, & TXT sitemaps. Supports custom depth, regex filters, & robots.txt. Compatible with Google Search Console.
Emmanuel Uchenna
36
lolletto/mermaid-diagram-actor
Convert Text description or Mermaid code to PNG, SVG, or PDF images. Flowcharts, sequence diagrams, Gantt charts, ER diagrams and more.
Lorenzo De Martinis
18
cerebral_aluminum/sitemap-extractor
Extract all URLs from website sitemaps. Pages, images, PDFs. Handles sitemap indexes and WordPress.
Benny
21
justa/llms-txt-file-generator
Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)
Benoit Eveillard
14
moving_beacon-owner1/my-actor-61
AI Resume Improver — Apify Actor Upload a resume (PDF, DOCX, or paste text) → get an ATS score (0-100), missing keywords, and actionable improvements tailored to 20+ job roles.
Jamshaid Arif
Description
Definition