Pricing

from $10.00 / 1,000 results

Go to Apify Store

Robots Txt Analyzer

Try for free

Analyze robots.txt files for multiple domains

Pricing

from $10.00 / 1,000 results

Rating

0.0

(0)

Developer

Donny

Actor stats

Bookmarked

Total users

Monthly active users

15 hours ago

Last modified

Robots.txt Analyzer

Fetch and analyze robots.txt files for multiple domains. Extracts user-agent rules, allowed/disallowed paths, sitemaps, and crawl delay settings.

Features

Bulk analysis for multiple domains
Parses all user-agent directives
Extracts allowed and disallowed paths
Identifies sitemap URLs
Reports crawl delay settings

Input

domains (required) - Array of domain names to analyze

Output

Each result contains:

domain - Domain name
userAgents - List of user-agent directives
allowedPaths - Allowed crawl paths
disallowedPaths - Disallowed crawl paths
sitemaps - Sitemap URLs found
crawlDelay - Crawl delay value

Robots Txt Analyzer

consummate_mandala/robots-txt-analyzer

Robots Txt Analyzer. Automated analysis with detailed reports and actionable insights. Fast, accurate, and scalable.

Donny Nguyen

Notion Marketplace Scraper

webdatalabs/notion-marketplace-scraper

Scrape templates, categories, ratings, and creator profiles from Notion's official template marketplace. Perfect for competitive analysis, market research, creator monitoring, and discovering trending Notion templates.

WebDataLabs

5.0

(2)

Website Metadata Extractor(sitemap, socialLinks, robotsTxt)

codescraper/website-metadata-extractor

A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.

CodeScraper

5.0

(1)

Seo Analyzer Mcp

sovereigntaylor/seo-analyzer-mcp

Ricardo Akiyoshi

SiteScanner - Visual Tech Stack Analyzer

lenient_grove/SiteScanner---Visual-Tech-Stack-Analyzer

Uncover any website's technology stack in seconds with a premium, enterprise-grade visual report.

Tejas Rawool

Orphan Content Analyzer

diao-bah-timbi/orphan-content-analyzer

Detect orphan and ghost pages that silently hurt your SEO. Crawl your website, analyze internal links, and instantly identify pages with zero inbound links. Ideal for SEO audits, content optimization, and large-scale website analysis.

Mamadou Diao Bah

Brand DNA

quantifiable_bouquet/brand-dna

Extract structured brand identity from any website — colors, fonts, tone, positioning, and reusable marketing templates — using deterministic heuristics with no AI hallucinations.

Hayder Al-Khalissi

Bug Bounty Recon Scanner

iamuendo/Bug-Bounty-Recon-Scanner

Find exposed admin panels, missing/weak security headers, sensitive file leaks, and HTTPS misconfigurations across target domains. Export prioritised risk scores and JSON reports. Run via API, schedule scans, or integrate with bug bounty tools.

Isaac Muendo

Goodreads Quotes

shahidirfan/Goodreads-Quotes

Goodreads Quotes Scraper: Effortlessly extract quotes, authors, and tags with this lightweight Goodreads Quotes Scraper. Designed for speed and reliability, it gathers wisdom from thousands of pages instantly. For the best performance, residential proxies are highly recommended.

Shahid Irfan

5.0

(1)

Google Ads Transparency Analyzer

amernas/google-ads-transparency-analyzer

Google Ads Transparency Analyzer Apify actor that extracts competitor ad data from Google Ads Transparency Center. Features batch processing, FULL/LITE modes, date/region filtering, and proxy support. Automates competitive intelligence gathering for marketers and analysts. Outputs to Apify datasets.