Pricing

Pay per usage

Try for free

Go to Apify Store

Robots.txt Analyzer

Try for free

Fetch and parse robots.txt for any site. Reports allowed/disallowed paths per user-agent, crawl-delay, sitemaps, and validates common issues.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

NICHOLAS SAVAGE

Actor stats

Bookmarked

Total users

Monthly active users

a month ago

Last modified

Use

Run this Actor with a JSON input matching .actor/input_schema.json. It writes structured results to the default dataset.

Input

urls (required): One or more website URLs to analyze (robots.txt will be fetched automatically)

Scope

This listing describes the checked-in Actor behavior and is bounded by the declared input and output schemas.

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

(1)

SEO Analyzer

miccho27/seo-analyzer

Analyze any webpage for SEO factors: title, meta, headings, images, links, Open Graph, structured data, and get an overall SEO score.

Tatsuya Mizuno

100

Website Sitemap Extractor

glassventures/website-sitemap-extractor

Extract all URLs from any website's sitemap. Auto-discovers sitemaps from robots.txt, supports sitemap index files and .gz compression. Filter by URL pattern, date range.

Glass Ventures

SEO Meta Tag Analyzer

constant_quadruped/seo-meta-analyzer

Audit any URL for SEO: title length, meta description, Open Graph tags, heading hierarchy, image alt texts, keyword density, mobile viewport, canonical URL, schema markup, and more.

LLMs.txt File Generator

justa/llms-txt-file-generator

Generate an llms.txt file from a website sitemap. Crawls all URLs, extracts titles and meta descriptions, and creates a Markdown-formatted file following the llms.txt specification. Upload then the output of your file directly on your website (Webflow, Wordpress etc.)

Benoit Eveillard

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt Checker & Parser - Crawl Rules API

pink_comic/robots-txt-validator

Check, parse, and validate robots.txt files in bulk. Extract crawl rules, sitemaps, crawl-delay, blocked paths, and per-user-agent allow/disallow results for SEO audits and crawler compliance.

Ava Torres

Orphan Content Analyzer

diao-bah-timbi/orphan-content-analyzer

Detect orphan and ghost pages that silently hurt your SEO. Crawl your website, analyze internal links, and instantly identify pages with zero inbound links. Ideal for SEO audits, content optimization, and large-scale website analysis.