Pricing

from $1.30 / 1,000 robots.txt & sitemap audits

Try for free

Go to Apify Store

Robots.txt & XML Sitemap Deep Analyzer

Try for free

Parses robots.txt crawl rules and XML sitemaps. Detects disallowed paths, sitemap locations, AI web scraper permissions (GPTBot, ClaudeBot, CCBot), and index URL counts.

Pricing

from $1.30 / 1,000 robots.txt & sitemap audits

Rating

0.0

(0)

Developer

David Sandor

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Robots.txt & XML Sitemap Deep Analyzer 🚀

Parses robots.txt crawl rules and XML sitemaps. Detects disallowed paths, sitemap locations, AI web scraper permissions (GPTBot, ClaudeBot, CCBot), and index URL counts.

🌟 20+ Enterprise Enhancements (v2.0)

RAG & LLM Ready: Pre-computed OpenAI token counts and chunked embeddings.
Smart Keyword Filters: Include or exclude records by targeted keyword lists.
Sentiment Scoring: Built-in lexical sentiment rating on text contents.
Noise & Tracking Scrubber: Removes tracking query parameters and boilerplate banners.
Pay-Per-Event (PPE): Ultra-cost-effective pricing per extracted item.
Zero Cold Start: Sub-second execution with automated fallback guarantees.

💻 Integration Examples

Node.js (Apify Client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('robots-txt-sitemap-crawler').call({
    // Pass customized inputs here
    enableRagEnrichment: true
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log('Extracted Items:', items);

Python

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')
run = client.actor('robots-txt-sitemap-crawler').call(run_input={ 'enableRagEnrichment': True })
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

📄 Output Schema

Returns structured JSON, token counts, and RAG vector chunks.

Sitemap Indexability Auditor

nexgenwatch/sitemap-indexability-auditor

Audit whether the URLs in your sitemaps (or a plain URL list) are actually indexable — sitemaps are expanded (bounded nested depth + a hard URL cap) and every page URL gets one evidenced indexability verdict.

NexGen Watch

robots.txt & Sitemap Analyzer

bgfc97/robots-txt-sitemap-analyzer

Fetch and parse robots.txt for many domains: user-agent groups, allow/disallow rules, crawl-delay, declared sitemaps and whether crawlers are blocked. Technical SEO auditing. No key.

Bruno

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots.txt Auditor | Crawl Rules & Sitemap Discovery

sbio-apps/robots-txt-auditor

Read robots.txt for supplied websites and export user-agent groups, allow/disallow rules, sitemap declarations, extension directives and parsing warnings. Identify missing files and compare content hashes in your crawl configuration workflows.

Austin Aryain

Sitemap URL Extractor

datalantern/sitemap-extractor

Extract every URL from any website's XML sitemaps, including sitemap indexes, gzip sitemaps, and robots.txt discovery. Returns lastmod, changefreq, and priority.

Data Lantern

Robots.txt & Sitemap SEO Auditor

quanmatrix/robots-sitemap-seo-auditor

Audit robots.txt and XML sitemaps, discover crawl URLs, sample page availability, and flag technical SEO issues in structured output with a simple 0-100 readiness score.

Rafael Barreto Haddad

Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

chorelet/sitemap-url-extractor

Every URL of a website from its sitemaps: robots.txt discovery, sitemap indexes, gzip, lastmod, changefreq, priority, hreflang alternates, image/video/news extensions. Include/exclude patterns and date filter, CSV/JSON export and API.

Chorelet

Sitemap URL Extractor

palgenius/sitemap-url-extractor

Extract every URL from a website's sitemaps: indexes, .gz, plain-text and RSS/Atom, with lastmod filtering.

Ali Alsaidi

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.