Pricing

from $2.00 / 1,000 audit scrapeds

Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

Pricing

from $2.00 / 1,000 audit scrapeds

Rating

0.0

(0)

Developer

Arman Hossain

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

Developer tools

SEO tools

MCP servers

Sites to audit

sites

Required

Domains or URLs. A bare domain ('nytimes.com'), a hostname with www, or a full page URL are all accepted. the path is stripped because every file audited here lives at the site root.

Type:array

Example:

[
  "nytimes.com",
  "https://www.shopify.com/"
]

Checks to run

checks

Optional

Which parts of the audit to perform. Fewer checks means fewer requests per site. 'meta', 'schema' and 'tech' share a single homepage fetch.

Type:string[]

Default:

[
  "robots",
  "sitemap",
  "ads",
  "security",
  "meta",
  "schema",
  "tech"
]

Expand sitemaps

expandSitemaps

Optional

Follow every discovered sitemap, walk sitemap indexes into their children, and count the URLs inside. Adds one request per sitemap file, so it is off by default.

Type:boolean

Default:false

Max URLs counted per site

maxUrlsPerSitemap

Optional

Stop counting sitemap URLs for a site once this many have been seen. Guards against news publishers with millions of URLs. Only applies when 'Expand sitemaps' is on.

Type:integer

Minimum:1

Default:1000

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

Ken Yao

Robots.txt & Sitemap Analyzer 🕷️

perryay/robots-txt-sitemap-analyzer

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.

Perry AY

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

Robots.txt Auditor

junipr/robots-txt-auditor

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

junipr

Robots.txt & Sitemap Discovery API

213x/robots-sitemap-discovery

Discover robots.txt rules, sitemap URLs, crawl delays, and basic crawlability signals for domains or website URLs.

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Robots.txt & Sitemap Auditor

phoenix2810/robots-sitemap-auditor

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Sanskar Jaiswal

Robots.txt Validator

predictable_function/my-actor-3

List of website base URLs whose robots.txt files will be validated

riya rawat

5.0

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

Robots.txt Checker - CMS-Aware Analysis with AI Recommendations

alizarin_refrigerator-owner/robots-txt-checker

The Robots.txt Checker provides comprehensive analysis of your robots.txt file: Syntax Validation CMS Detection - Identify WordPress, Shopify, Drupal,& 6+ other CMS platforms Best Practice Check Companion File Checks - sitemap.xml, llms.txt, security.txt AI Recommendations - CMS-specific suggestions