Pricing

from $4.90 / 1,000 domain auditeds

Go to Apify Store

Robots.txt Auditor

Try for free

Fetch and audit robots.txt syntax, user-agent rules, blocked paths, sitemap declarations, and crawl risks.

Pricing

from $4.90 / 1,000 domain auditeds

Rating

0.0

(0)

Developer

junipr

Actor stats

Bookmarked

Total users

Monthly active users

16 hours ago

Last modified

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Robots.txt & Sitemap Auditor

phoenix2810/robots-sitemap-auditor

Audit a website's robots.txt directives and sitemap discovery in one fast API call. Detect blocked paths, missing sitemaps, and crawlability issues. Built for SEO agencies, technical SEO consultants, and site migration QA teams.

Sanskar Jaiswal

Sitemap & Robots SEO Index Auditor

glowing_glove/sitemap-robots-index-auditor

Audit robots.txt, sitemap discovery, indexability signals, canonical tags, and SEO crawl readiness for business websites.

Ushba Khan

SSL/TLS Certificate Expiry Monitor

gp005/tls-cert-monitor

Bulk-check TLS certificates for any HTTPS hosts: days-to-expiry, expiry alerts, and issuer/SAN/fingerprint change detection. Pure protocol, no scraping.

Geo

AI Readiness Auditor

rationalistic_counsel/ai-readiness-auditor

Check how AI-ready any website is. Get an AI Readiness Score (0-100) checking llms.txt, robots.txt AI crawler directives, Schema.org structured data, and meta tags. No API key needed.

J N

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Stas Persiianenko

Lighthouse Batch Auditor — Performance, SEO & Core Web Vitals

accurate_pouch/lighthouse-auditor

Batch Lighthouse audits via Google PageSpeed Insights API. Performance, SEO, accessibility scores + Core Web Vitals (LCP, FID, CLS) for 100s of URLs. 3 audits free per run.

Manchitt Sanan

GEO Auditor — AI Search Readiness & Citability Audit

foxlabs/geo-auditor

Audit how ready your site is to be found, read & cited by AI search (ChatGPT, Perplexity, Gemini, Claude). Checks AI-crawler access, structured data, content extractability, speed & trust — scored 0-100 with a prioritized fix list. GEO / AEO technical audit.

Berkan Kaplan

B2B Website Contact & SEO Lead Auditor

glowing_glove/b2b-website-contact-seo-lead-auditor

Audit business websites and extract public contact details, social profiles, lead forms, tech stack hints, and SEO sales signals.

Ushba Khan

5.0

(2)

Sitemap URL Discovery (sitemap.xml + robots.txt → all URLs)

gochujang/sitemap-url-discovery

Given a domain, finds sitemap.xml / sitemap_index.xml (also via robots.txt), recursively expands sitemap indexes, returns one row per discovered URL with lastmod / changefreq / priority. SEO audits, crawl-target prep, content cataloging. $0.0001/URL + $0.01 site fee.