Pricing

Pay per usage

Robots & Sitemap Inspector

Fetches /robots.txt + sitemap.xml for any site, parses directives, validates structure, returns a structured SEO-health report with A-F grade. Catches missing sitemaps, disallow-all blunders, oversized robots files, and stale lastmod dates.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Coleton Patton

Actor stats

Bookmarked

Total users

Monthly active users

20 hours ago

Last modified

What it does

Fetches /robots.txt, parses every directive (User-agent groups, Allow/Disallow, Sitemap, Crawl-delay)
Walks linked sitemaps (or falls back to /sitemap.xml), up to 5 sub-sitemaps from a sitemap index
Counts URLs, tracks oldest + newest lastmod, flags health issues
Detects the catastrophic "disallow: /" that silently kills SEO

Input

{ "url": "https://example.com" }

Batch mode:

{ "urls": ["https://a.com", "https://b.com"] }

Output

{
  "url": "https://example.com",
  "grade": "B",
  "score": 75,
  "robots": {
    "found": true,
    "size": 1240,
    "userAgents": ["*", "Googlebot"],
    "sitemaps": ["https://example.com/sitemap.xml"],
    "crawlDelay": null,
    "disallowAll": false
  },
  "sitemaps": [
    { "url": "https://example.com/sitemap.xml", "found": true, "urlCount": 124, "type": "urlset", "lastmodCount": 124 }
  ],
  "summary": {
    "totalUrls": 124,
    "sitemapCount": 1,
    "oldestLastmod": "2024-03-15",
    "newestLastmod": "2026-06-17"
  },
  "issues": []
}

Use cases

SEO migration QA — verify sitemaps survived a CMS move
Crawl-budget triage — find sites whose robots.txt accidentally blocks Googlebot
Lead enrichment for SEO agencies — find prospects with broken sitemaps
Pre-launch checks — confirm staging robots.txt isn't pushed to prod

Pricing

Pay-per-result: $0.005/URL audited. Fetch-only, so this Actor is cheap to run — perfect for nightly audits across hundreds of sites.

Author

Built by Peak Post — 22+ Chrome extensions, 6+ Apify Actors. Part of an "SEO portfolio" that pairs with Meta Tag Inspector, Schema Validator, and Security Headers Scanner.

Broken Link Checker - Find 404s and Dead Links

santamaria-automations/broken-link-checker

Crawl any website and find broken links, 404 errors, redirect chains, timeouts, and SSL failures. Essential for SEO audits, QA, and content maintenance. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Ale

Sitemap URL Extractor

seemuapps/sitemap-extractor

Extract every URL from a website's sitemap.xml. Recursively walks nested sitemap indexes and returns loc, lastmod, changefreq, and priority for each page.

Andrew

Meta Tag Inspector

pattonholdings/meta-tag-inspector

Extracts SEO meta tags from any webpage: title, description, Open Graph (og:*), Twitter Cards, canonical URL, robots, viewport, charset, favicon. Returns a 0-100 SEO completeness score plus length warnings for title/description. Plain-fetch only — no browser overhead, fast and cheap.

Coleton Patton

Open Graph & Meta Tag Extractor

automation-lab/og-meta-extractor

This actor fetches any list of URLs and extracts all social media meta tags (Open Graph, Twitter Cards), SEO metadata (title, description, canonical, robots), structured data (JSON-LD), and internationalization (hreflang). Use it for social media audits, SEO analysis, link preview...

Stas Persiianenko

Company Tech Stack Detector

dash_authority/company-tech-stack-detector

Detect technologies used by websites including CMS, analytics, marketing tools, CDN, and hosting providers. Alternative to BuiltWith for competitive intelligence and sales prospecting.

Dash Authority

Site Governance Page Scraper

taroyamada/site-governance-monitor

Track deployment regressions and structural drift across your website. Automate robots.txt, sitemap, and schema checks immediately after code pushes.

naoki anzai

Sitemap Audit

apage/sitemap-audit

Get a Sitemap Health Score (0-100) for any website. Discover, parse, and validate XML sitemaps. Find 404s, redirects, canonical mismatches, noindex conflicts, hreflang issues, missing pages and estimate crawl budget waste.

Andy Page

Woorank SEO Auditor & Website Analyzer

dev00/woorank-seo-auditor

Comprehensive website auditor analyzing On-Page SEO, Mobile friendliness, structured schema validation, Core Web Vitals standing, security standard, classification tags and estimated traffic.

dev00

Robots.txt Validator - Crawl Rules Analyzer

pink_comic/robots-txt-validator

Analyze robots.txt files for any domain. Extract crawl rules, sitemaps, blocked paths, and crawl-delay settings. Validate configuration and identify SEO issues in bulk.

Ava Torres

Sitemap Extractor (urlset + sitemap index)

tom2turnt/sitemap-extractor

Parse a sitemap.xml (urlset) or a sitemap index into a clean URL list — loc, lastmod, changefreq, priority — with automatic index detection and de-duplication. Bulk URL discovery to feed downstream scrapers and directory builds. Structured output with ok/error parity.