Robots and Sitemap Peek

Expired Domains Scraper

ib4ngz/expired-domains-scraper

This actor scrapes expired domain data from expireddomains.net, supporting user authentication with optional multi-factor authentication (MFA). It also allows users to apply filters for targeted domain scraping.

Extract all URLs from website sitemaps. Pages, images, PDFs. Handles sitemap indexes and WordPress.

🔎 Crawl websites and turn technical SEO, accessibility, security, performance, broken-link, structured-data, robots.txt, llms.txt, and AI crawler signals into prioritized 0–100 audit scores and client-ready reports.

Sitemap Change Orchestrator

tri_angle/sitemap-change-orchestrator

Monitor website sitemaps for new, updated, or removed URLs. Integration with the Website Content Crawler (WCC) allows feeding only relevant URLs. This ensures your web crawls are efficient, targeted, and resource-optimized, keeping your datasets fresh for any application.

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

CMS Checker (Bulk)

canadesk/cms-checker-bulk

Get the technology stack and competitors for most websites. It's fast and costs little.

Company Reality Check

dravara/company-check

Enter a public company domain and supplied name. Get live-site evidence, discovered social links, and a confidence score—not legal verification. Run the prefilled example.

Competitor Snapshot

dravara/competitor-snapshot

Enter one public competitor homepage. Get JSON for its visible offer, pricing signals, calls to action, and claims without a full HTML dump or multi-page crawl. Run the prefilled example.

Date String Normalizer

dravara/date-normalize

Paste text containing a date. Get the first recognized date as ISO YYYY-MM-DD plus the original matched text. Unsupported or missing dates stay empty. Run the prefilled example.

Structured Summarizer

dravara/structured-summarize

Paste a document or long text and state what the next agent needs. Get a structured answer, key facts, unknowns, and quoted evidence. Run the prefilled example.