Pricing

from $0.0188 / robots.txt read

Robots and Sitemap Peek

Enter one public domain. Get its robots.txt sitemap URLs, user-agent groups, and disallow rules as JSON. This does not determine legal permission to crawl. Run the prefilled example.

Pricing

from $0.0188 / robots.txt read

Rating

0.0

(0)

Developer

Dravara LLC

Actor stats

Bookmarked

Total users

Monthly active users

2 months ago

Last modified

Expired Domains Scraper

ib4ngz/expired-domains-scraper

This actor scrapes expired domain data from expireddomains.net, supporting user authentication with optional multi-factor authentication (MFA). It also allows users to apply filters for targeted domain scraping.

Iqbal R

Sitemap Extractor

cerebral_aluminum/sitemap-extractor

Extract all URLs from website sitemaps. Pages, images, PDFs. Handles sitemap indexes and WordPress.

Benny

Website SEO, Accessibility & AI Readiness Auditor

feeng/website-seo-ai-readiness-auditor

🔎 Crawl websites and turn technical SEO, accessibility, security, performance, broken-link, structured-data, robots.txt, llms.txt, and AI crawler signals into prioritized 0–100 audit scores and client-ready reports.

Daniel S.

Sitemap Change Orchestrator

tri_angle/sitemap-change-orchestrator

Monitor website sitemaps for new, updated, or removed URLs. Integration with the Website Content Crawler (WCC) allows feeding only relevant URLs. This ensures your web crawls are efficient, targeted, and resource-optimized, keeping your datasets fresh for any application.

Tri⟁angle

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

CMS Checker (Bulk)

canadesk/cms-checker-bulk

Get the technology stack and competitors for most websites. It's fast and costs little.

Canadesk Support

Company Reality Check

dravara/company-check

Enter a public company domain and supplied name. Get live-site evidence, discovered social links, and a confidence score—not legal verification. Run the prefilled example.

Dravara LLC

Competitor Snapshot

dravara/competitor-snapshot

Enter one public competitor homepage. Get JSON for its visible offer, pricing signals, calls to action, and claims without a full HTML dump or multi-page crawl. Run the prefilled example.

Dravara LLC

Date String Normalizer

dravara/date-normalize

Paste text containing a date. Get the first recognized date as ISO YYYY-MM-DD plus the original matched text. Unsupported or missing dates stay empty. Run the prefilled example.

Dravara LLC

Structured Summarizer

dravara/structured-summarize

Paste a document or long text and state what the next agent needs. Get a structured answer, key facts, unknowns, and quoted evidence. Run the prefilled example.

Dravara LLC