robots.txt Parser & AI Crawler Block Checker

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Website Content Extractor — Docs & Pages to Markdown

taroyamada/website-content-extractor

Extract website pages to Markdown. No API key / no login. Open Input → leave defaults → Start (1 URL ≈ USD 0.01/run): https://console.apify.com/actors/2SEQ0bWRQ39RCeI0P/input From USD 9/1k.

Wayback Machine Checker — Snapshot Diff Alerts

taroyamada/wayback-machine-checker

Compare Wayback snapshots and detect changes. No API key / no login. Open Input → leave defaults → Start (1 URL ≈ USD 0.008): https://console.apify.com/actors/fy1UnkQNIzGyy8HIs/input From USD 8/1k.

DMARC & Email Security Checker

taroyamada/dns-dmarc-security-checker

Validate email-domain infrastructure. No API key / no login. Open Input → leave defaults → Start (1 domain ≈ USD 0.008 total): https://console.apify.com/actors/PUumFXvaFfJrkaMF3/input

Bulk URL Health Checker — Status and Redirects

taroyamada/bulk-url-health-checker

Check HTTP status, redirects, and response time across URLs. No API key / no login. Open Input → leave defaults → Start: https://console.apify.com/actors/hHvfTzycQXbARYlZs/input

Short URL Resolver & Scraper

taroyamada/url-shortener-resolver

Resolve short links. No API key / no login. Open Input → leave defaults → Start (1 short URL ≈ USD 0.008/row): https://console.apify.com/actors/NHTWuZjw0n2kAQuOW/input From USD 8/1k.

Apple Podcasts Reviews Scraper — Ratings and Listener Feedback

taroyamada/apple-podcast-reviews-monitor

Apple Podcasts reviews from iTunes RSS. No API key / no login. Open Input → leave defaults → Start (1 show + 1 page): https://console.apify.com/actors/ulldRldNdzzThgg7J/input From $3 / 1,000 results.

Sitemap Analyzer — Nested XML URL Audit

taroyamada/sitemap-analyzer

Analyze XML sitemaps. No API key / no login. Open Input → leave defaults → Start (1 sitemap + maxUrls 3 ≈ USD 0.005 total): https://console.apify.com/actors/wJILhGwhAXWPfqBA2/input

Structured Data Scraper & Validator

taroyamada/structured-data-validator

Validate JSON-LD and Microdata on public pages. No API key / no login. Open Input → leave defaults → Start (1 URL ≈ USD 0.01 total): https://console.apify.com/actors/LupMh9Kdj6vgwlHMb/input

HHS Healthcare Data Breach Change Scraper

taroyamada/data-breach-disclosure-monitor

Track HHS OCR breach disclosures. No API key / no login. Open Input → leave defaults → Start (evidence window ≈ USD 0.01/row): https://console.apify.com/actors/iKd7RyZ5vXJgbj6V5/input From USD 10/1k.