Pricing

from $15.00 / 1,000 ai crawler permission changes

AI Crawler & Content-Signal Permission Delta Monitor

Monitors any domain's robots.txt, Cloudflare Content-Signal line, and llms.txt for AI-crawler permission changes across tracked bot tokens (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more). Delta-only structured JSON output, free baseline, pay-per-event billing only for real changes.

Pricing

from $15.00 / 1,000 ai crawler permission changes

Rating

0.0

(0)

Developer

Stefano Seggio

Actor stats

Bookmarked

Total users

Monthly active users

23 days ago

Last modified

Website Content Crawler — Extract Full Site Content

oneary/website-content-crawler

🌐 Full website crawler that extracts structured content (text, headings, metadata, links, images) from any domain. Free platform compute pricing.

Luan M.

Entre Rios Argentina Licitaciones - Tender Delta API

stefano_seggio/entrerios-compras-monitor

Automated, schedule-based monitor for Entre Rios Province, Argentina's tender register (licitaciones) — Unidad Central de Contrataciones records verified live back to 2004. Detects and flags new listings, status changes, and closures as structured, delta-aware JSON records for compliance tracking.

Stefano Seggio

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

David

1.1K

3.0

(1)

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Locomint

Blog / Dated Content Crawler

diarmuidr/blog-content-crawler

Crawl an entire blog / knowledge base or filter to just the new content. Supporting relevant AI queries by filtering pages by date

Diarmuid

221

5.0

(1)

Stealth Website Crawler

nocturne/stealth-website-crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.

Nocturne

5.0

(1)

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

Tim Zinin

5.0

(6)

SEO Duplicate Content Detector

gr_59017/seo-duplicate-content-detector

Detects duplicate or identical content across multiple webpages by analyzing visible page text. Helps identify SEO duplicate content issues, content reuse, and potential ranking risks using simple content comparison and scoring.

Gautam Rana

No-BS Content Crawler 🖕

successful_nonagon/no-bs-content-crawler

Fast web crawler that extracts clean text from websites. Returns readable content, headings, and links. Perfect for content aggregation, SEO research, and data collection.

hafsah nuzhat

5.0

(4)

Ai Content Gap Agent

apilab/ai-content-gap-agent

An advanced SEO tool that identifies content gaps in search results using AI-powered analysis. This Apify Actor analyzes Google Search results, Reddit discussions, and competitor content to find opportunities for creating superior content that ranks better.