Duplicate & Near-Duplicate Content Checker

RAG Dataset Quality Auditor

gifted_wagon/rag-dataset-quality-auditor

Find duplicate, stale, incomplete, repetitive, and poorly chunked documents before they weaken RAG retrieval.

Image Similarity API | Compare Images, Find Duplicates

johnvc/image-similarity-api

Compare images by visual similarity with a CLIP vision model plus perceptual hashing. Score a source image against up to 500 targets, catch exact and near-duplicates, and get one clean row per comparison. URLs, base64, or file upload. Pay per result, MCP ready for AI agents.

Shopify SEO & Performance Auditor

trovevault/shopify-seo-performance-auditor

Audits public Shopify stores for prioritized technical SEO, canonical, duplicate-content, structured-data, image, Lighthouse, and third-party bloat findings.

Chinese AI Training Corpus Engine

zhorex/chinese-corpus-engine

Turn China's public web into AI-training-ready text. Pulls Weibo, Bilibili, Xueqiu, Douban & RedNote, then deduplicates, quality-scores, PII-scrubs and provenance-stamps every document. From $0.025/doc, pay-as-you-go. For LLM training-data teams, data vendors & academic NLP researchers.

Domain Health Checker

zinin/domain-health-checker

Audit DNS resolution, MX, SPF, DMARC, name servers, and TLS for a list of domains. Complete evidence cards include a 0–100 score, source states, issues, confidence, data gaps, and a next action. Incomplete or unsafe destinations remain unscored and are not billed as results.

SEO Service Area Visibility Checker

solutionssmart/seo-service-area-visibility

Checks how visible a business is in Google across multiple service areas and generates an SEO visibility report.

SEO Page Audit — Sitemap Crawl & On-Page Checker

insight.solutions/seo-page-audit

Point it at a domain. It reads robots.txt, walks the sitemap (gzip and nested indexes included) or crawls the site, then audits every page: title, description, H1s, canonical, hreflang, schema, images, links, speed. 23 issue codes, a transparent 0-100 score, optional broken-link check. No API key.

AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

locomint/ai-crawler-checker

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

Website Content Crawler – Markdown for AI, Emails & Contacts

locomint/website-crawler-content-contacts

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

AI Crawler & Content-Signal Permission Delta Monitor

stefano_seggio/ai-crawler-content-signal-permission-monitor

Watches any domain and reports the instant its AI-crawler permissions change: a robots.txt Allow/Disallow flip for GPTBot, ClaudeBot and 15 other tracked tokens, a Cloudflare Content-Signal flip, or llms.txt appearing/changing. Pay-per-event: billed only for what changed, with a free baseline audit.

RSS Feed Reader - Bulk RSS & Atom Feed Parser

logiover/bulk-rss-feed-reader

Read and parse RSS, Atom and RDF feeds in bulk, or auto-discover feeds from any website. Extract thousands of articles with full metadata for news monitoring, content aggregation and AI/RAG pipelines. No API key, export to CSV or JSON.