LLMs.txt Scraper - AI Documentation Index Data

Ads.txt Scraper - Publisher Ad Seller Data

benthepythondev/ads-txt-scraper

Scrape ads.txt files and extract authorized ad sellers, publisher IDs, relationship types and certification authority IDs.

App-ads.txt Scraper - Mobile Ad Seller Data

benthepythondev/app-ads-txt-scraper

Scrape app-ads.txt files and extract authorized mobile app ad sellers, publisher IDs and relationship metadata.

Humans.txt Scraper - Website Team Metadata

benthepythondev/humans-txt-scraper

Scrape humans.txt files and extract team, site, technology, contact and credit metadata from public websites.

Security.txt Checker - Website Security Contacts

benthepythondev/security-txt-checker

Check websites for security.txt disclosure files and extract contacts, policies, encryption keys, expiry dates and raw security metadata.

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

Facebook Group Posts & All Photos Scraper

spbotdel/facebook-group-posts-all-photos-scraper

Scrape public Facebook group posts with all recoverable photo URLs, including hidden +N grids. Get text, timestamps, authors, engagement, stable post URLs, and agent-ready JSON for Apify API, MCP, monitoring, and research workflows.

/llms.txt Generator

jakub.kopecky/llmstxt-generator

The /llms.txt Generator 🕸️📄 extracts website content to create an llms.txt file for AI apps 🤖✨ like LLM fine-tuning and indexing. Output is available 📥 in the Key-Value Store for easy download and integration into workflows. 🚀

Website Content Scraper

qaseemiqbal/website-content-scraper

Extract clean Markdown, plain text, linked files, and RAG-ready chunks from websites, documentation, help centers, knowledge bases, and authenticated portals. Preserve structure, metadata, URLs, and crawl context for AI search, training, and retrieval workflows.

Apify Actor Discoverability & SEO Audit

shelvick/apify-discoverability-audit

Audit an Apify Actor for discoverability by LLM agents and Apify Store ranking. Scores description quality, schema, disambiguation, agentic-pay, maintenance, and live agent-search retrieval across 10 dimensions, returning copy-paste-ready fixes so agents can find and call it as a tool.

FAQ & Help Center Scraper with Change Monitor

scrapepilot/faq

Scrape and monitor documentation, FAQs, and help centers effortlessly. Extract structured Q&As, body text, breadcrumbs, and metadata. Features automatic MD5 change detection to track page updates over time. Fast, async, and powered by cost-effective Pay Per Event billing