Katana Web Crawler (ProjectDiscovery)

ProjectDiscovery Notify

rl1987/pd-notify

Stream text or dataset records to Slack, Discord, Telegram, Email and other platforms using the ProjectDiscovery notify CLI.

Full Website HTML Crawler

kahan_anghan/full-website-html-crawler

A fast and efficient web scraping Actor that crawls websites and extracts structured page-level data. Supports configurable crawl limits and exports results in structured dataset format.

Validate a stream of JSONL/NDJSON records against a JSON Schema (draft-07 subset) one line at a time. Per-record error rows, drop-or-tag policy, summary by error type. Pairs with json-schema-validator + jsonl-to-csv.

Structured Data Crawler

tempting_district/structured-data-crawler

Crawl public web pages and convert unstructured content into clean, deterministic, schema-first structured records.

Paste a homepage or sitemap and get clean Markdown, metadata, JSONL chunks, and source URLs for RAG at a low per-page price.

Paste webpage and PDF URLs and get Markdown, JSONL chunks, PDF inventory, source warnings, and RAG-ready records.

Example Process Crawl Results

apify/example-process-crawl-results

Iterates through all results from a crawler run and count them. Needs to be called from the crawler's finish webhook by adding an URL to finish the webhook of your crawler. Use this actor as a starting point to develop custom post-processing of data from the crawler.

RAG Web Browser Scraper

datapilot/rag-web-browser-scraper

RAG Web Browser Search & Crawl Actor uses to search Bing or crawl URLs, then extracts page content as clean markdown. It captures title, description, language, HTTP status, and structured metadata. Supports multiple queries, proxies, and outputs organized crawl + search results.

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

AI Training Data Scraper - LLM and RAG-Ready

george.the.developer/ai-training-data-scraper

Extract web content formatted for LLM fine-tuning and RAG pipelines. Output in OpenAI JSONL, Claude JSONL, Markdown, or raw text.