Pricing

from $0.75 / 1,000 results

Site to LLM Knowledge Base

Turn any website or docs into clean, LLM-ready Markdown for RAG and AI agents — one record per page, each with a token count. Sitemap- and robots.txt-aware, with predictable per-page pricing (no token credits). Simple knowledge-base ingestion.

Pricing

from $0.75 / 1,000 results

Rating

0.0

(0)

Developer

Mohamed Adam BOUNHAR

Actor stats

Bookmarked

Total users

Monthly active users

13 hours ago

Last modified

Input

Field	Type	Notes
`startUrl`	string	Site/docs URL to crawl. Uses `sitemap.xml` if present, else same-domain links.
`maxPages`	integer	Hard cap on pages (default 25, max 500).
`respectRobots`	boolean	Default `true`. Disable only for sites you own.

Output (one dataset item per page)

{ "url": "https://site/docs/intro", "title": "Intro",
  "markdown": "# Intro...", "word_count": 540, "est_tokens": 720 }

Run locally

python scripts/new_actor.py --sync     # from repo root
cd actors/site-to-knowledge-base
apify run

Monetization

Pay-per-event, charging one page event per crawled page. See docs/pricing.md. Crawl logic is shared (shared/crawl.py) — edit there, then --sync.

Known limits

Fetches server-rendered HTML (no headless browser), so JavaScript-only pages return little content. A renderJs premium mode is the natural future upgrade.

AI Web Content Crawler - Markdown for LLMs

intelscrape/ai-web-content-crawler

Crawl any website and extract clean Markdown optimized for LLM training, RAG pipelines, and AI knowledge bases - removes boilerplate and outputs structured JSON with URL, title, markdown, and metadata.

IntelScrape

Website Content Crawler API - Markdown for RAG

tugelbay/website-content-crawler

Crawl public websites and extract clean Markdown, text, or HTML for RAG pipelines, AI agents, documentation indexing, and content monitoring. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler

Tugelbay Konabayev

Web Scraper For Llms

abotapi/web-scraper-for-llms

Stealth web scraping engine built for LLMs. Converts any web page to clean markdown or HTML

AbotAPI

Schema.Org Json Ld Extractor

sync-network/schema-org-json-ld-extractor

Extract Schema.org JSON-LD structured data from any website. Fast, lightweight HTTP-based scraper that pulls all JSON-LD scripts - perfect for SEO analysis, product data extraction, and AI/RAG pipelines. No browser overhead.

Alam

Web Page to Markdown Converter Pro — Clean Content Extraction

maged120/url-to-markdown-pro

Convert any web page to clean, formatted Markdown. Advanced content extraction handles complex layouts, paywalls, and JavaScript-rendered pages — strips ads and returns just the content.

Maged

Website Content Crawler — AI & RAG Ready

santamaria-automations/website-content-crawler

Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.

Ale

AI Dataset Converter - Website to Training Data

boztek-ltd/ai-dataset-converter

Crawl websites and convert content into AI-ready formats: RAG chunks, fine-tuning JSONL, Q&A pairs, clean Markdown. Token-aware chunking, quality scoring, deduplication. No external LLM API needed.

Boztek LTD

RAG-Ready Website Crawler

themineworks/rag-crawler

Pre-chunked markdown website crawler for RAG pipelines and LLM context. Only charges for pages that successfully crawl.

The Mine Works

Second Brain Builder — Karpathy's LLM Wiki, Automated

openclawai/second-brain-builder

Transform web content into an organized Obsidian-compatible knowledge base with AI-powered compilation. Scrapes any website and generates a Second Brain structure with wikilinks.

Pika Choo

Content Intelligence Extractor

werednalux/content-intelligence-extractor

Extract clean Markdown from Reddit threads and news sites. Built for LLM pipelines, n8n workflows, and AI content analysis. Uses Mozilla Readability + Reddit JSON API for noise-free output.