Under maintenance

Pricing

from $0.50 / 1,000 results

Try for free

Go to Apify Store

Website To LLM Knowledge Pack

Under maintenance

Try for free

Crawl any website and turn it into an LLM-ready knowledge pack. This Actor extracts clean main text + metadata, follows links with depth/URL filters, and outputs per-page dataset items plus knowledge.jsonl, knowledge.md, and manifest.json for RAG/embeddings pipelines.

Pricing

from $0.50 / 1,000 results

Rating

0.0

(0)

Developer

M Junaid Shaukat

Actor stats

Bookmarked

Total users

Monthly active users

5 months ago

Last modified

Categories

Automation

Agents

You can access the Website To LLM Knowledge Pack programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        {
13            "url": "https://apify.com"
14        }
15    ]
16};
17
18// Run the Actor and wait for it to finish
19const run = await client.actor("attainable_iota/website-to-llm-knowledge-pack").call(input);
20
21// Fetch and print Actor results from the run's dataset (if any)
22console.log('Results from dataset');
23console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
24const { items } = await client.dataset(run.defaultDatasetId).listItems();
25items.forEach((item) => {
26    console.dir(item);
27});
28
29// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Website To LLM Knowledge Pack API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Website To LLM Knowledge Pack API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Website To LLM Knowledge Pack API in Python

Website To LLM Knowledge Pack API through CLI

Website To LLM Knowledge Pack OpenAPI definition

Website To LLM Knowledge Pack API

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

5.0

Website to Markdown Crawler â€” AI/RAG Data Pipeline

sovereigntaylor/website-to-markdown

Crawl any website and convert every page to clean, structured Markdown. Perfect for RAG pipelines, LLM training data, vector database ingestion, knowledge base building, and AI-powered search. Extracts main content, strips boilerplate, handles metadata, and chunks output for embeddings. Works with L

Ricardo Akiyoshi

RAG-Ready Web Scraper & Smart Chunker for AI Knowledge Bases

adinfosys-labs/rag-ready-web-scraper-smart-chunker-for-ai-knowledge-bases

RAG-ready web scraper that collects, cleans, deduplicates, filters, and chunks web content into structured datasets for AI pipelines. Generates high-quality knowledge-base data optimized for LLMs, embeddings, and vector databases

Artashes Arakelyan

GPT Crawler MCP — Knowledge files for ChatGPT, Claude, RAG

kazkn/gpt-crawler-mcp

Crawl any website and turn it into a clean knowledge file for your custom GPT, Claude Project, or RAG pipeline. Native MCP server in Standby mode + classic batch mode.

KazKN

github-knowledge-to-ai-markdown

angelbeats/github-knowledge-to-ai-markdown

于思远

Front Knowledge Base

canadesk/front-knowledge-base

Get Categories and Articles from any public Front Knowledge Base. It's fast and costs little.

Canadesk Support

Universal Knowledge Base Scraper (RAG Ready)

actums/universal-rag-scraper

Turn any Help Center into LLM-ready Markdown. Supports Zendesk, Intercom, Docusaurus, and generic sites. Perfect for RAG and AI Agents.

Actums

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!

ParseForge

Website Content Crawler

tugelbay/website-content-crawler

Website content crawler for RAG and LLM pipelines. Crawl entire websites and extract clean Markdown. HTTP-first: 10x faster than browser-based crawlers. Recursive crawling with depth control. Perfect for knowledge base ingestion, documentation scraping.

Tugelbay Konabayev

Deep Knowledge Mcp Server

eager_cornet/deep-knowledge-mcp-server

Deep Knowledge MCP Server gives LLMs real-time access to fresh technical docs using Google Search and web crawling. Instead of a basic scraper, it acts as a live knowledge engine for Claude or Cursor, letting developers instantly fetch and understand new libraries, and frameworks to code faster.