# Website to Knowledge Base (RAG / Chatbot Ready) (`codepal/website-to-knowledge-base`) Actor

Crawl a website and produce clean Markdown, semantic chunks, FAQs and an llms.txt for chatbot and RAG development.

- **URL**: https://apify.com/codepal/website-to-knowledge-base.md
- **Developed by:** [Jared Keanche](https://apify.com/codepal) (community)
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.01 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Knowledge Base — RAG Chunks, FAQs & llms.txt

Paste a URL and get a chatbot-ready knowledge base in one run: clean Markdown, embedding-ready chunks, FAQs, `llms.txt`, products and a site profile.

### Why use it

- **RAG-ready chunks:** heading-aware, with context headers, stable IDs and token counts. Never split inside links, tables or code.
- **Less noise:** removes menus, cookie banners, chat widgets, repeated blocks and near-duplicate pages.
- **Auto rendering:** fast HTTP first, a browser only for JavaScript or blocked pages.
- **Incremental mode:** scheduled runs output only new and changed pages and list removed ones. Unchanged pages are free.
- **FAQs & products** extracted and de-duplicated. Optional PDFs.
- **Optional AI (your Anthropic key):** summaries, key facts and a chatbot system prompt.

### Outputs

- **Dataset:** one row per page, or per chunk for vector-DB integrations (Pinecone, Qdrant…)
- `KNOWLEDGE_BASE.md`: upload to a Custom GPT or Claude Project
- `chunks.jsonl`, `faqs.jsonl`, `products.jsonl`, `llms.txt`, `llms-full.txt`
- `RUN_REPORT`: what was skipped and why, with hints

### Pricing

| Event | Per 1,000 |
|---|---|
| Page saved | $0.80 |
| + browser rendering | +$1.20 |
| PDF | $2.00 |
| AI enrichment | $1.00 |

Thin, duplicate, blocked and unchanged pages are free. Runs stop cleanly at your cost limit.

### Quick start

1. Add a homepage, docs or help-center URL.
2. Click **Start** (defaults suit most sites).
3. Download `KNOWLEDGE_BASE.md` or use the chunks dataset.

Only crawl sites you may crawl. robots.txt is respected by default.

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl begins: a homepage, docs root or help center. You can also paste a sitemap URL (e.g. https://example.com/sitemap.xml) to crawl exactly the pages it lists.

## `maxPages` (type: `integer`):

Hard limit on pages fetched. You only pay for pages saved to the knowledge base (thin, duplicate and blocked pages are free).

## `renderingType` (type: `string`):

<b>Auto</b> (recommended) fetches pages over fast HTTP and only opens a headless browser for pages that need JavaScript or block plain requests. <b>HTTP only</b> is fastest and cheapest. <b>Browser</b> renders every page.

## `crawlScope` (type: `string`):

Which links are followed. www/non-www and redirects are handled automatically.

## `maxDepth` (type: `integer`):

How many clicks away from the start URLs to go. 0 = only the start URLs.

## `includePdfs` (type: `boolean`):

Download linked PDFs (manuals, policies, brochures, price lists) and add their text to the knowledge base.

## `chunkSize` (type: `integer`):

Target size of each RAG chunk. 300–800 suits most embedding models. Small sections are merged instead of becoming fragment chunks.

## `chunkOverlap` (type: `integer`):

Context carried between consecutive chunks of the same section. Overlap is made of whole sentences, never cut words.

## `chunkLinks` (type: `string`):

<b>Text</b> keeps anchor text in the chunk and moves URLs into the chunk's <code>links</code> field — cleaner embeddings and fewer tokens. <b>Keep</b> leaves Markdown links in the text.

## `includeFaqChunks` (type: `boolean`):

Add each Q\&A pair found on the site as its own chunk. Question-shaped chunks match user questions very well.

## `datasetOutput` (type: `string`):

Choose <b>Chunks</b> when you connect this Actor to a vector-database integration (Pinecone, Qdrant, Weaviate, Milvus, Chroma, Supabase…), which reads the run's default dataset. The other type is always saved too (see OUTPUT).

## `incrementalMode` (type: `boolean`):

Remembers page fingerprints between runs. Unchanged pages are skipped and <b>not charged</b>; a CHANGES record lists new, changed and removed pages. Ideal for scheduled runs that keep a chatbot up to date.

## `incrementalStateKey` (type: `string`):

Optional. Runs with the same name share history. Defaults to the start URL's domain.

## `skipUnchangedPages` (type: `boolean`):

On: results contain only new and changed pages (delta upload). Off: everything is output, with a changeStatus field on each page.

## `useSitemap` (type: `boolean`):

Also queue URLs from the site's sitemaps (found via robots.txt) for better coverage of pages that aren't linked.

## `maxSitemapUrls` (type: `integer`):

Large sitemaps are sampled evenly down to this many URLs so one section (e.g. products) can't dominate.

## `includeUrlGlobs` (type: `array`):

Only follow URLs matching these globs, e.g. <code>https://example.com/docs/\*\*</code>. Empty = everything in scope.

## `excludeUrlGlobs` (type: `array`):

Never crawl URLs matching these globs, e.g. <code>**/tag/**</code>, <code>\*\*/page/\*</code>.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs the site disallows for crawlers. Turn off only for sites you own or have permission to crawl.

## `skipUtilityPages` (type: `boolean`):

Skip cart, checkout, login, account, wishlist and on-site search result pages.

## `allowedLanguages` (type: `array`):

Optional. Keep only pages whose language starts with one of these codes, e.g. <code>en</code>, <code>fr</code>. Pages without a declared language are kept.

## `ignoreQueryParams` (type: `array`):

Removed from URLs so the same page isn't crawled many times (utm\_\* is always removed).

## `urlLimits` (type: `array`):

Advanced. Cap any URL section, e.g. <code>\[{"pattern":"**/blog/**","maxPages":30}]</code>.

## `maxProductPages` (type: `integer`):

Caps individual product pages so help, policy and category pages are not crowded out on large stores.

## `followLinksFromProducts` (type: `boolean`):

Off by default: product pages mostly link to other products.

## `contentSelector` (type: `string`):

Optional. Force the content area (e.g. <code>article.post</code>). Empty = auto-detect.

## `removeSelectors` (type: `array`):

Additional CSS selectors to strip (popups, related-posts widgets…).

## `minWords` (type: `integer`):

Thinner pages are skipped (and not charged).

## `removeDuplicateBlocks` (type: `boolean`):

A paragraph that repeats on many pages (promo strips, shared disclaimers, product cards) is kept in the chunks only once.

## `skipNearDuplicates` (type: `boolean`):

Skip pages that are almost identical to one already saved (print versions, tracking variants, mirrored pages).

## `enableAIEnrichment` (type: `boolean`):

Adds per-page summaries, key facts, topics, entities and FAQs, plus a site overview and a ready-to-use chatbot system prompt. Uses your own Anthropic API key (you pay Anthropic directly at their prices).

## `anthropicApiKey` (type: `string`):

Required only when AI enrichment is on. Stored encrypted.

## `aiModel` (type: `string`):

Model ID, e.g. <code>claude-haiku-5-5</code> (fast and cheapest, recommended) or <code>claude-sonnet-5-5</code> (higher quality).

## `aiMaxPages` (type: `integer`):

Cost control. Help, policy and info pages are prioritized; products get at most half of this budget.

## `aiConcurrency` (type: `integer`):

Lower it if your Anthropic account hits rate limits.

## `dynamicContentWaitSecs` (type: `integer`):

Browser only. Max time to wait for the network to go idle before reading the page.

## `waitForSelector` (type: `string`):

Browser only. Optional CSS selector to wait for before extracting (e.g. <code>.article-body</code>).

## `expandClickableElements` (type: `boolean`):

Browser only. Opens collapsed sections (<code>\[aria-expanded=false]</code>, <code>\<details></code>) so hidden FAQ answers are captured.

## `maxScrollHeightPixels` (type: `integer`):

Browser only. Scrolls to load lazy content. 0 disables scrolling.

## `proxyConfiguration` (type: `object`):

Datacenter proxy is enough for most sites. Use residential only for sites that block crawlers.

## `initialCookies` (type: `array`):

Crawl pages behind a login (only sites you are allowed to access). JSON list like <code>\[{"name":"session","value":"…","domain":".example.com"}]</code>.

## `customHttpHeaders` (type: `object`):

Extra headers for every request, e.g. <code>{"Authorization": "Bearer …"}</code>.

## `maxConcurrency` (type: `integer`):

Max parallel requests (scales automatically with available memory). Browser pages use at most 20.

## `maxRequestsPerMinute` (type: `integer`):

Be polite to small sites. 0 = no limit.

## `requestTimeoutSecs` (type: `integer`):

Max time to load one page before retrying.

## `maxRequestRetries` (type: `integer`):

Retries on network errors and blocks (with a new session/IP each time).

## `ignoreHttpsErrors` (type: `boolean`):

Accept sites with invalid or self-signed certificates (HTTP mode).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "maxPages": 25,
  "renderingType": "auto",
  "crawlScope": "same-hostname",
  "maxDepth": 3,
  "includePdfs": false,
  "chunkSize": 500,
  "chunkOverlap": 60,
  "chunkLinks": "text",
  "includeFaqChunks": true,
  "datasetOutput": "pages",
  "incrementalMode": false,
  "skipUnchangedPages": true,
  "useSitemap": true,
  "maxSitemapUrls": 300,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "respectRobotsTxt": true,
  "skipUtilityPages": true,
  "allowedLanguages": [],
  "ignoreQueryParams": [
    "sort",
    "ref",
    "source",
    "fbclid",
    "gclid",
    "tag",
    "tab"
  ],
  "urlLimits": [],
  "maxProductPages": 100,
  "followLinksFromProducts": false,
  "removeSelectors": [],
  "minWords": 20,
  "removeDuplicateBlocks": true,
  "skipNearDuplicates": true,
  "enableAIEnrichment": false,
  "aiModel": "claude-haiku-5-5",
  "aiMaxPages": 100,
  "aiConcurrency": 4,
  "dynamicContentWaitSecs": 5,
  "expandClickableElements": true,
  "maxScrollHeightPixels": 5000,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "initialCookies": [],
  "customHttpHeaders": {},
  "maxConcurrency": 50,
  "maxRequestsPerMinute": 0,
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 3,
  "ignoreHttpsErrors": false
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset: one row per page, or one row per RAG chunk when 'Default dataset contains' = Chunks.

## `knowledgeBaseMarkdown` (type: `string`):

Whole site in one clean Markdown file for Custom GPTs, Claude Projects and other LLM tools.

## `chunks` (type: `string`):

Embedding-ready chunks with contextual headers and metadata, one JSON object per line.

## `llmsTxt` (type: `string`):

Site index following the llms.txt convention.

## `summary` (type: `string`):

Run summary, change report and links to every output file.

## `runReport` (type: `string`):

What was crawled, skipped or blocked and why, with hints.

## `files` (type: `string`):

All key-value store records (faqs.jsonl, products.jsonl, llms-full.txt, system_prompt.txt, CHANGES…).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/js/docs/quick-start"
        }
    ],
    "maxPages": 25,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("codepal/website-to-knowledge-base").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "maxPages": 25,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("codepal/website-to-knowledge-base").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "maxPages": 25,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call codepal/website-to-knowledge-base --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,codepal/website-to-knowledge-base"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WP6V5TwG6BTJDTjaM/builds/F7vQYctj5QaxnqVNM/openapi.json