# LLM Wiki & OKF Bundle Generator: Website to Knowledge Base (`scrape-lads/llm-wiki-okf-generator`) Actor

Turn any docs site, help center or Apify dataset into an Open Knowledge Format (OKF) bundle for AI agents: markdown concepts with frontmatter, index files, a zip and a graph viewer.

- **URL**: https://apify.com/scrape-lads/llm-wiki-okf-generator.md
- **Developed by:** [Scrape Lads](https://apify.com/scrape-lads) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.60 / 1,000 concepts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does the LLM Wiki & OKF Bundle Generator do?

**This Actor turns any documentation site, help center, API reference or Apify dataset into an LLM wiki: an [Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/open-knowledge-format) bundle that AI agents can read directly.** OKF is the open, vendor-neutral format Google published in 2026 for packaging the context agents need, and it formalizes Andrej Karpathy's "LLM wiki" pattern: one markdown file per concept with YAML frontmatter, `index.md` files for step-by-step navigation, and links that turn the folder into a knowledge graph.

Give it a start URL and get back a **ready-to-use bundle.zip**, an **llms.txt** file, a **graph viewer** you can open in any browser, and every concept as a dataset row. No page limit, and AI summaries are included. It runs on the Apify platform, so you also get the API, scheduling (refresh your wiki every week), integrations and monitoring.

### Why use this OKF bundle generator?

- **Give your AI agent a product's docs in a format it understands.** Drop the bundle into a repo, a Claude/Cursor project, a RAG index or Google Cloud Knowledge Catalog.
- **AI summaries and key facts per page.** Each concept gets a type (Guide, Tutorial, API Reference, FAQ, Pricing...), a one-line description, tags, a summary and up to 8 concrete facts, so agents can answer from the summary and open the full page only when needed.
- **Document your scraped data.** Point it at Apify datasets and get a schema table (types, fill rates, examples, AI field descriptions), usage notes and joins between datasets.
- **Trust and freshness built in.** Every concept carries OKF v0.2 provenance: `sources` with the page URL and last-modified date, `generated` (who wrote it and when), `status: draft` and `stale_after`, so agents know where a fact came from and when to refresh it.
- **Diffable.** Paths come from URLs, so the same page always lands in the same file. Commit bundles to git and review changes between runs.

### How to turn a website into an LLM wiki (OKF bundle)

1. Open the Actor and add one or more **Start URLs**, for example `https://docs.apify.com/academy`.
2. Optionally add **Apify datasets** to document.
3. Set **Max pages** (your price cap) and click **Start**.
4. When the run finishes, download **bundle.zip** from the Output tab, or open **viz.html** to explore the graph.

### Input

All fields are on the Input tab. The main ones:

| Field | What it does |
|---|---|
| `startUrls` | Websites or sections to crawl. The default scope stays under each URL's path. |
| `datasetIds` | Apify datasets to document as concepts (picked in the Input tab, read-only access). |
| `aiEnrichment` | AI type, description, tags, summary and key facts per concept (default on). |
| `includePageContent` | Keep the full cleaned page under `# Content` (default on). |
| `groupBy` | `url` mirrors the site structure, `type` groups into `guides/`, `tutorials/`, `api-references/`... |
| `maxPages`, `maxCrawlDepth`, `crawlScope`, URL globs, `useSitemaps` | Crawl controls. |
| `staleAfterDays` | Sets `stale_after` this many days ahead (default 90). |

```json
{
    "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
    "datasetIds": ["6KWdtf2y4oezh4ift"],
    "maxPages": 50,
    "groupBy": "type"
}
```

### Output

The key-value store holds **bundle.zip** (the whole bundle), **llms.txt** (an index of every concept for AI crawlers and tools), **viz.html** (a self-contained graph viewer) and **index.md**. Each concept is also a dataset row, which you can download as JSON, HTML, CSV or Excel:

```json
{
    "conceptId": "tutorials/scraping-basics-javascript-crawling",
    "file": "tutorials/scraping-basics-javascript-crawling.md",
    "kind": "web",
    "type": "Tutorial",
    "title": "Crawling websites with Node.js",
    "description": "Lesson on following links between pages with Node.js and Cheerio.",
    "tags": ["crawling", "nodejs", "cheerio"],
    "url": "https://docs.apify.com/academy/scraping-basics-javascript/crawling",
    "aiEnriched": true,
    "document": "---\ntype: Tutorial\ntitle: Crawling websites with Node.js\n..."
}
```

A concept file looks like this:

```markdown
---
type: Tutorial
title: Crawling the Store
description: "Crawling category and product pages with enqueueLinks() selectors and labels."
resource: https://crawlee.dev/js/docs/introduction/crawling
tags: [crawlee, crawling, enqueue-links]
status: draft
generated: { by: okf-bundle-builder/1.0+deepseek-flash, at: "2026-11-01T00:00:00Z" }
stale_after: "2027-01-30T00:00:00Z"
sources:
  - id: page
    resource: https://crawlee.dev/js/docs/introduction/crawling
---

## Summary
## Key facts
## Related
## Content
```

#### Data fields

| Field | Description |
|---|---|
| `conceptId` | Concept ID (path in the bundle without `.md`) |
| `type` | OKF concept type |
| `title`, `description`, `tags` | Frontmatter metadata |
| `url` | Source page or dataset |
| `aiEnriched` | Whether the AI summary was added |
| `document` | The full concept file |

### How much does it cost to build an OKF bundle?

Pay per result, platform usage included:

- **$0.002 per concept** (one page or dataset), $2 per 1,000.
- **$0.004 AI add-on per concept** when AI summaries are on, so $6 per 1,000 AI concepts.
- **$0.005 per run.**

A 100-page docs site with AI costs about $0.61. Failed, empty and duplicate pages are free. Set **Max pages** or a maximum cost per run and the crawl stops there.

### Tips

- Start from the docs root and keep the default "Same path" scope to stay inside the docs.
- Turn on **Use sitemaps** for large sites to find pages faster.
- Turn off **Include full page content** for a compact bundle an agent can load in one go.
- Schedule the Actor weekly and commit each bundle to git to track documentation changes.

### FAQ and support

**What is the Open Knowledge Format (OKF)?** An open specification (v0.2) published by Google Cloud for packaging knowledge for AI agents as plain markdown with YAML frontmatter. Only `type` is required; v0.2 adds provenance (`sources`), trust (`generated`, `verified`) and freshness (`status`, `stale_after`). Google Cloud Knowledge Catalog can ingest OKF bundles.

**OKF vs llms.txt vs RAG: which do I need?** llms.txt is a single index file that points to pages. RAG chunks text into a vector database and loses the structure. An OKF bundle (LLM wiki) keeps one linked, typed concept per page, so an agent can browse the index, open only what it needs, and see where each fact came from. This Actor gives you the OKF bundle and an llms.txt in the same run, and the dataset rows can still feed a RAG index.

**How do I use the bundle with Claude, Cursor or an MCP server?** Unzip it into your repo or project knowledge, or point a file-system MCP server at the folder. Agents start from `index.md` and follow links.

**Is this an official Google tool?** No. OKF is free for anyone to use; this Actor is independent and not affiliated with Google.

**Can it crawl sites behind a login?** No, only public pages. It respects robots.txt by default.

**Is it legal?** It reads public pages like a browser. Check each site's terms before reusing its content.

Found a bug or want a feature? Open an issue on the Issues tab. Need a custom pipeline? Get in touch.

# Actor input Schema

## `startUrls` (type: `array`):

Websites or sections to turn into concepts, for example a docs site, help center or API reference. With the default scope the crawler stays under each URL's path, so https://docs.apify.com/academy only crawls pages under /academy. Leave empty to build a bundle from datasets only.

## `datasetIds` (type: `array`):

Optional. Pick Apify datasets to document (read-only access). Each dataset becomes one concept with its inferred schema, fill rates, sample items and joins to the other datasets.

## `aiEnrichment` (type: `boolean`):

Each concept gets an AI-written type, one-line description, tags, summary and key facts. Turn off for a cheaper bundle with the page content only.

## `includePageContent` (type: `boolean`):

Keep the cleaned page Markdown under # Content in each concept. Turn off for compact concepts with only the AI summary and key facts (always kept when AI is off).

## `groupBy` (type: `string`):

How concept files are arranged in the bundle. Grouping by type needs AI enrichment; without it the URL layout is used.

## `bundleName` (type: `string`):

Name of the bundle folder and viewer title. Defaults to the website's host name.

## `staleAfterDays` (type: `integer`):

Sets each concept's stale_after date this many days ahead, so agents know when to refresh it. 0 leaves it out.

## `maxPages` (type: `integer`):

Stop after this many pages. Each page becomes one concept, so this is also your price cap.

## `maxCrawlDepth` (type: `integer`):

How many links away from a start URL to follow. 0 crawls only the start URLs.

## `crawlScope` (type: `string`):

Which discovered links to follow. "Same path" stays under the start URL's folder, "Same hostname" allows the whole host, "Same domain" also allows subdomains.

## `includeUrlGlobs` (type: `array`):

Optional. Only follow links matching one of these glob patterns, e.g. https://example.com/docs/\*\*. Applied on top of the crawl scope.

## `excludeUrlGlobs` (type: `array`):

Optional. Never follow links matching these glob patterns, e.g. **/changelog/** or \*\*/*?page=*.

## `useSitemaps` (type: `boolean`):

Also read robots.txt Sitemap entries and /sitemap.xml to find pages that are not linked. In-scope sitemap URLs are crawled alongside discovered links.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that the site's robots.txt disallows.

## `removeBoilerplate` (type: `boolean`):

Strip navigation, headers, footers, sidebars, cookie banners and share widgets, and keep the main content only.

## `removeElementsCssSelector` (type: `string`):

Optional CSS selector of additional elements to delete before conversion, e.g. .ad-slot, #comments.

## `maxConcurrency` (type: `integer`):

Maximum parallel HTTP requests. Browser pages are always limited to a few at a time.

## `crawlerType` (type: `string`):

Auto fetches pages over HTTP and renders only empty JavaScript app shells in a headless browser.

## `datasetSampleSize` (type: `integer`):

How many items of each dataset are read to infer its schema.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "aiEnrichment": true,
  "includePageContent": true,
  "groupBy": "url",
  "staleAfterDays": 90,
  "maxPages": 50,
  "maxCrawlDepth": 5,
  "crawlScope": "path",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "useSitemaps": false,
  "respectRobotsTxt": true,
  "removeBoilerplate": true,
  "maxConcurrency": 10,
  "crawlerType": "auto",
  "datasetSampleSize": 500
}
```

# Actor output Schema

## `bundleZip` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `viewer` (type: `string`):

No description

## `concepts` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrape-lads/llm-wiki-okf-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }] }

# Run the Actor and wait for it to finish
run = client.actor("scrape-lads/llm-wiki-okf-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ]
}' |
apify call scrape-lads/llm-wiki-okf-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrape-lads/llm-wiki-okf-generator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/naCBx49FVkOVw2imv/builds/bGSKaW6ReEROs5ov0/openapi.json