LLM Wiki & OKF Bundle Generator: Website to Knowledge Base

What does the LLM Wiki & OKF Bundle Generator do?

This Actor turns any documentation site, help center, API reference or Apify dataset into an LLM wiki: an Open Knowledge Format (OKF) bundle that AI agents can read directly. OKF is the open, vendor-neutral format Google published in 2026 for packaging the context agents need, and it formalizes Andrej Karpathy's "LLM wiki" pattern: one markdown file per concept with YAML frontmatter, index.md files for step-by-step navigation, and links that turn the folder into a knowledge graph.

Give it a start URL and get back a ready-to-use bundle.zip, an llms.txt file, a graph viewer you can open in any browser, and every concept as a dataset row. No page limit, and AI summaries are included. It runs on the Apify platform, so you also get the API, scheduling (refresh your wiki every week), integrations and monitoring.

Why use this OKF bundle generator?

• Give your AI agent a product's docs in a format it understands. Drop the bundle into a repo, a Claude/Cursor project, a RAG index or Google Cloud Knowledge Catalog.
• AI summaries and key facts per page. Each concept gets a type (Guide, Tutorial, API Reference, FAQ, Pricing...), a one-line description, tags, a summary and up to 8 concrete facts, so agents can answer from the summary and open the full page only when needed.
• Document your scraped data. Point it at Apify datasets and get a schema table (types, fill rates, examples, AI field descriptions), usage notes and joins between datasets.
• Trust and freshness built in. Every concept carries OKF v0.2 provenance: sources with the page URL and last-modified date, generated (who wrote it and when), status: draft and stale_after, so agents know where a fact came from and when to refresh it.
• Diffable. Paths come from URLs, so the same page always lands in the same file. Commit bundles to git and review changes between runs.

How to turn a website into an LLM wiki (OKF bundle)

1. Open the Actor and add one or more Start URLs, for example https://docs.apify.com/academy.
2. Optionally add Apify datasets to document.
3. Set Max pages (your price cap) and click Start.
4. When the run finishes, download bundle.zip from the Output tab, or open viz.html to explore the graph.

Input

All fields are on the Input tab. The main ones:

Field	What it does
startUrls	Websites or sections to crawl. The default scope stays under each URL's path.
datasetIds	Apify datasets to document as concepts (picked in the Input tab, read-only access).
aiEnrichment	AI type, description, tags, summary and key facts per concept (default on).
includePageContent	Keep the full cleaned page under # Content (default on).
groupBy	url mirrors the site structure, type groups into guides/, tutorials/, api-references/...
maxPages, maxCrawlDepth, crawlScope, URL globs, useSitemaps	Crawl controls.
staleAfterDays	Sets stale_after this many days ahead (default 90).

{
    "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
    "datasetIds": ["6KWdtf2y4oezh4ift"],
    "maxPages": 50,
    "groupBy": "type"
}

Output

The key-value store holds bundle.zip (the whole bundle), llms.txt (an index of every concept for AI crawlers and tools), viz.html (a self-contained graph viewer) and index.md. Each concept is also a dataset row, which you can download as JSON, HTML, CSV or Excel:

{
    "conceptId": "tutorials/scraping-basics-javascript-crawling",
    "file": "tutorials/scraping-basics-javascript-crawling.md",
    "kind": "web",
    "type": "Tutorial",
    "title": "Crawling websites with Node.js",
    "description": "Lesson on following links between pages with Node.js and Cheerio.",
    "tags": ["crawling", "nodejs", "cheerio"],
    "url": "https://docs.apify.com/academy/scraping-basics-javascript/crawling",
    "aiEnriched": true,
    "document": "---\ntype: Tutorial\ntitle: Crawling websites with Node.js\n..."
}

A concept file looks like this:

---
type: Tutorial
title: Crawling the Store
description: "Crawling category and product pages with enqueueLinks() selectors and labels."
resource: https://crawlee.dev/js/docs/introduction/crawling
tags: [crawlee, crawling, enqueue-links]
status: draft
generated: { by: okf-bundle-builder/1.0+deepseek-flash, at: "2026-11-01T00:00:00Z" }
stale_after: "2027-01-30T00:00:00Z"
sources:
  - id: page
    resource: https://crawlee.dev/js/docs/introduction/crawling
---

# Summary
# Key facts
# Related
# Content

Data fields

Field	Description
conceptId	Concept ID (path in the bundle without .md)
type	OKF concept type
title, description, tags	Frontmatter metadata
url	Source page or dataset
aiEnriched	Whether the AI summary was added
document	The full concept file

How much does it cost to build an OKF bundle?

Pay per result, platform usage included:

• $0.002 per concept (one page or dataset), $2 per 1,000.
• $0.004 AI add-on per concept when AI summaries are on, so $6 per 1,000 AI concepts.
• $0.005 per run.

A 100-page docs site with AI costs about $0.61. Failed, empty and duplicate pages are free. Set Max pages or a maximum cost per run and the crawl stops there.

Tips

• Start from the docs root and keep the default "Same path" scope to stay inside the docs.
• Turn on Use sitemaps for large sites to find pages faster.
• Turn off Include full page content for a compact bundle an agent can load in one go.
• Schedule the Actor weekly and commit each bundle to git to track documentation changes.

FAQ and support

What is the Open Knowledge Format (OKF)? An open specification (v0.2) published by Google Cloud for packaging knowledge for AI agents as plain markdown with YAML frontmatter. Only type is required; v0.2 adds provenance (sources), trust (generated, verified) and freshness (status, stale_after). Google Cloud Knowledge Catalog can ingest OKF bundles.

OKF vs llms.txt vs RAG: which do I need? llms.txt is a single index file that points to pages. RAG chunks text into a vector database and loses the structure. An OKF bundle (LLM wiki) keeps one linked, typed concept per page, so an agent can browse the index, open only what it needs, and see where each fact came from. This Actor gives you the OKF bundle and an llms.txt in the same run, and the dataset rows can still feed a RAG index.

How do I use the bundle with Claude, Cursor or an MCP server? Unzip it into your repo or project knowledge, or point a file-system MCP server at the folder. Agents start from index.md and follow links.

Is this an official Google tool? No. OKF is free for anyone to use; this Actor is independent and not affiliated with Google.

Can it crawl sites behind a login? No, only public pages. It respects robots.txt by default.

Is it legal? It reads public pages like a browser. Check each site's terms before reusing its content.

Found a bug or want a feature? Open an issue on the Issues tab. Need a custom pipeline? Get in touch.