# Website to FAQ & Knowledge Pack for Support Bots (`seasons-ai/site-to-knowledge-pack`) Actor Turn a Website Content Crawler run into a support-bot knowledge pack: embedded chunks, a sourced FAQ per topic, a topic map and a grounded system prompt. - **URL**: https://apify.com/seasons-ai/site-to-knowledge-pack.md - **Developed by:** [Seasons AI](https://apify.com/seasons-ai) (community) - **Categories:** AI, Automation, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 processed pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Website to FAQ & Knowledge Pack for Support Bots Crawled your help center or docs with **Website Content Crawler**? This Actor turns that crawl into a **ready-to-use knowledge pack for a support chatbot**: - **`chunks.jsonl`**: clean, de-duplicated, heading-aware chunks (~500 tokens, with overlap). Each chunk keeps its page URL, page title and heading path (`Pricing > Refunds > Annual plans`) and comes with an embedding vector (`text-embedding-3-small`, 256–1536 dimensions). Load it into any vector store: Pinecone, Qdrant, pgvector, Weaviate, Chroma, OpenAI vector stores. - **`faq.json` + `faq.md`**: a support FAQ for each topic of your site. Every answer cites the page URLs it came from, and any answer with a number we could not find on the cited pages is flagged for review. - **`topics.json`**: a topic map that groups your pages into topics, with a summary and sample questions for each topic. - **`system-prompt.md`**: a system prompt for your support bot, grounded in the pack. It has fixed rules: answer only from retrieved passages, say "I don't know" and hand off to a human when the answer is missing, quote numbers exactly, cite source URLs and ignore instructions hidden in page text. No vector database to set up and no prompt engineering. Point the Actor at your crawl and you get files you can load into your bot. ### How to use 1. Run [Website Content Crawler](https://apify.com/apify/website-content-crawler) on **your own** site (help center, docs, product pages). The default settings work; keep "Save Markdown" on. 2. Open this Actor, pick that run's dataset in **Crawled website dataset**, and start it. 3. Download the files from the run's **Storage → Key-value store** (links are also in the log and in the `OUTPUT` record). No crawl yet? Paste pages into **Pages** as a JSON array (`url`, `title`, `markdown` or `text`), or give a link to a JSON / JSON-lines file in **Pages file URL**. ### Input | Field | Default | Meaning | |---|---|---| | `datasetId` | — | Dataset of a Website Content Crawler run (dataset picker). Read-only access | | `items` | — | Instead of a dataset: JSON array of `{url, title, markdown \| text}` | | `itemsUrl` | — | Instead of a dataset: link to a JSON array or JSON-lines file | | `maxPages` | 100 | Cap on unique pages processed and charged (1–1000) | | `maxCharsPerPage` | 30,000 | Longer pages are cut at a paragraph boundary (1,000–50,000) | | `chunkTokens` | 500 | Target chunk size (150–1000) | | `chunkOverlapTokens` | 50 | Overlap between consecutive chunks of one section | | `maxTopics` | 20 | Max topic clusters; at most one topic per 4 pages | | `faqPerTopic` | 5 | Max FAQ items per topic (1–10) | | `language` | site language | Language of the FAQ and system prompt, e.g. `German` | | `companyName`, `botName`, `supportContact` | — | Optional, used in the system prompt | | `embeddingDimensions` | 1536 | 1536, 1024, 512 or 256 | | `includeEmbeddings` | true | Turn off to get text-only chunks | | `redactContacts` | true | Replace emails and phone numbers on pages with `[email]` / `[phone]` | ```json { "datasetId": "", "maxPages": 200, "faqPerTopic": 5, "companyName": "Acme", "supportContact": "https://acme.com/contact" } ``` ### What it does with your pages - **Cleans**: removes images, permalink anchors, link targets, "On this page" and "Copy page" lines, and navigation and footer lines repeated across most pages. - **De-duplicates**: drops the same URL (canonical, without tracking parameters), exact duplicate text, near-duplicate pages (≥ 90% similar) and sections repeated across pages. Duplicates are free. - **Chunks by headings**: a chunk never mixes two top-level sections, small sibling sections are merged, and code blocks stay whole. - **Embeds and clusters**: groups pages into topics by meaning. - **Writes the FAQ** from the most central passages of each topic. It uses only facts stated in those passages. An answer that cannot be tied to a page of its topic is dropped and not charged. ### Output Dataset: one row per input page. ```json { "url": "https://docs.example.com/storage/dataset", "title": "Dataset | Example Docs", "status": "processed", "chars": 14721, "truncated": false, "chunks": 16, "tokens": 4398, "headings": 12, "topicId": 0, "topic": "Storage", "faqItems": 1, "reason": null, "duplicateOf": null } ``` `status` is `processed`, `duplicate` (with `duplicateOf`) or `skipped` (with `reason`, e.g. HTTP 404, too little text, `maxPages` reached). `chunks.jsonl` (one JSON object per line): ```json {"id":"9f2c…","url":"https://docs.example.com/storage/dataset","title":"Dataset | Example Docs","headingPath":["Dataset","Basic usage","Apify Console"],"topicId":0,"tokens":312,"text":"To view your datasets in the Console…","embedding":[0.0123,-0.0456,…]} ``` `faq.json`: ```json [{"question":"How long are unnamed datasets kept?","answer":"Unnamed datasets expire after 7 days unless otherwise specified; named datasets are kept indefinitely.","sources":["https://docs.example.com/storage/dataset"],"topicId":0,"topic":"Storage","needsReview":false,"unverifiedNumbers":[]}] ``` Large sites: when the chunks are larger than 8 MB, `chunks.jsonl` holds the first part and the rest continue in `chunks-002.jsonl`, `chunks-003.jsonl`, and so on (all listed in the `OUTPUT` record). Lower `embeddingDimensions` to get smaller files. File links stay valid while the run's storage is kept, so download the files to keep them. ### Pricing You pay for delivered work only. Duplicate, empty and failed pages are free, and so are FAQ answers that are dropped. | Event | Price | |---|---| | Page processed (cleaned, chunked, embedded) | $0.003 | | FAQ item delivered | $0.002 | | Knowledge pack built (topic map + system prompt, once per run) | $0.01 | Example: a 100-page help center with 20 topics and 5 FAQ items each costs 100 × $0.003 + 100 × $0.002 + $0.01 = **$0.51**. Before any paid step the Actor works out the maximum cost of the run and checks it against your run's maximum cost. If the limit is too low, it stops before spending and tells you the amount to set. Available on paid Apify plans. ### Limits - Up to 1,000 unique pages per run and 50,000 characters per page, after cleaning (about 30,000 characters for Chinese, Japanese or Korean text). Longer pages are cut at a paragraph boundary, and the dataset row shows `truncated: true`. - Each topic's FAQ is written from up to ~4,500 tokens of its most central passages. Very large topics get FAQ items about their main themes, not every page. - Only text is used. The Actor does not read images, PDFs linked from pages or content behind a login. - FAQ answers come from your pages, but review them before you publish. An answer with a number that is not on the cited pages gets `needsReview: true`. - Use it on sites you own or are allowed to reuse. Emails and phone numbers are removed by default, and the pack stays at company and content level. ### Good to know - On the free Apify plan the Actor returns a short preview with an example result and prices instead of running. # Actor input Schema ## `datasetId` (type: `string`): The dataset of your Website Content Crawler run (apify/website-content-crawler) on your own site. Pick it from the list or paste its ID. The Actor only reads it. ## `items` (type: `array`): Optional. A JSON array of pages, each with "url", "title" and "markdown" (or "text"). Used only when no dataset is chosen. ## `itemsUrl` (type: `string`): Optional. Link to a JSON array or JSON-lines file with the same fields (for example a dataset export link). Used only when neither a dataset nor pages are given. ## `maxPages` (type: `integer`): Hard cap on unique pages processed (and charged). Duplicate and empty pages are skipped for free and do not count. ## `maxCharsPerPage` (type: `integer`): Longer pages are cut at a paragraph boundary after this many characters (after cleaning). ## `chunkTokens` (type: `integer`): Target size of each chunk. Chunks follow the page headings and never mix two top-level sections. ## `chunkOverlapTokens` (type: `integer`): Text repeated between consecutive chunks of the same section, so answers that span a boundary are still retrieved. At most a third of the chunk size. ## `maxTopics` (type: `integer`): Upper limit on topic clusters. The Actor makes at most one topic per 4 pages. ## `faqPerTopic` (type: `integer`): Up to this many question-answer pairs per topic. Fewer are written when the pages do not support more; you pay only for items delivered. ## `language` (type: `string`): Language of the FAQ and the system prompt, e.g. "German". Leave empty to use the site's language. ## `companyName` (type: `string`): Optional. Used in the system prompt. Detected from the site when empty. ## `botName` (type: `string`): Optional. How the support bot introduces itself. ## `supportContact` (type: `string`): Optional. Where the bot sends users it cannot help (e.g. a help-desk URL or support address). Contact details found on pages are removed from the pack. ## `embeddingDimensions` (type: `string`): Size of each chunk vector. Use 1536, 1024, 512 or 256. 1536 is the most accurate; 512 or 256 make files 3-6× smaller for large sites. Same price. ## `includeEmbeddings` (type: `boolean`): Turn off if you embed the chunks with your own model; the text, URL and heading path stay. Same price. ## `redactContacts` (type: `boolean`): Replace emails and phone numbers found on pages with \[email] / \[phone] in every output. ## Actor input object example ```json { "items": [ { "url": "https://example.com/help/returns", "title": "Returns and refunds", "markdown": "# Returns and refunds\n\n## How long do I have to return an item?\n\nYou can return unused items within 30 days of delivery. Start a return from the Orders page in your account and print the prepaid label.\n\n## When will I get my refund?\n\nRefunds are sent to the original payment method within 5 business days after the warehouse receives the item. Shipping fees are refunded only if the item arrived damaged.\n\n## Can I exchange an item instead?\n\nYes. Choose Exchange instead of Refund when you start the return and pick the new size or colour. Exchanges ship free of charge." }, { "url": "https://example.com/help/shipping", "title": "Shipping", "markdown": "# Shipping\n\n## Where do you ship?\n\nWe ship to the EU, the UK and the United States. Orders over 50 EUR ship free; smaller orders cost 4.90 EUR.\n\n## How fast is delivery?\n\nStandard delivery takes 2 to 4 business days in the EU and 5 to 8 business days to the UK and the US. Express delivery arrives the next business day in Germany and Austria when ordered before 2 pm.\n\n## How do I track my order?\n\nYou get a tracking link by email as soon as the parcel leaves the warehouse. The same link is shown on the Orders page." } ], "maxPages": 100, "maxCharsPerPage": 30000, "chunkTokens": 500, "chunkOverlapTokens": 50, "maxTopics": 20, "faqPerTopic": 5, "embeddingDimensions": "1536", "includeEmbeddings": true, "redactContacts": true } ``` # Actor output Schema ## `pages` (type: `string`): No description ## `systemPrompt` (type: `string`): No description ## `faqMarkdown` (type: `string`): No description ## `faqJson` (type: `string`): No description ## `topics` (type: `string`): No description ## `chunks` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "items": [ { "url": "https://example.com/help/returns", "title": "Returns and refunds", "markdown": "# Returns and refunds\n\n## How long do I have to return an item?\n\nYou can return unused items within 30 days of delivery. Start a return from the Orders page in your account and print the prepaid label.\n\n## When will I get my refund?\n\nRefunds are sent to the original payment method within 5 business days after the warehouse receives the item. Shipping fees are refunded only if the item arrived damaged.\n\n## Can I exchange an item instead?\n\nYes. Choose Exchange instead of Refund when you start the return and pick the new size or colour. Exchanges ship free of charge." }, { "url": "https://example.com/help/shipping", "title": "Shipping", "markdown": "# Shipping\n\n## Where do you ship?\n\nWe ship to the EU, the UK and the United States. Orders over 50 EUR ship free; smaller orders cost 4.90 EUR.\n\n## How fast is delivery?\n\nStandard delivery takes 2 to 4 business days in the EU and 5 to 8 business days to the UK and the US. Express delivery arrives the next business day in Germany and Austria when ordered before 2 pm.\n\n## How do I track my order?\n\nYou get a tracking link by email as soon as the parcel leaves the warehouse. The same link is shown on the Orders page." } ] }; // Run the Actor and wait for it to finish const run = await client.actor("seasons-ai/site-to-knowledge-pack").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "items": [ { "url": "https://example.com/help/returns", "title": "Returns and refunds", "markdown": """# Returns and refunds ## How long do I have to return an item? You can return unused items within 30 days of delivery. Start a return from the Orders page in your account and print the prepaid label. ## When will I get my refund? Refunds are sent to the original payment method within 5 business days after the warehouse receives the item. Shipping fees are refunded only if the item arrived damaged. ## Can I exchange an item instead? Yes. Choose Exchange instead of Refund when you start the return and pick the new size or colour. Exchanges ship free of charge.""", }, { "url": "https://example.com/help/shipping", "title": "Shipping", "markdown": """# Shipping ## Where do you ship? We ship to the EU, the UK and the United States. Orders over 50 EUR ship free; smaller orders cost 4.90 EUR. ## How fast is delivery? Standard delivery takes 2 to 4 business days in the EU and 5 to 8 business days to the UK and the US. Express delivery arrives the next business day in Germany and Austria when ordered before 2 pm. ## How do I track my order? You get a tracking link by email as soon as the parcel leaves the warehouse. The same link is shown on the Orders page.""", }, ] } # Run the Actor and wait for it to finish run = client.actor("seasons-ai/site-to-knowledge-pack").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "items": [ { "url": "https://example.com/help/returns", "title": "Returns and refunds", "markdown": "# Returns and refunds\\n\\n## How long do I have to return an item?\\n\\nYou can return unused items within 30 days of delivery. Start a return from the Orders page in your account and print the prepaid label.\\n\\n## When will I get my refund?\\n\\nRefunds are sent to the original payment method within 5 business days after the warehouse receives the item. Shipping fees are refunded only if the item arrived damaged.\\n\\n## Can I exchange an item instead?\\n\\nYes. Choose Exchange instead of Refund when you start the return and pick the new size or colour. Exchanges ship free of charge." }, { "url": "https://example.com/help/shipping", "title": "Shipping", "markdown": "# Shipping\\n\\n## Where do you ship?\\n\\nWe ship to the EU, the UK and the United States. Orders over 50 EUR ship free; smaller orders cost 4.90 EUR.\\n\\n## How fast is delivery?\\n\\nStandard delivery takes 2 to 4 business days in the EU and 5 to 8 business days to the UK and the US. Express delivery arrives the next business day in Germany and Austria when ordered before 2 pm.\\n\\n## How do I track my order?\\n\\nYou get a tracking link by email as soon as the parcel leaves the warehouse. The same link is shown on the Orders page." } ] }' | apify call seasons-ai/site-to-knowledge-pack --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,seasons-ai/site-to-knowledge-pack" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/YVjnY0vgGYH4i6hYN/builds/OVRdCk7GBd9FPb5Fb/openapi.json