Website to FAQ & Knowledge Pack for Support Bots

Crawled your help center or docs with Website Content Crawler? This Actor turns that crawl into a ready-to-use knowledge pack for a support chatbot:

• chunks.jsonl: clean, de-duplicated, heading-aware chunks (~500 tokens, with overlap). Each chunk keeps its page URL, page title and heading path (Pricing > Refunds > Annual plans) and comes with an embedding vector (text-embedding-3-small, 256–1536 dimensions). Load it into any vector store: Pinecone, Qdrant, pgvector, Weaviate, Chroma, OpenAI vector stores.
• faq.json + faq.md: a support FAQ for each topic of your site. Every answer cites the page URLs it came from, and any answer with a number we could not find on the cited pages is flagged for review.
• topics.json: a topic map that groups your pages into topics, with a summary and sample questions for each topic.
• system-prompt.md: a system prompt for your support bot, grounded in the pack. It has fixed rules: answer only from retrieved passages, say "I don't know" and hand off to a human when the answer is missing, quote numbers exactly, cite source URLs and ignore instructions hidden in page text.

No vector database to set up and no prompt engineering. Point the Actor at your crawl and you get files you can load into your bot.

How to use

1. Run Website Content Crawler on your own site (help center, docs, product pages). The default settings work; keep "Save Markdown" on.
2. Open this Actor, pick that run's dataset in Crawled website dataset, and start it.
3. Download the files from the run's Storage → Key-value store (links are also in the log and in the OUTPUT record).

No crawl yet? Paste pages into Pages as a JSON array (url, title, markdown or text), or give a link to a JSON / JSON-lines file in Pages file URL.

Input

Field	Default	Meaning
datasetId	—	Dataset of a Website Content Crawler run (dataset picker). Read-only access
items	—	Instead of a dataset: JSON array of {url, title, markdown | text}
itemsUrl	—	Instead of a dataset: link to a JSON array or JSON-lines file
maxPages	100	Cap on unique pages processed and charged (1–1000)
maxCharsPerPage	30,000	Longer pages are cut at a paragraph boundary (1,000–50,000)
chunkTokens	500	Target chunk size (150–1000)
chunkOverlapTokens	50	Overlap between consecutive chunks of one section
maxTopics	20	Max topic clusters; at most one topic per 4 pages
faqPerTopic	5	Max FAQ items per topic (1–10)
language	site language	Language of the FAQ and system prompt, e.g. German
companyName, botName, supportContact	—	Optional, used in the system prompt
embeddingDimensions	1536	1536, 1024, 512 or 256
includeEmbeddings	true	Turn off to get text-only chunks
redactContacts	true	Replace emails and phone numbers on pages with [email] / [phone]

{
  "datasetId": "<dataset ID of your Website Content Crawler run>",
  "maxPages": 200,
  "faqPerTopic": 5,
  "companyName": "Acme",
  "supportContact": "https://acme.com/contact"
}

What it does with your pages

• Cleans: removes images, permalink anchors, link targets, "On this page" and "Copy page" lines, and navigation and footer lines repeated across most pages.
• De-duplicates: drops the same URL (canonical, without tracking parameters), exact duplicate text, near-duplicate pages (≥ 90% similar) and sections repeated across pages. Duplicates are free.
• Chunks by headings: a chunk never mixes two top-level sections, small sibling sections are merged, and code blocks stay whole.
• Embeds and clusters: groups pages into topics by meaning.
• Writes the FAQ from the most central passages of each topic. It uses only facts stated in those passages. An answer that cannot be tied to a page of its topic is dropped and not charged.

Output

Dataset: one row per input page.

{
  "url": "https://docs.example.com/storage/dataset",
  "title": "Dataset | Example Docs",
  "status": "processed",
  "chars": 14721,
  "truncated": false,
  "chunks": 16,
  "tokens": 4398,
  "headings": 12,
  "topicId": 0,
  "topic": "Storage",
  "faqItems": 1,
  "reason": null,
  "duplicateOf": null
}

status is processed, duplicate (with duplicateOf) or skipped (with reason, e.g. HTTP 404, too little text, maxPages reached).

chunks.jsonl (one JSON object per line):

{"id":"9f2c…","url":"https://docs.example.com/storage/dataset","title":"Dataset | Example Docs","headingPath":["Dataset","Basic usage","Apify Console"],"topicId":0,"tokens":312,"text":"To view your datasets in the Console…","embedding":[0.0123,-0.0456,…]}

faq.json:

[{"question":"How long are unnamed datasets kept?","answer":"Unnamed datasets expire after 7 days unless otherwise specified; named datasets are kept indefinitely.","sources":["https://docs.example.com/storage/dataset"],"topicId":0,"topic":"Storage","needsReview":false,"unverifiedNumbers":[]}]

Large sites: when the chunks are larger than 8 MB, chunks.jsonl holds the first part and the rest continue in chunks-002.jsonl, chunks-003.jsonl, and so on (all listed in the OUTPUT record). Lower embeddingDimensions to get smaller files. File links stay valid while the run's storage is kept, so download the files to keep them.

Pricing

You pay for delivered work only. Duplicate, empty and failed pages are free, and so are FAQ answers that are dropped.

Event	Price
Page processed (cleaned, chunked, embedded)	$0.003
FAQ item delivered	$0.002
Knowledge pack built (topic map + system prompt, once per run)	$0.01

Example: a 100-page help center with 20 topics and 5 FAQ items each costs 100 × $0.003 + 100 × $0.002 + $0.01 = $0.51. Before any paid step the Actor works out the maximum cost of the run and checks it against your run's maximum cost. If the limit is too low, it stops before spending and tells you the amount to set. Available on paid Apify plans.

Limits

• Up to 1,000 unique pages per run and 50,000 characters per page, after cleaning (about 30,000 characters for Chinese, Japanese or Korean text). Longer pages are cut at a paragraph boundary, and the dataset row shows truncated: true.
• Each topic's FAQ is written from up to ~4,500 tokens of its most central passages. Very large topics get FAQ items about their main themes, not every page.
• Only text is used. The Actor does not read images, PDFs linked from pages or content behind a login.
• FAQ answers come from your pages, but review them before you publish. An answer with a number that is not on the cited pages gets needsReview: true.
• Use it on sites you own or are allowed to reuse. Emails and phone numbers are removed by default, and the pack stays at company and content level.

Good to know

• On the free Apify plan the Actor returns a short preview with an example result and prices instead of running.