AI Knowledge Base Monitor

Keep your AI knowledge base current

Monitor public product documentation and help centers, spot new or edited pages, and export cleaned, source-linked content for a RAG pipeline or chatbot. Configure the public documentation URLs you are allowed to use; the Actor checks robots.txt, stays on each starting host and path area, and saves a content hash between runs so it can label pages as new, changed, or unchanged.

Useful for: SaaS support teams, chatbot developers, documentation teams, and AI consultancies maintaining a knowledge base from public product docs.

What you get

• Clean Markdown and plain text with page titles, descriptions, and headings.
• A SHA-256 content hash and change status to help downstream jobs process updates.
• A complete current dataset by default, or only new and changed pages when you turn off Include unchanged pages.
• Sitemap discovery plus same-host link following with configurable limits.
• Persistent change history across runs in the Actor’s named Apify key-value store.
• Dataset export as JSON, CSV, or Excel, or direct use through the Apify API.

Quick start

1. Add a public documentation start page or sitemap URL under Documentation URLs or sitemap URLs.
2. Choose a page limit and link depth suited to the site.
3. Keep Respect robots.txt enabled and review the website’s terms for your intended use.
4. Run the Actor. Schedule later runs to refresh the pages and compare their content hashes.
5. Send the Markdown or text fields to your own embedding, search, or RAG workflow.

Example input

{
  "startUrls": ["https://docs.example.com/"],
  "maxPages": 100,
  "maxDepth": 2,
  "maxTextLength": 30000,
  "requestDelayMs": 1000,
  "includeUnchanged": true,
  "respectRobotsTxt": true
}

Change detection

The Actor keeps the last observed hash for each canonical page in a persistent key-value store. The first successful observation is new; a different hash on a later run is changed; a matching hash is unchanged. changedAt stays at the first timestamp for the current version and resets when the content changes.

Each saved task keeps its own change history, so two tasks watching the same site both receive every change. Runs started without a task share one history per set of start URLs.

Change history is saved during the run and when it stops, and a page’s hash is kept only after its record is saved, so pages a stopped run did not reach are reported on the next run.

This MVP detects new and edited pages it can reach. It does not report deleted pages, make semantic judgments about whether an edit matters, create embeddings, or connect to a vector database. Keep unchanged pages enabled when you need a complete refreshable corpus; disable it to reduce downstream processing to new and edited pages.

Coverage and safety limits

• Start URLs are limited to 10. Per-run HTML page count is capped at 1,000.
• Link crawling stays on the exact host and beneath the configured start path. A start URL at / covers that host’s public paths.
• Sitemap entries are limited to the same host. Cross-origin links and redirects are skipped.
• A run reads at most 25 sitemaps and queues at most 10,000 discovered URLs, so a large sitemap index cannot use up the run.
• Responses are capped at 5 MB; extracted text is capped per page by maxTextLength.
• The crawler makes one request at a time and applies requestDelayMs plus any robots.txt crawl-delay rule.
• Robots disallowed paths are skipped by default, including * wildcard and $ end-of-path rules. If a site’s robots.txt cannot be safely read, the Actor skips that site and continues with the other start URLs.
• Navigation, sidebars, headers, footers, and cookie banners are left out of the text, but their links are still used to discover pages.
• Pages are saved to the dataset as they are collected, so a run that is stopped or fails keeps the pages it already saved. URLs that declare the same canonical page are saved once.
• It does not log in, bypass access controls, defeat CAPTCHAs, or use proxies.

Publicly accessible content is not automatically free to reuse. Check the source’s terms, copyright, and applicable rules before collecting, storing, or sending content to an AI provider. Preserve source URLs and use only content for which you have permission.

Output fields

title, url, description, headings, markdown, text, contentHash, changeType, changedAt, scrapedAt, and depth.

Pricing

This Actor is currently free to use as a public beta; no separate per-page Actor fee is configured. Apify platform usage may still apply according to your Apify account plan. Pricing can be revisited after real users validate the results and typical run costs are known.