# AI Knowledge Base Monitor (`pharaoh_data/ai-knowledge-base-monitor`) Actor

Monitor public documentation and help centers, detect page changes, and export clean content for RAG and AI knowledge bases.

- **URL**: https://apify.com/pharaoh_data/ai-knowledge-base-monitor.md
- **Developed by:** [Pharaoh Data](https://apify.com/pharaoh_data) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Knowledge Base Monitor

### Keep your AI knowledge base current

Monitor public product documentation and help centers, spot new or edited pages, and export cleaned, source-linked content for a RAG pipeline or chatbot. Configure the public documentation URLs you are allowed to use; the Actor checks `robots.txt`, stays on each starting host and path area, and saves a content hash between runs so it can label pages as **new**, **changed**, or **unchanged**.

**Useful for:** SaaS support teams, chatbot developers, documentation teams, and AI consultancies maintaining a knowledge base from public product docs.

### What you get

- Clean Markdown and plain text with page titles, descriptions, and headings.
- A SHA-256 content hash and change status to help downstream jobs process updates.
- A complete current dataset by default, or only new and changed pages when you turn off **Include unchanged pages**.
- Sitemap discovery plus same-host link following with configurable limits.
- Persistent change history across runs in the Actor’s named Apify key-value store.
- Dataset export as JSON, CSV, or Excel, or direct use through the Apify API.

### Quick start

1. Add a public documentation start page or sitemap URL under **Documentation URLs or sitemap URLs**.
2. Choose a page limit and link depth suited to the site.
3. Keep **Respect robots.txt** enabled and review the website’s terms for your intended use.
4. Run the Actor. Schedule later runs to refresh the pages and compare their content hashes.
5. Send the Markdown or text fields to your own embedding, search, or RAG workflow.

#### Example input

```json
{
  "startUrls": ["https://docs.example.com/"],
  "maxPages": 100,
  "maxDepth": 2,
  "maxTextLength": 30000,
  "requestDelayMs": 1000,
  "includeUnchanged": true,
  "respectRobotsTxt": true
}
```

### Change detection

The Actor keeps the last observed hash for each canonical page in a persistent key-value store. The first successful observation is `new`; a different hash on a later run is `changed`; a matching hash is `unchanged`. `changedAt` stays at the first timestamp for the current version and resets when the content changes.

Each saved task keeps its own change history, so two tasks watching the same site both receive every change. Runs started without a task share one history per set of start URLs.

Change history is saved during the run and when it stops, and a page’s hash is kept only after its record is saved, so pages a stopped run did not reach are reported on the next run.

This MVP detects new and edited pages it can reach. It does **not** report deleted pages, make semantic judgments about whether an edit matters, create embeddings, or connect to a vector database. Keep unchanged pages enabled when you need a complete refreshable corpus; disable it to reduce downstream processing to new and edited pages.

### Coverage and safety limits

- Start URLs are limited to 10. Per-run HTML page count is capped at 1,000.
- Link crawling stays on the exact host and beneath the configured start path. A start URL at `/` covers that host’s public paths.
- Sitemap entries are limited to the same host. Cross-origin links and redirects are skipped.
- A run reads at most 25 sitemaps and queues at most 10,000 discovered URLs, so a large sitemap index cannot use up the run.
- Responses are capped at 5 MB; extracted text is capped per page by `maxTextLength`.
- The crawler makes one request at a time and applies `requestDelayMs` plus any robots.txt crawl-delay rule.
- Robots disallowed paths are skipped by default, including `*` wildcard and `$` end-of-path rules. If a site’s robots.txt cannot be safely read, the Actor skips that site and continues with the other start URLs.
- Navigation, sidebars, headers, footers, and cookie banners are left out of the text, but their links are still used to discover pages.
- Pages are saved to the dataset as they are collected, so a run that is stopped or fails keeps the pages it already saved. URLs that declare the same canonical page are saved once.
- It does not log in, bypass access controls, defeat CAPTCHAs, or use proxies.

Publicly accessible content is not automatically free to reuse. Check the source’s terms, copyright, and applicable rules before collecting, storing, or sending content to an AI provider. Preserve source URLs and use only content for which you have permission.

### Output fields

`title`, `url`, `description`, `headings`, `markdown`, `text`, `contentHash`, `changeType`, `changedAt`, `scrapedAt`, and `depth`.

### Pricing

This Actor is currently free to use as a public beta; no separate per-page Actor fee is configured. Apify platform usage may still apply according to your Apify account plan. Pricing can be revisited after real users validate the results and typical run costs are known.

# Actor input Schema

## `startUrls` (type: `array`):

One or more public documentation pages, documentation roots, or sitemap.xml URLs. Only crawl sites whose terms permit your intended use.

## `maxPages` (type: `integer`):

Maximum HTML documentation pages to fetch per run, across all start URLs.

## `maxDepth` (type: `integer`):

How many same-site link hops beyond each start URL to follow. Sitemap URLs are read independently of this limit.

## `maxTextLength` (type: `integer`):

Maximum extracted text characters per page to keep outputs bounded.

## `requestDelayMs` (type: `integer`):

Minimum pause between requests to the same site. robots.txt crawl-delay rules are also respected when available.

## `includeUnchanged` (type: `boolean`):

Keep the dataset as a complete current knowledge base (true), or return only new and changed pages (false).

## `respectRobotsTxt` (type: `boolean`):

Skip paths disallowed by each site’s robots.txt. Enabled by default.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/"
  ],
  "maxPages": 100,
  "maxDepth": 2,
  "maxTextLength": 30000,
  "requestDelayMs": 1000,
  "includeUnchanged": true,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `pages` (type: `string`):

Cleaned pages with content hashes and change status. Export as JSON, CSV, or Excel, or use the Apify API.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pharaoh_data/ai-knowledge-base-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://docs.apify.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("pharaoh_data/ai-knowledge-base-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/"
  ]
}' |
apify call pharaoh_data/ai-knowledge-base-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pharaoh_data/ai-knowledge-base-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oWu4L8qVxKAr6Ezwf/builds/1NjRX4UrOpDJX5U7j/openapi.json