# Website Change Monitor & RAG Knowledge Sync (`skusol/website-knowledge-monitor`) Actor

Monitor documentation, help centers and policy pages. Get readable changes, stable document IDs, chunk upserts and confirmed deletion events for AI knowledge bases. No LLM API key required.

- **URL**: https://apify.com/skusol/website-knowledge-monitor.md
- **Developed by:** [Kusol Sukhakul](https://apify.com/skusol) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$5.00 / 1,000 page checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Change Monitor & RAG Knowledge Sync

**Keep your AI knowledge base aligned with the pages it learns from.** Monitor selected documentation, help-center and policy URLs. Get readable before/after differences, stable document IDs, new text chunks to embed, and IDs of old chunks to remove.

No LLM API key is required. This Actor extracts and compares content deterministically; it does not generate speculative AI summaries.

### What you get

- **First run:** clean Markdown and initial chunks for each supported page.
- **Later runs:** `NEW`, `UPDATED`, `UNCHANGED`, `MISSING_PENDING`, `DELETED`, `RESTORED` or `ERROR` for each URL.
- **Useful evidence:** source URL, retrieval time, HTTP status, content hashes, added text and removed text.
- **Incremental RAG updates:** `documentId`, `upsertChunks` and `deleteChunkIds` for your database integration.
- **Conservative deletion:** a page must return HTTP 404/410 on at least two checks before a deletion event is emitted. Timeout, blocked access, rate limiting and server errors never become deletion events.
- **Persistent baselines:** named storage in your Apify account, separate for each monitor name.

### Start in three steps

1. Paste 1–100 exact public HTTPS page URLs. Start with two pages to check compatibility.
2. Choose a monitor name, such as `support-docs`. Run once to create the baseline.
3. Save as a task and schedule non-overlapping runs with the **same monitor name and extraction settings**. Download `Actionable changes (JSON)` or integrate it using the Apify API.

```json
{
  "startUrls": [
    {"url": "https://docs.apify.com/actors/development/actor-definition/actor-json"},
    {"url": "https://docs.apify.com/actors/development/actor-definition/output-schema"}
  ],
  "monitorName": "support-docs",
  "chunkSize": 1500,
  "includeContent": true
}
```

The supplied URLs are checked individually. The Actor does **not** discover links, crawl the entire website or treat a URL removed from the input list as a deleted page.

### Example: a return policy changes

Illustrative abbreviated output; this is not a customer result:

```json
{
  "url": "https://example.com/returns",
  "documentId": "stable-sha256-of-url",
  "changeType": "UPDATED",
  "action": "UPSERT",
  "addedText": "Return unopened products within 14 days.",
  "removedText": "Return unopened products within 30 days.",
  "upsertChunks": [{"id": "new-chunk-id", "index": 0, "text": "Return unopened products within 14 days.", "contentHash": "sha256"}],
  "deleteChunkIds": ["old-chunk-id"]
}
```

### Connect to a vector database or automation

Use the **Actionable changes** output for your Make, n8n, Zapier or custom API workflow. The Actor prepares the update data; your workflow performs database writes and notifications.

| Action | What your integration should do |
| --- | --- |
| `UPSERT` | Delete `deleteChunkIds`, then embed and upsert `upsertChunks`. Use chunk `id` as the vector ID and attach `documentId` and `url` as metadata. |
| `DELETE` | Delete all vectors for `documentId`; `deleteChunkIds` lists previously tracked chunk IDs when available. |
| `NONE` | Make no change. Inspect the check status if it is an error or a pending deletion. |

The default dataset contains **all page checks**, including unchanged pages and errors. `CHANGES` contains only newly actionable events. `SUMMARY` contains counts and the state-store name. Document IDs are stable per normalized input URL. Chunk IDs are based on document ID and content; edited chunks can change IDs. Index is informational; reordering identical chunks may not emit an upsert.

Process successful runs in order. Make downstream writes idempotent using IDs. If a downstream sync fails, replay the original run's `CHANGES` output; the next monitoring run compares against the already checked content, not your database's delivery status. A timed-out or aborted run can have partial results in its dataset even if final JSON outputs are absent.

### Pricing

Launch pricing: **$5 per 1,000 successful page checks** ($0.005 per page), with no Actor startup fee. A successful check includes a new, changed, unchanged, pending-missing or confirmed-missing page. Error rows do not incur a `page-checked` charge. There is no separate charge per chunk or per detected change.

- 10 pages checked daily for 30 days: **$1.50 in Actor event charges**.
- 100 pages checked daily for 30 days: **$15.00 in Actor event charges**.

See the Pricing tab for the authoritative current pricing and any applicable platform costs. Your account's storage usage/retention rules still apply. The Actor observes Apify's maximum run charge and stops when the remaining budget cannot cover another page check.

### Extraction settings

| Input | Default | Purpose |
| --- | --- | --- |
| `startUrls` | Example documentation URLs | Exact HTTPS pages to check; maximum 100. |
| `monitorName` | `default` | Persistent baseline namespace; 1–50 letters, digits, hyphens or underscores. |
| `contentSelector` | Automatic | Optional CSS selector for the content region. |
| `removeSelectors` | Empty | Optional CSS selectors for dynamic sections to exclude. |
| `chunkSize` | 1500 | Chunk length in characters, between 500 and 4000. |
| `includeContent` | true | Include full Markdown for new, updated and restored pages. |

Without a content selector the Actor tries `main`, `[role="main"]`, `article`, then `body`. Navigation, headers, footers, scripts, hidden elements, forms and images are removed. This can exclude content your use case needs: inspect the first-run result before scheduling. To change selectors or chunk size, use a new monitor name and rebuild the downstream baseline.

### Supported pages and limits

- Public HTTPS HTML, plain-text and Markdown pages with useful content available in the HTTP response.
- No login, cookies, CAPTCHA solving, residential proxies, browser JavaScript rendering, PDF or OCR support.
- Respects robots.txt and skips a site when robots rules cannot be verified. Private network addresses and embedded URL credentials are blocked.
- 2 MB response and 150,000 extracted-character limit per page; pages below 50 extracted characters are skipped.
- HTTP 404 and 410 are deletion signals; “soft 404” pages returning HTTP 200 cannot reliably be identified automatically.
- Different locations, personalization, timestamps or site experiments may change content. Use selectors to exclude known noise.
- UTF-8 content is supported. Complex tables, diagrams, images and original page layout are not preserved faithfully.
- Baselines use named key-value storage. Keep it for continuity; removing the baseline starts a fresh first run. Retain only data you need.
- Run one job at a time per monitor name. A best-effort lease rejects an already active monitor; it is not an atomic distributed lock. After an abrupt termination, the lease expires after one hour.

### FAQ

**Does it update my chatbot automatically?** It produces the exact update payloads. Connect an API or automation workflow to apply them to your own knowledge base.

**Why am I charged when nothing changes?** The Actor still fetches, extracts and compares the page, returning a verified unchanged result. You avoid re-embedding unchanged text.

**Why is the first run all NEW?** It establishes the initial baseline. Run again with the same monitor name to detect changes.

**Is this a semantic AI comparison?** No. It compares extracted Markdown and produces deterministic differences. A wording change counts as an update even when its meaning is equivalent.

**Can I use Thai pages?** Yes, UTF-8 Thai text can be extracted and compared. Chunk sizes count characters rather than model tokens, and Thai text without newlines can be split mid-sentence.

**What if every page fails?** The run is marked failed, the dataset explains each error, and no page-check events are billed. Try a narrower selector or a supported server-rendered page.

### Support

Open an issue on this Actor with the run URL, affected public URL and expected behavior. Do not include passwords, tokens or private content. Created by skusol.

# Actor input Schema

## `startUrls` (type: `array`):

1–100 exact public HTTPS page URLs. Links are not crawled automatically. Use pages you are permitted to collect.

## `monitorName` (type: `string`):

Keep this name unchanged for scheduled runs. Use a different name for a separate baseline. Letters, digits, hyphens and underscores only.

## `contentSelector` (type: `string`):

Example: main or .article-body. If empty, selects main, role=main, article or body. An unmatched selector produces an error and preserves the baseline. Changing this requires a new monitor name.

## `removeSelectors` (type: `string`):

CSS selectors to remove before comparing, e.g. .last-updated, .cookie-banner. Navigation, headers, footers, scripts and forms are removed automatically. Changing this requires a new monitor name.

## `chunkSize` (type: `integer`):

Maximum chunk length, with preference for newline boundaries. Changing this requires a new monitor name.

## `includeContent` (type: `boolean`):

Includes the complete extracted Markdown on new, updated and restored pages. Chunk updates and readable differences are always included.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/actors/development/actor-definition/actor-json"
    },
    {
      "url": "https://docs.apify.com/actors/development/actor-definition/output-schema"
    }
  ],
  "monitorName": "default",
  "contentSelector": "",
  "removeSelectors": "",
  "chunkSize": 1500,
  "includeContent": true
}
```

# Actor output Schema

## `checks` (type: `string`):

No description

## `changes` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/actors/development/actor-definition/actor-json"
        },
        {
            "url": "https://docs.apify.com/actors/development/actor-definition/output-schema"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("skusol/website-knowledge-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        { "url": "https://docs.apify.com/actors/development/actor-definition/actor-json" },
        { "url": "https://docs.apify.com/actors/development/actor-definition/output-schema" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("skusol/website-knowledge-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/actors/development/actor-definition/actor-json"
    },
    {
      "url": "https://docs.apify.com/actors/development/actor-definition/output-schema"
    }
  ]
}' |
apify call skusol/website-knowledge-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,skusol/website-knowledge-monitor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/b9exkngRjm7CXfmRL/builds/D074l1c0UmHSKGinN/openapi.json