Website Change Monitor & RAG Knowledge Sync

Keep your AI knowledge base aligned with the pages it learns from. Monitor selected documentation, help-center and policy URLs. Get readable before/after differences, stable document IDs, new text chunks to embed, and IDs of old chunks to remove.

No LLM API key is required. This Actor extracts and compares content deterministically; it does not generate speculative AI summaries.

What you get

• First run: clean Markdown and initial chunks for each supported page.
• Later runs: NEW, UPDATED, UNCHANGED, MISSING_PENDING, DELETED, RESTORED or ERROR for each URL.
• Useful evidence: source URL, retrieval time, HTTP status, content hashes, added text and removed text.
• Incremental RAG updates: documentId, upsertChunks and deleteChunkIds for your database integration.
• Conservative deletion: a page must return HTTP 404/410 on at least two checks before a deletion event is emitted. Timeout, blocked access, rate limiting and server errors never become deletion events.
• Persistent baselines: named storage in your Apify account, separate for each monitor name.

Start in three steps

1. Paste 1–100 exact public HTTPS page URLs. Start with two pages to check compatibility.
2. Choose a monitor name, such as support-docs. Run once to create the baseline.
3. Save as a task and schedule non-overlapping runs with the same monitor name and extraction settings. Download Actionable changes (JSON) or integrate it using the Apify API.

{
  "startUrls": [
    {"url": "https://docs.apify.com/actors/development/actor-definition/actor-json"},
    {"url": "https://docs.apify.com/actors/development/actor-definition/output-schema"}
  ],
  "monitorName": "support-docs",
  "chunkSize": 1500,
  "includeContent": true
}

The supplied URLs are checked individually. The Actor does not discover links, crawl the entire website or treat a URL removed from the input list as a deleted page.

Example: a return policy changes

Illustrative abbreviated output; this is not a customer result:

{
  "url": "https://example.com/returns",
  "documentId": "stable-sha256-of-url",
  "changeType": "UPDATED",
  "action": "UPSERT",
  "addedText": "Return unopened products within 14 days.",
  "removedText": "Return unopened products within 30 days.",
  "upsertChunks": [{"id": "new-chunk-id", "index": 0, "text": "Return unopened products within 14 days.", "contentHash": "sha256"}],
  "deleteChunkIds": ["old-chunk-id"]
}

Connect to a vector database or automation

Use the Actionable changes output for your Make, n8n, Zapier or custom API workflow. The Actor prepares the update data; your workflow performs database writes and notifications.

Action	What your integration should do
UPSERT	Delete deleteChunkIds, then embed and upsert upsertChunks. Use chunk id as the vector ID and attach documentId and url as metadata.
DELETE	Delete all vectors for documentId; deleteChunkIds lists previously tracked chunk IDs when available.
NONE	Make no change. Inspect the check status if it is an error or a pending deletion.

The default dataset contains all page checks, including unchanged pages and errors. CHANGES contains only newly actionable events. SUMMARY contains counts and the state-store name. Document IDs are stable per normalized input URL. Chunk IDs are based on document ID and content; edited chunks can change IDs. Index is informational; reordering identical chunks may not emit an upsert.

Process successful runs in order. Make downstream writes idempotent using IDs. If a downstream sync fails, replay the original run's CHANGES output; the next monitoring run compares against the already checked content, not your database's delivery status. A timed-out or aborted run can have partial results in its dataset even if final JSON outputs are absent.

Pricing

Launch pricing: $5 per 1,000 successful page checks ($0.005 per page), with no Actor startup fee. A successful check includes a new, changed, unchanged, pending-missing or confirmed-missing page. Error rows do not incur a page-checked charge. There is no separate charge per chunk or per detected change.

• 10 pages checked daily for 30 days: $1.50 in Actor event charges.
• 100 pages checked daily for 30 days: $15.00 in Actor event charges.

See the Pricing tab for the authoritative current pricing and any applicable platform costs. Your account's storage usage/retention rules still apply. The Actor observes Apify's maximum run charge and stops when the remaining budget cannot cover another page check.

Extraction settings

Input	Default	Purpose
startUrls	Example documentation URLs	Exact HTTPS pages to check; maximum 100.
monitorName	default	Persistent baseline namespace; 1–50 letters, digits, hyphens or underscores.
contentSelector	Automatic	Optional CSS selector for the content region.
removeSelectors	Empty	Optional CSS selectors for dynamic sections to exclude.
chunkSize	1500	Chunk length in characters, between 500 and 4000.
includeContent	true	Include full Markdown for new, updated and restored pages.

Without a content selector the Actor tries main, [role="main"], article, then body. Navigation, headers, footers, scripts, hidden elements, forms and images are removed. This can exclude content your use case needs: inspect the first-run result before scheduling. To change selectors or chunk size, use a new monitor name and rebuild the downstream baseline.

Supported pages and limits

• Public HTTPS HTML, plain-text and Markdown pages with useful content available in the HTTP response.
• No login, cookies, CAPTCHA solving, residential proxies, browser JavaScript rendering, PDF or OCR support.
• Respects robots.txt and skips a site when robots rules cannot be verified. Private network addresses and embedded URL credentials are blocked.
• 2 MB response and 150,000 extracted-character limit per page; pages below 50 extracted characters are skipped.
• HTTP 404 and 410 are deletion signals; “soft 404” pages returning HTTP 200 cannot reliably be identified automatically.
• Different locations, personalization, timestamps or site experiments may change content. Use selectors to exclude known noise.
• UTF-8 content is supported. Complex tables, diagrams, images and original page layout are not preserved faithfully.
• Baselines use named key-value storage. Keep it for continuity; removing the baseline starts a fresh first run. Retain only data you need.
• Run one job at a time per monitor name. A best-effort lease rejects an already active monitor; it is not an atomic distributed lock. After an abrupt termination, the lease expires after one hour.

FAQ

Does it update my chatbot automatically? It produces the exact update payloads. Connect an API or automation workflow to apply them to your own knowledge base.

Why am I charged when nothing changes? The Actor still fetches, extracts and compares the page, returning a verified unchanged result. You avoid re-embedding unchanged text.

Why is the first run all NEW? It establishes the initial baseline. Run again with the same monitor name to detect changes.

Is this a semantic AI comparison? No. It compares extracted Markdown and produces deterministic differences. A wording change counts as an update even when its meaning is equivalent.

Can I use Thai pages? Yes, UTF-8 Thai text can be extracted and compared. Chunk sizes count characters rather than model tokens, and Thai text without newlines can be split mid-sentence.

What if every page fails? The run is marked failed, the dataset explains each error, and no page-check events are billed. Try a narrower selector or a supported server-rendered page.

Support

Open an issue on this Actor with the run URL, affected public URL and expected behavior. Do not include passwords, tokens or private content. Created by skusol.