Documentation Change Monitor — Markdown & RAG Updates

Keep a documentation index current without processing every unchanged page again. Give this Actor a list of public documentation URLs. The first run produces Markdown snapshots; subsequent runs produce updated content, exact line diffs and confirmed removal records.

Useful for support-bot knowledge bases, developer documentation watchlists and RAG ingestion workflows. No embedding provider, proxy subscription or external API key is needed. This Actor checks explicit URLs; it does not crawl a whole website or execute JavaScript.

Quick start

{
  "urls": [
    "https://docs.apify.com/platform/actors/running/actors-in-store",
    "https://docs.python.org/3/library/asyncio.html"
  ],
  "monitorName": "docs-watchlist",
  "includeUnchanged": false
}

Run once to save a baseline. Reuse monitorName on later runs to compare with that history. You can save a task and schedule it daily in Apify. Use a separate monitor name for each independent downstream index. The Actor itself does not create a schedule.

The default dataset contains page records plus one record_type: "summary" row. If nothing changed, the summary is still returned. The CHANGES key-value record contains only operation: "upsert" and operation: "delete" records; SUMMARY contains every page's status, errors and billing count.

Output and integration

Each document has a stable document_id derived from its requested URL, a content_hash of its Markdown and an event_id for deduplicating a replay. An upsert includes the complete current Markdown. A content edit also includes a unified diff against the last successful snapshot. Diff text is capped at 50,000 characters and diff_truncated explicitly reports that limit; the current Markdown remains complete within the documented page limit.

Change type	Operation	Meaning
baseline	upsert	First observation in this monitor; not a claim that the page was newly published.
changed	upsert	Extracted Markdown changed.
metadata_changed	upsert	Title or final URL changed while the content hash stayed the same.
reappeared	upsert	A previously confirmed missing page is readable again.
missing_pending	none	HTTP 404/410 observed, awaiting confirmation. Retain your indexed content.
removed	delete	At least two HTTP 404/410 checks spanning six hours confirmed unavailability.
unchanged / still_removed	none	No update is required. Included as page rows only when includeUnchanged is true.

Illustrative excerpt using synthetic documentation, not a real vendor change:

{
  "record_type": "page",
  "change_type": "changed",
  "operation": "upsert",
  "url": "https://docs.example.org/quickstart",
  "title": "Quickstart",
  "markdown": "# Quickstart\n\nThe API now accepts a page_size parameter.",
  "diff": "--- previous.md\n+++ current.md\n@@ -1 +1 @@\n-The API accepts a limit parameter.\n+The API now accepts a page_size parameter.",
  "diff_truncated": false
}

For a RAG pipeline, process only rows whose record_type is page. On upsert, replace that document's prior chunks and store its new content hash; re-embed only when content changed. On delete, remove chunks belonging to that document ID. Ignore operation: "none". Chunking, embeddings, vector-database writes and semantic assessments happen in your own pipeline. A textual change is not necessarily a breaking API change.

Consumers should check the run status and SUMMARY, process available successful results, and retry failed pages deliberately. The Actor never sends emails or writes to your vector database.

Extraction

For HTML, the Actor looks for a unique article, main, [role=main], .document, then body. You can choose an exact contentSelector and remove volatile content with excludeSelectors. Scripts, navigation, footers, sidebars, forms and buttons are excluded. Markdown conversion retains headings, links, lists, tables and code blocks; source layout can affect the result.

Markdown and UTF-8 plain-text responses are supported directly. CSS selectors cannot be applied to them. Sites that return a JavaScript shell, login page or access challenge fail the check instead of becoming a new baseline. Very short content is rejected. A page with at least 500 prior characters that shrinks by more than 80% is held for inspection; use a new monitor name to rebaseline if that shrinkage was intentional.

HTTP conditional requests use ETag or Last-Modified where available. A 304 counts as a successful unchanged check and reuses the retained snapshot. Content is still fetched and compared if the source does not support validators. Extraction is deterministic text comparison, not an AI judgment of meaning.

Persistent history and removals

State is kept in a named store in your own Apify account. Runs using the same monitor are serialized with a queue lock. stateStoreId is optional and normally left blank. Changing content selectors, exclusions or the minimum content length requires a fresh monitor name so parsing changes cannot manufacture edits.

Removing a URL from your input does not delete its history or emit a deletion. A missing page must return 404 or 410 on at least two checks spanning six hours. Timeouts, 429s, server failures and rejected extractions do not advance that page's baseline. This reduces false removals but cannot prove that an upstream server's response is correct or permanent.

Up to 500 distinct URLs may be retained per monitor, subject to the state-size limit. There is no automatic expiration of page history. Deleting the named state store resets the baseline. State storage is retained in your account and follows Apify's storage/retention rules.

Billing

One page-check event is charged per successfully processed URL at the price shown on the Store page. Baseline, changed, unchanged and valid 304 checks count. A 404/410 for an already tracked page also counts because it contributes removal evidence. A 404/410 with no prior snapshot fails and is not charged.

Failed, blocked, oversized or budget-skipped pages do not produce a page-check fee. There are no separate fees for Markdown, diffs, summary rows or change records. Successful pages in a mixed-failure run are charged and retain their results. Use Apify's maximum charge setting to cap billable checks.

Output, billing and state storage are separate operations. A crash can leave partial output, replay records or a charged check whose state update did not complete. Deduplicate by event_id; upserts using document_id should be idempotent. No exactly-once settlement guarantee is made.

Supported scope and limits

• 1–50 explicit public HTTPS URLs per run, port 443, no credentials or custom authentication headers.
• Same-host redirects only; enter the final URL for cross-host redirects. URLs differing by query string are different documents; fragments are ignored.
• Public network destinations only. DNS is checked and the selected address pinned while TLS verifies the hostname.
• robots.txt is honored, including wildcard rules. Site requests are sequential with a minimum one-second spacing; crawl delays above 15 seconds are unsupported.
• Response cap: 3 MiB; extracted Markdown cap: 150,000 characters per page; retained state cap: 20 MiB. No silent truncation of a content snapshot.
• The Actor stops starting checks after eight minutes. Unfinished URLs are reported as skipped and the run fails after preserving successful results. Use smaller batches for slower sites.
• No browser rendering, PDF/OCR, private knowledge bases, historical changes from before your first run, or full-site discovery. Automatic extraction cannot identify every soft error page; review representative output before connecting an index.

Independent tool maintained by Apter Enterprises. No affiliation with the documentation publishers is implied. Use the Store Issues tab for a reproducible extraction problem without posting credentials or private content.