Docs Change Feed

Monitor documentation and knowledge-base pages and get clean Markdown only when meaningful content changes. Docs Change Feed is a stateful website change monitor built for AI agents, RAG pipelines, developer documentation, changelogs, policies, help centers, and API docs.

Give it one or more public URLs. The Actor extracts readable content, normalizes it to Markdown, hashes the result, remembers the previous version, and emits a dataset item when a page is new or changed. No LLM or external AI key is required.

What you can automate

• Refresh RAG knowledge only when sources change instead of re-ingesting the same pages every run.
• Monitor API and developer documentation for updates that may affect integrations.
• Track help centers, policies, changelogs, and product docs on an Apify schedule.
• Feed clean Markdown change events into agents, webhooks, Make, Zapier, n8n, or your own API.
• Ignore noisy page elements such as timestamps or rotating banners with CSS selectors.

Quick start

Use a single documentation page:

{
  "urls": ["https://docs.example.com/guide"]
}

Or monitor several pages:

{
  "urls": [
    "https://docs.example.com/guide",
    "https://docs.example.com/api"
  ],
  "emitOnlyChanged": true,
  "includeMarkdown": true
}

On the first successful check the page is reported as new. On later runs, unchanged pages are suppressed by default. When the normalized content changes, the Actor emits a changed result with both the new and previous SHA-256 hashes.

Output data

Each result can contain:

• source, final, and canonical URL;
• page title;
• new, changed, unchanged, or error status;
• current and previous SHA-256 content hash;
• first-seen, checked, and previous-change timestamps;
• HTTP status, ETag, and Last-Modified metadata;
• normalized Markdown for new or changed pages.

This makes the dataset useful both as a human-readable change log and as a machine-readable trigger for downstream automation.

Stateful change detection

Docs Change Feed stores the last known page state in an Apify key-value store. Later runs send conditional HTTP headers when the server supports them:

• If-None-Match
• If-Modified-Since

A 304 Not Modified response avoids downloading and parsing the document again. Otherwise, the Actor extracts and normalizes the content before comparing its SHA-256 hash with the previous version.

The state store is configurable, so separate monitoring workflows can keep independent histories.

Input options

Documentation URLs

urls is the list of public HTTP or HTTPS pages to monitor.

Emit only new or changed pages

emitOnlyChanged defaults to true. Set it to false when you also want an unchanged dataset record on every check.

Include Markdown

includeMarkdown defaults to true. Disable it when you only need status and hash metadata.

Extraction mode

Use readability for the main readable article/documentation content. Use body when the relevant content is spread across the full page body.

Ignore volatile elements

Use removeSelectors for CSS selectors that should not affect change detection:

{
  "removeSelectors": [
    ".last-updated",
    ".rotating-banner",
    "[data-build-time]"
  ]
}

Persistent state store

stateStoreName lets different jobs maintain independent histories. The default is docs-change-feed-state.

Run on a schedule or through the API

For continuous documentation monitoring, create an Apify schedule such as hourly or daily. Each scheduled run reuses the persistent state and returns only the events you asked for.

You can also call the Actor through the Apify API or integrate its dataset with automation platforms and AI-agent workflows. The default dataset view exposes compact change metadata; the full dataset item contains the detailed fields produced by the run.

Pricing

The Actor uses Pay Per Event pricing:

• $0.001 per page check;
• Apify's Actor-start event is $0.00005 per run.

The Actor checks the caller's maximum-charge budget before each URL. Large URL lists therefore remain bounded by the maximum cost configured for the run.

Reliability and noise reduction

Content is normalized before hashing to reduce false positives from insignificant whitespace. For site-specific volatile elements, add removeSelectors.

HTTP validators are used when available, but not every website supplies reliable ETag or Last-Modified headers. Hash comparison remains the final content-change check.

Limits and expected behavior

• Intended for public HTML documentation and content pages.
• It is not a browser automation Actor and does not execute complex client-side applications before extraction.
• Pages requiring login, CAPTCHA, private network access, or anti-bot bypass are outside the intended use case.
• A site can change markup in ways that affect extracted content even when the underlying article meaning is similar.
• Per-page timeout is configurable from 1 to 120 seconds.

Privacy

The Actor fetches only URLs supplied in the run input. It does not send page content to an LLM, advertising network, analytics service, or third-party AI provider.

Run data and datasets follow the privacy and retention settings of your Apify account.

FAQ

Can I use this for RAG document refresh?

Yes. Run it on a schedule and trigger re-embedding or re-indexing only for new and changed items.

Can I monitor several documentation sites at once?

Yes. Put all target pages in urls. Each URL keeps its own stored state.

Why did a page trigger even though the text looks similar?

Inspect the emitted Markdown and add unstable elements to removeSelectors. Build timestamps, navigation counters, or rotating banners are common causes.

Does it need OpenAI or another AI API key?

No. Extraction, normalization, hashing, persistence, and comparison are deterministic and do not require an LLM.

Can I get an output item for unchanged pages?

Yes. Set emitOnlyChanged to false.