# AI-Ready Website Change Feed - Chunked Diffs for RAG Pipelines (`mibedk/website-diff-rag-feed`) Actor

Crawls a list of URLs or a sitemap on a schedule and emits only the content chunks that are new, changed, or removed since the previous run - token-windowed with stable chunk IDs and content hashes, ready for direct vector-database upsert/delete.

- **URL**: https://apify.com/mibedk/website-diff-rag-feed.md
- **Developed by:** [Mikkel Bech-Hansen](https://apify.com/mibedk) (community)
- **Categories:** AI, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $6.00 / 1,000 content chunk changeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI-Ready Website Change Feed - Chunked Diffs for RAG Pipelines

Give this actor a list of URLs (or a sitemap), run it on a schedule, and each run it
tells you exactly what changed since last time - not the whole page again, just the
individual content chunks that are new, updated, or removed, each with a stable ID and
a content hash. Point the output at your vector database's upsert/delete API and your
RAG index stays in sync without re-embedding pages that didn't change.

### How it works

1. You supply `startUrls` and/or `sitemapUrls`. Every resolved page URL gets fetched
   with a plain HTTP request (no browser).
2. Boilerplate (nav, footer, ads, cookie banners, etc.) is stripped and the main content
   is split into overlapping, token-windowed chunks (default: 500 tokens, 50 overlap).
3. Each chunk gets a SHA-256 content hash and a stable `chunkId` (derived from the URL
   and its position on the page).
4. The actor compares this run's chunks against the hashes it stored last time (in a
   named key-value store, so it persists across scheduled runs) and only pushes a
   dataset record for chunks that are **new**, **updated**, or **removed**. Unchanged
   chunks produce no record and cost nothing.

The very first run against any page has nothing to compare against, so every chunk on
that run is reported as `new` - that's the baseline your vector DB ingests once.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `startUrls` | array of `{url}` | 2 example Crawlee docs pages | Pages to track. Pre-filled so the actor works with zero configuration - replace with your own pages. |
| `sitemapUrls` | array of strings | `[]` | Sitemap URLs to expand into page URLs. One level of nested `<sitemapindex>` is followed (capped at 20 nested sitemaps). |
| `maxPagesPerRun` | integer | `20` | Cap on combined `startUrls` + sitemap-expanded URLs per run (1-500). |
| `chunkTokens` | integer | `500` | Target chunk size in GPT-style (cl100k_base) tokens (50-2000). |
| `chunkOverlapTokens` | integer | `50` | Tokens shared between consecutive chunks (0-500); auto-capped below `chunkTokens`. |
| `forceFullRescan` | boolean | `false` | Ignore stored hashes and report every chunk as new. Use for an initial vector-DB backfill or after changing chunk size/overlap. |
| `stateKeyValueStoreName` | string | `website-diff-rag-feed-state` | Name of the persistent store holding chunk hashes between runs. Keep it identical across scheduled runs of the same site to get real diffs; use a different name to track a second site independently. |
| `requestTimeoutSecs` | integer | `30` | Per-page request timeout (5-120s). |
| `maxRequestRetries` | integer | `3` | Retries per page before it's treated as failed (0-10). |
| `proxyConfiguration` | object | no proxy | Optional. Most sites work fine with no proxy; enable only if a target blocks Apify's data-center IPs. |

### Output

One dataset record per changed chunk:

```json
{
  "chunkId": "b07ee562b3a1e3c8",
  "url": "https://example.com/docs/quick-start",
  "chunkIndex": 0,
  "changeType": "new",
  "contentHash": "bb5e6f5042f7dacb0fea260760dd4e301e752d3dfac8a4463748a7257f0cd2c",
  "previousContentHash": null,
  "tokenCount": 500,
  "text": "Quick Start With this short tutorial you can start scraping in a minute or two...",
  "pageTitle": "Quick Start | Example Docs",
  "checkedAt": "2026-08-19T23:25:25.713Z"
}
```

`changeType` is `"new"`, `"updated"`, or `"removed"`. On `"removed"`, `contentHash` is
`null` and `text` is empty - use `chunkId` to delete that vector from your index.
`previousContentHash` is set on `"updated"`/`"removed"` records so you can confirm which
version you're replacing.

### Cost guidance

Pay-per-event: **$0.001 per page checked** + **$0.006 per chunk delivered** (new,
updated, or removed - unchanged chunks are free). A daily run tracking 20 pages where 2
pages change a couple of chunks each costs roughly **$0.04-0.05/run**; a first-run
baseline over 20 pages (~5 chunks/page, all reported new) costs roughly **$0.60-0.70**.
See `MONETIZATION.md` for the full breakdown.

### Known limitations

- **No JavaScript rendering.** This actor uses a plain HTTP crawler, not a browser.
  Pages whose real content only appears after client-side JavaScript runs will not diff
  correctly - only the static HTML is checked.
- **Content extraction is a heuristic, not full Readability.** It strips common
  boilerplate selectors and picks the first plausible `main`/`article` container. Pages
  with unusual layouts may retain some boilerplate (e.g. an in-page table of contents)
  or, rarely, miss real content if it sits outside the detected main container.
- **Chunk boundaries are positional, not structural.** Chunks are windows over the
  page's token stream. Inserting a paragraph near the top of a page shifts every later
  chunk's boundaries, so they'll show up as `"updated"` even though their underlying
  content just moved - not a text-diff false positive, but not free of noise either.
- **Only crawls the exact URLs you give it.** It does not follow on-page links, so
  `startUrls`/`sitemapUrls` need to actually cover the pages you want tracked.
- **Sitemap fetches bypass the configured proxy** (they use a direct HTTP request);
  only the page crawl itself respects `proxyConfiguration`.
- **Removal detection is per-tracked-URL, not automatic for dropped input.** A page is
  reported `"removed"` only when it's still in your input but fails to fetch (e.g. a
  real 404). If you simply delete a URL from `startUrls`, its old chunks stay in the
  state store untouched until you either re-add it or it starts erroring.
- Built for tens to a few hundred tracked pages. State is one key-value record per URL,
  which is efficient at that scale but isn't a general-purpose datastore for thousands
  of pages.

### Legal

This actor is a generic web-content crawler: it fetches whatever URLs you configure it
with. It is not affiliated with, endorsed by, or sponsored by any specific website or
platform.

Fetching a page's public HTML is not the same as that site's Terms of Service permitting
automated access - many sites' ToS restrict scraping or automated monitoring even for
content that's publicly visible, regardless of what this actor's default configuration
allows it to technically do. You are responsible for checking the terms of service,
`robots.txt`, and applicable law for every URL and domain you point this actor at, and
for how you use and redistribute the content it returns (including into a downstream
vector database or RAG pipeline). Only track pages you have the right to crawl.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to track. Every run re-fetches each URL here and diffs its content chunks against the previous run's stored hashes.

## `sitemapUrls` (type: `array`):

One or more sitemap.xml URLs. Every <loc> page URL found is added to the crawl list (one level of nested sitemap-index is followed), up to 'Max pages per run'.

## `maxPagesPerRun` (type: `integer`):

Stop after fetching this many pages, combining Start URLs and sitemap-expanded URLs. Keeps a large sitemap from blowing the run budget.

## `chunkTokens` (type: `integer`):

Target size of each content chunk, measured in GPT-style (cl100k_base) tokens. Matches common embedding-model input windows.

## `chunkOverlapTokens` (type: `integer`):

How many tokens consecutive chunks share, to avoid losing context at a chunk boundary. Must be smaller than 'Chunk size (tokens)'; it is automatically capped if not.

## `forceFullRescan` (type: `boolean`):

Treat every chunk on every page as new, ignoring previously stored hashes. Use for the first backfill into a vector database, or after a chunking-config change. The stored state is still overwritten with this run's results.

## `stateKeyValueStoreName` (type: `string`):

Name of the named key-value store used to persist chunk hashes between runs, so this actor can diff against 'last time'. Keep this the same across scheduled runs of the same site to get real diffs. Use a different value to track a second site independently, or to force a clean baseline.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for each page to respond before treating it as failed and retrying.

## `maxRequestRetries` (type: `integer`):

How many times to retry a page after a failed fetch before giving up on it and marking its previously known chunks as removed.

## `proxyConfiguration` (type: `object`):

Optional. Most sites work fine with no proxy. Turn this on only if a target site blocks Apify's data-center IPs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    },
    {
      "url": "https://crawlee.dev/js/docs/introduction"
    }
  ],
  "sitemapUrls": [],
  "maxPagesPerRun": 20,
  "chunkTokens": 500,
  "chunkOverlapTokens": 50,
  "forceFullRescan": false,
  "stateKeyValueStoreName": "website-diff-rag-feed-state",
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `chunkDiffs` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/js/docs/quick-start"
        },
        {
            "url": "https://crawlee.dev/js/docs/introduction"
        }
    ],
    "sitemapUrls": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("mibedk/website-diff-rag-feed").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://crawlee.dev/js/docs/quick-start" },
        { "url": "https://crawlee.dev/js/docs/introduction" },
    ],
    "sitemapUrls": [],
}

# Run the Actor and wait for it to finish
run = client.actor("mibedk/website-diff-rag-feed").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    },
    {
      "url": "https://crawlee.dev/js/docs/introduction"
    }
  ],
  "sitemapUrls": []
}' |
apify call mibedk/website-diff-rag-feed --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mibedk/website-diff-rag-feed"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vCxFVLn0MnmKpRs38/builds/XofiG4VVu7jMSZRJ0/openapi.json