AI-Ready Website Change Feed - Chunked Diffs for RAG Pipelines

Give this actor a list of URLs (or a sitemap), run it on a schedule, and each run it tells you exactly what changed since last time - not the whole page again, just the individual content chunks that are new, updated, or removed, each with a stable ID and a content hash. Point the output at your vector database's upsert/delete API and your RAG index stays in sync without re-embedding pages that didn't change.

How it works

1. You supply startUrls and/or sitemapUrls. Every resolved page URL gets fetched with a plain HTTP request (no browser).
2. Boilerplate (nav, footer, ads, cookie banners, etc.) is stripped and the main content is split into overlapping, token-windowed chunks (default: 500 tokens, 50 overlap).
3. Each chunk gets a SHA-256 content hash and a stable chunkId (derived from the URL and its position on the page).
4. The actor compares this run's chunks against the hashes it stored last time (in a named key-value store, so it persists across scheduled runs) and only pushes a dataset record for chunks that are new, updated, or removed. Unchanged chunks produce no record and cost nothing.

The very first run against any page has nothing to compare against, so every chunk on that run is reported as new - that's the baseline your vector DB ingests once.

Input

Field	Type	Default	Notes
startUrls	array of {url}	2 example Crawlee docs pages	Pages to track. Pre-filled so the actor works with zero configuration - replace with your own pages.
sitemapUrls	array of strings	[]	Sitemap URLs to expand into page URLs. One level of nested <sitemapindex> is followed (capped at 20 nested sitemaps).
maxPagesPerRun	integer	20	Cap on combined startUrls + sitemap-expanded URLs per run (1-500).
chunkTokens	integer	500	Target chunk size in GPT-style (cl100k_base) tokens (50-2000).
chunkOverlapTokens	integer	50	Tokens shared between consecutive chunks (0-500); auto-capped below chunkTokens.
forceFullRescan	boolean	false	Ignore stored hashes and report every chunk as new. Use for an initial vector-DB backfill or after changing chunk size/overlap.
stateKeyValueStoreName	string	website-diff-rag-feed-state	Name of the persistent store holding chunk hashes between runs. Keep it identical across scheduled runs of the same site to get real diffs; use a different name to track a second site independently.
requestTimeoutSecs	integer	30	Per-page request timeout (5-120s).
maxRequestRetries	integer	3	Retries per page before it's treated as failed (0-10).
proxyConfiguration	object	no proxy	Optional. Most sites work fine with no proxy; enable only if a target blocks Apify's data-center IPs.

Output

One dataset record per changed chunk:

{
  "chunkId": "b07ee562b3a1e3c8",
  "url": "https://example.com/docs/quick-start",
  "chunkIndex": 0,
  "changeType": "new",
  "contentHash": "bb5e6f5042f7dacb0fea260760dd4e301e752d3dfac8a4463748a7257f0cd2c",
  "previousContentHash": null,
  "tokenCount": 500,
  "text": "Quick Start With this short tutorial you can start scraping in a minute or two...",
  "pageTitle": "Quick Start | Example Docs",
  "checkedAt": "2026-08-19T23:25:25.713Z"
}

changeType is "new", "updated", or "removed". On "removed", contentHash is null and text is empty - use chunkId to delete that vector from your index. previousContentHash is set on "updated"/"removed" records so you can confirm which version you're replacing.

Cost guidance

Pay-per-event: $0.001 per page checked + $0.006 per chunk delivered (new, updated, or removed - unchanged chunks are free). A daily run tracking 20 pages where 2 pages change a couple of chunks each costs roughly $0.04-0.05/run; a first-run baseline over 20 pages (~5 chunks/page, all reported new) costs roughly $0.60-0.70. See MONETIZATION.md for the full breakdown.

Known limitations

• No JavaScript rendering. This actor uses a plain HTTP crawler, not a browser. Pages whose real content only appears after client-side JavaScript runs will not diff correctly - only the static HTML is checked.
• Content extraction is a heuristic, not full Readability. It strips common boilerplate selectors and picks the first plausible main/article container. Pages with unusual layouts may retain some boilerplate (e.g. an in-page table of contents) or, rarely, miss real content if it sits outside the detected main container.
• Chunk boundaries are positional, not structural. Chunks are windows over the page's token stream. Inserting a paragraph near the top of a page shifts every later chunk's boundaries, so they'll show up as "updated" even though their underlying content just moved - not a text-diff false positive, but not free of noise either.
• Only crawls the exact URLs you give it. It does not follow on-page links, so startUrls/sitemapUrls need to actually cover the pages you want tracked.
• Sitemap fetches bypass the configured proxy (they use a direct HTTP request); only the page crawl itself respects proxyConfiguration.
• Removal detection is per-tracked-URL, not automatic for dropped input. A page is reported "removed" only when it's still in your input but fails to fetch (e.g. a real 404). If you simply delete a URL from startUrls, its old chunks stay in the state store untouched until you either re-add it or it starts erroring.
• Built for tens to a few hundred tracked pages. State is one key-value record per URL, which is efficient at that scale but isn't a general-purpose datastore for thousands of pages.

Legal

This actor is a generic web-content crawler: it fetches whatever URLs you configure it with. It is not affiliated with, endorsed by, or sponsored by any specific website or platform.

Fetching a page's public HTML is not the same as that site's Terms of Service permitting automated access - many sites' ToS restrict scraping or automated monitoring even for content that's publicly visible, regardless of what this actor's default configuration allows it to technically do. You are responsible for checking the terms of service, robots.txt, and applicable law for every URL and domain you point this actor at, and for how you use and redistribute the content it returns (including into a downstream vector database or RAG pipeline). Only track pages you have the right to crawl.