# Website Change Tracker | Page Change Detection, RAG Sync (`aurelix/website-change-monitor`) Actor

Extract website changes for RAG sync: only pages added, changed or deleted since your last run, as Markdown chunks with stable IDs to upsert and chunk IDs to delete, not line or section diffs. $1.50 per 1,000 changed pages + $0.50 per 1,000 checked.

- **URL**: https://apify.com/aurelix/website-change-monitor.md
- **Developed by:** [Aurelix](https://apify.com/aurelix) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.05 / 1,000 page changes

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What is Website Change Tracker?

**Keep a RAG index in sync: only pages added, changed or deleted since your last run, as Markdown chunks with stable IDs, for $1.50 per 1,000 changed pages, no login.**

For AI teams keeping a knowledge base current. Each checked page costs $0.50 per 1,000 on top; menus, footers, cookie banners and 'last updated' lines never count as changes. Your first run with the example input costs about $0.043 (20 changed pages in our test run).

[![Website Change Tracker: example output from a real run](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/website-change-monitor-card-v2.png)](https://console.apify.com/actors/c9coCX7iJRFP8kLsX?addFromActorId=c9coCX7iJRFP8kLsX)

| | |
|---|---|
| **Start from** | Start URLs or sitemap URLs, plus a corpus ID |
| **Returns** | Only added, changed and deleted pages, with chunks to upsert and chunk IDs to delete |
| **IDs** | Stable document ID (canonical URL) and content-anchored chunk IDs, valid UUIDs |
| **Automate** | Schedules, webhook push of the delta, Zapier, Make, n8n, MCP for AI agents |
| **Price** | $1.50 per 1,000 changed pages ($1.05 per 1,000 changed pages on paid Apify plans) |
| **Login needed** | No |

### Why Website Change Tracker users pick this one

- **Re-embed only what changed.** Each run returns just the pages that changed.
- **Stable chunk IDs.** Anchored on heading path and text, so edits elsewhere don't move them.
- **Explicit deletions.** Chunk IDs to delete for edited and removed pages.
- **Webhook delta.** Chunks to upsert and IDs to delete, pushed in batches.
- **No false changes.** Broken or blocked pages are flagged, never mistaken for deletions.

### Get started in 3 steps

1. Paste start URLs (or a sitemap URL), choose a corpus ID such as 'acme-docs' and set Max pages above the size of the site.
2. Run it once: the baseline returns every page as 'added' with all its chunks (or switch 'Return the baseline' off to record it silently).
3. Schedule the same input (daily or weekly). Each run returns only added, changed and deleted pages; upsert 'upsert_chunks' and delete 'delete_chunk_ids' in your vector database, by hand, through the API, a webhook or n8n (see the FAQ below).

**[▶ Try it with the example input](https://console.apify.com/actors/c9coCX7iJRFP8kLsX?addFromActorId=c9coCX7iJRFP8kLsX)**

The form opens with an example input already filled in: that run costs about $0.043 on the free plan.

### How much does it cost to track website changes?

Pay per event: you pay only for what the run returns, plus a small start fee. Bronze, Silver, Gold and higher Apify plans pay less per result.

| Changed pages | Free plan | Paid plans (Bronze, Silver, Gold) |
|---|---|---|
| 100 | $0.15 | $0.11 |
| 1,000 | $1.50 | $1.05 |
| 10,000 | $15.00 | $10.50 |

Totals include the $0.003 start fee per run, rounded to the cent. Apify's free plan includes $5 of credit a month, enough for about 3,300 changed pages (or, with checks at $0.50 per 1,000, a 250-page docs site checked daily for a month).

Every returned page was also checked, so 1,000 changed pages cost $2.00 in all on the free plan ($1.50 + $0.50), and a run with no changes costs only the checks and the start fee. Not charged: extraction failures (a known page that came back empty, blocked, JavaScript-only or broken; flagged as 'extraction_failed' for free and its previous version kept), pages blocked by robots.txt, fetch errors, non-HTML responses and duplicates. A page check is charged only when its change record also fits under the maximum charge you set, so nothing is paid for a change you did not receive; changes not returned because of the maximum charge come in the next run.

#### Billing events

| Event | Free plan | Paid plans (Bronze, Silver, Gold) |
|---|---|---|
| `page-change` | $1.50 per 1,000 changed pages | $1.05 per 1,000 changed pages |
| `page-checked` | $0.50 per 1,000 pages checked | $0.50 per 1,000 pages checked |
| `apify-actor-start` | $0.003 per run | $0.003 per run |

### Website Change Tracker vs alternatives

| | **This Actor** | ryanclinton/website-change-monitor | muhammad-bilal/web-drift-detector |
|---|---|---|---|
| **Price per 1,000, free plan** | $0.50 per 1,000 checked + $1.50 per 1,000 changed pages | $100 per 1,000 sites monitored | $0.20 per 1,000 checks + $0.60 per 1,000 results |
| **Price per 1,000, paid plans** | $0.50 per 1,000 checked + $1.05 per 1,000 changed | $47 per 1,000 sites monitored | $0.20 per 1,000 checks + $0.60 per 1,000 results |
| **Start fee per run** | $0.003 | $0.0002 (at 4 GB) | $0.0002 (at 4 GB) |
| **Platform usage billed on top** | No | No | No |
| **Markdown chunks to upsert, stable chunk IDs** | ✓ | ✗ (line diffs) | ✗ (section diffs) |
| **Chunk IDs to delete for edited and removed pages** | ✓ | ✗ | not stated |
| **Blocked or empty pages never reported as deleted** | ✓ | not stated | not stated |
| **Delta pushed to a webhook** | ✓ chunks and delete IDs | Change alerts | not stated |
| **Severity scores for human alerts** | ✗ | ✓ | ✓ |

Prices checked regularly on each Actor's Store page.

### Benchmarks: how we compare

| Test | Input | Result | Run time |
|---|---|---|---|
| Baseline | docs.apify.com/academy, Max pages 20 | 20 pages recorded as 284 chunks | 22 s |
| Baseline | docs.python.org/3/tutorial/, Max pages 100 | 17 pages recorded as 300 chunks | 20 s |
| Re-check with no edits | the same Python tutorial corpus, run again | 17 pages checked, 0 added, 0 changed, 0 deleted | 16 s |
| Controlled edit and deletion | an Apify Academy corpus with one stored chunk made stale and one stored page that now returns 404 | exactly the expected result: 1 changed page (1 chunk to upsert, the stale ID to delete) and 1 deleted page; 18 unchanged and 6 new pages | 24 s |
| Busy page | Wikipedia's Current events portal (83 chunks), re-checked 3 times in a row | unchanged each time: no false changes from layout or menus | 3 to 5 s |
| Slow site | a site asking for a 6-second Crawl-delay, Max pages 8 | 8 pages recorded, delay honoured | 62 s |
| Nothing readable | 4 silent hosts, an unknown host and a site asking for a 30-second Crawl-delay | nothing recorded and no page charges; only the start fee | 28 to 32 s |

Apify cloud runs at the default 256 MB with plain HTTP and no proxy, one run per row. The controlled test edited the stored corpus record before the run and compared the run's output with the expected chunk IDs. Pages are fetched at most two a second per site, so a run takes at least half a second per page.

### What data you get

**Website Change Tracker** keeps a website's content in sync with your RAG index: first run = baseline (every page 'added'); later runs only what changed. The first run with a corpus ID records every page as Markdown chunks; every later run re-checks the site and returns only the pages that were added, changed or deleted, each with the exact chunks to upsert and the chunk IDs to delete. Chunk IDs are anchored on content (page URL, heading path and a hash of the normalized text), so an unchanged section keeps its ID when the sections around it change, and you only re-embed what really changed. Navigation, footers, cookie banners and 'last updated' lines are ignored, and a page that comes back empty, blocked or broken is flagged, never reported as deleted.

- change: added, changed, deleted, or extraction_failed (free; the page could not be read this run and its previous version is kept)
- doc_id, url (canonical), fetched_url, title, reason (why deleted or unreadable)
- upsert_chunks: \[{chunk_id, position, heading_path, text, char_count, url, citation}] to embed and upsert
- delete_chunk_ids: chunk IDs to remove from your index; upsert_count, delete_count, unchanged_chunk_count, chunk_count
- markdown (whole main content of added and changed pages), word_count, language, last_modified, http_status
- first_seen_at, previous_check_at, scraped_at (ISO 8601 UTC), corpus_id; plus a SUMMARY record per run with all counts

| change | url | title | upsert_count | delete_count | unchanged_chunk_count |
|---|---|---|---|---|---|
| added | https://docs.apify.com/academy | Apify Academy | Academy | Apify Documentation | 9 | 0 | 0 |
| added | https://docs.apify.com/academy/advanced-web-scraping | Advanced web scraping | Academy | Apify Documentation | 3 | 0 | 0 |
| added | https://docs.apify.com/academy/tutorials | What's this section? | Academy | Apify Documentation | 1 | 0 | 0 |
| added | https://docs.apify.com/academy/scraping-with-ai | Web scraping with AI and Apify | Academy | Apify Docume... | 9 | 0 | 0 |
| added | https://docs.apify.com/academy/scraping-basics-javascript | Web scraping basics for JavaScript devs | Academy | Api... | 22 | 0 | 0 |

One full changed page record, exactly as a real run returned it (long text and lists shortened here):

```json
{
  "corpus_id": "apify-academy-docs",
  "change": "added",
  "doc_id": "d67cabce-5772-5274-acc5-2b5cf3bf4520",
  "url": "https://docs.apify.com/academy",
  "fetched_url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "reason": null,
  "upsert_count": 9,
  "delete_count": 0,
  "unchanged_chunk_count": 0,
  "chunk_count": 9,
  "upsert_chunks": [
    {
      "chunk_id": "2f2b178a-262e-52a2-8e63-40fe11712104",
      "position": 0,
      "heading_path": [
        "Apify Academy"
      ],
      "text": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper devel... [5 more characters]",
      "char_count": 125,
      "url": "https://docs.apify.com/academy",
      "citation": "Apify Academy (https://docs.apify.com/academy)"
    },
    {
      "chunk_id": "e7b0e55b-909b-5bdf-824a-d81f49cdfe9f",
      "position": 1,
      "heading_path": [
        "Apify Academy",
        "Beginner courses",
        "Web scraping with AI"
      ],
      "text": "Learn how to use AI to extract information from websites in this practical course, starting from the absolute basics.\n\n[... [54 more characters]",
      "char_count": 174,
      "url": "https://docs.apify.com/academy",
      "citation": "Apify Academy > Beginner courses > Web scraping with AI (https://docs.apify.com/academy)"
    },
    {
      "chunk_id": "3fa16237-c166-577f-9709-514f29e55a77",
      "position": 2,
      "heading_path": [
        "Apify Academy",
        "Beginner courses",
        "Web scraping basics with JS"
      ],
      "text": "Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute ba... [72 more characters]",
      "char_count": 192,
      "url": "https://docs.apify.com/academy",
      "citation": "Apify Academy > Beginner courses > Web scraping basics with JS (https://docs.apify.com/academy)"
    },
    "... [6 more]"
  ],
  "delete_chunk_ids": [],
  "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an exp... [2378 more characters]",
  "word_count": 377,
  "language": "en",
  "last_modified": "2026-11-04T21:51:26Z",
  "http_status": 200,
  "first_seen_at": "2026-11-04T21:57:10Z",
  "previous_check_at": null,
  "source_url": "https://docs.apify.com/academy",
  "scraped_at": "2026-11-04T21:57:10Z"
}
```

### Use cases

- Keeping a RAG chatbot or support assistant in sync with a help center or documentation site
- Updating only changed embeddings in Pinecone, Qdrant, Weaviate, Chroma or pgvector on a schedule
- Feeding AI agents a daily 'what changed in these docs' digest with source citations
- Monitoring competitor docs, policy or pricing pages for real content changes, without boilerplate noise

### Use-case recipes

#### Keep a docs RAG index in sync

For AI developers. Run once with a new `corpusId` to fill the index, then schedule the same input daily. For each record, embed and upsert every item of `upsert_chunks` with `chunk_id` as the vector ID and `citation` as metadata, delete every ID in `delete_chunk_ids`, and skip `extraction_failed` records.

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "corpusId": "apify-academy",
  "maxPages": 50,
  "chunkSize": 1500
}
```

#### Start tracking after a full copy

For teams that already loaded the site with Website to Markdown. Turn off `returnBaseline` so the first run only records the corpus and you pay per page checked, not per page returned. From the second run on, you get only the changes.

```json
{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "corpusId": "py-tutorial",
  "maxPages": 50,
  "returnBaseline": false
}
```

#### Push changes to n8n or your own endpoint

For automation builders. Set `webhookUrl` to an n8n, Make or Zapier webhook. Each POST holds up to 25 change records; when `final` is true, `summary` has the run's counts, so the workflow knows the sync is complete.

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "corpusId": "apify-academy-hook",
  "maxPages": 30,
  "webhookUrl": "https://example.com/hooks/rag-sync"
}
```

#### Track a whole help center, including orphan pages

For support teams. Use `scope` = `domain` and turn on `useSitemaps`, so pages no menu links to are tracked too. Exclude listings that change on every visit, such as search and tag pages, and set `maxPages` above the size of the site.

```json
{
  "startUrls": [
    "https://example.com/help/"
  ],
  "corpusId": "help-center",
  "maxPages": 100,
  "scope": "domain",
  "useSitemaps": true,
  "excludeGlobs": [
    "**/search**",
    "**/tags/**"
  ]
}
```

### Input

| Field | Type | What it does | Default |
|---|---|---|---|
| `startUrls` (Start URLs or sitemaps) | array | Pages to start from, e.g. \['https://docs.apify.com/academy'], or sitemap URLs ending in .xml, e.g. \['https://example.com/sitemap.xml']. | \["https://docs.apify.com/academy"] |
| `corpusId` (Corpus ID) | string | Name of the tracked corpus, e.g. 'apify-academy'. | "default" |
| `maxPages` (Max pages checked per run) | integer | Stop after this many pages are checked (and charged), e.g. 500. | 500 |
| `scope` (Crawl scope) | string | 'path' = only URLs under the start URL's folder (start https://example.com/docs/ tracks /docs/...), 'domain' = the whole site of each... | "path" |
| `includeGlobs` (Include URL globs) | array | Track only links whose full URL matches one of these patterns, e.g. \['https://docs.apify.com/academy/**']. |  |
| `excludeGlobs` (Exclude URL globs) | array | Never follow links whose full URL matches one of these patterns, e.g. \['**/changelog/**', '**?page=\*']. |  |

All 10 fields, with examples: see the **Input** tab.

### Switching from ryanclinton/website-change-monitor

Already running `ryanclinton/website-change-monitor`? Map its input fields to this Actor's like this:

| `ryanclinton/website-change-monitor` input | This Actor's input | Notes |
|---|---|---|
| `urls` | `startUrls` | Links are followed within the chosen `scope`, so one start URL tracks a whole docs section. To watch only the pages you list, add them to `includeGlobs` too. |
| `kvStoreName` | `corpusId` | State is always kept in the named store `aurelix-website-change-tracker` in your account, one record per corpus ID; named stores do not expire. |
| `webhookUrl` | `webhookUrl` | Ours POSTs the delta itself (chunks to upsert, chunk IDs to delete) in batches of 25, ending with a final batch that holds the run summary. |
| `notifyOnlyChanges` | not needed | After the baseline only changed pages are returned. `returnBaseline` decides whether the first run returns every page or records it silently. |
| `mode` / `cssSelector` | no equivalent | Always compares the page's main content as Markdown chunks; a single CSS selector cannot be watched. |
| `ignoreNoise` / `ignorePatterns` / `ignoreSelectors` | not needed | Navigation, footers, cookie banners and volatile lines such as Last updated are removed automatically; custom patterns are not supported. |
| `discoverFromSitemap` / `maxDiscoveredPages` | `useSitemaps` + `maxPages` | Sitemap URLs in scope seed the crawl; `maxPages` (up to 20,000) caps the pages checked per run. |
| `baselineMode` | no equivalent | Each run compares against the corpus as last recorded, like a rolling baseline; a locked baseline is not offered. |
| `watchKeywords` / `minSeverity` | no equivalent | No severity scores or keyword escalation; this tool feeds a vector index, not human alerts. |
| `slackWebhookUrl` | no equivalent | Use `webhookUrl` with a Zapier, Make or n8n step if you want a Slack message. |

- Output changes from one alert per page to one record per added, changed or deleted page, with `upsert_chunks` and `delete_chunk_ids` ready for a vector database.
- Pricing changes from $100 per 1,000 sites monitored to $0.50 per 1,000 pages checked plus $1.50 per 1,000 changed pages, and a $0.003 start fee per run (comparison table above).
- A page that comes back blocked, empty or broken is a free `extraction_failed` record and its previous version is kept; only HTTP 404/410 or a redirect away counts as deleted.
- Your first run with a corpus ID is the baseline, so expect every page as added (or none returned with `returnBaseline` off).

### Use with AI agents (Apify MCP server, Claude, ChatGPT, n8n AI agent)

Website Change Tracker is a ready-made tool for AI agents. Through the Apify MCP server an agent reads this Actor's input schema, fills in the input from your request, starts the run and reads the changed pages back. You pay the same per-result price as a run started by hand, from your own Apify account.

#### Example prompts

- "Track https://docs.apify.com/academy as corpus apify-academy, up to 50 pages, and tell me which pages were added, changed or deleted since the last run"
- "Re-check the Python tutorial at https://docs.python.org/3/tutorial/ for corpus py-tutorial and give me the chunk IDs I need to delete from my vector database"
- "Record a silent baseline of https://example.com/help/ as corpus help-center without returning the pages, so later runs only send me changes"

#### Claude (claude.ai and Claude Desktop)

Open **Settings > Connectors**, choose **Add custom connector**, paste `https://mcp.apify.com/?tools=aurelix/website-change-monitor` and sign in to Apify when asked. Then ask Claude one of the prompts above.

#### Claude Code, Cursor, VS Code and other MCP clients

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=aurelix/website-change-monitor"
```

Clients configured with JSON take the same URL:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=aurelix/website-change-monitor"
    }
  }
}
```

A client without Apify sign-in sends the header `Authorization: Bearer <YOUR_APIFY_TOKEN>`; to run the server locally instead, use `npx -y @apify/actors-mcp-server --actors aurelix/website-change-monitor` with `APIFY_TOKEN` set.

#### ChatGPT

Where your ChatGPT plan offers custom connectors (developer mode), add `https://mcp.apify.com/?tools=aurelix/website-change-monitor` as a connector, sign in to Apify, and ask for the data in plain words.

#### n8n AI agent

Give an **AI Agent** node an **MCP Client Tool** with the endpoint `https://mcp.apify.com/?tools=aurelix/website-change-monitor` and a Bearer credential holding your Apify token. The agent can then call Website Change Tracker whenever a workflow step needs changed pages; for a fixed, non-agent workflow use Apify's n8n node with **Run Actor** instead.

### Use it from AI agents (MCP), the API or no-code

#### Copy this to your AI assistant

```text
Use the Apify Actor aurelix/website-change-monitor (Website Change Tracker) through Apify's MCP server or API. Main input: startUrls (Start URLs or sitemaps); corpusId (Corpus ID). Example input: {"startUrls": ["https://docs.apify.com/academy"], "corpusId": "apify-academy", "maxPages": 300, "webhookUrl": "https://example.com/hooks/rag-sync"}. Each result is one changed page with fields such as corpus_id, change, doc_id, url, fetched_url, title, reason, upsert_count. It costs $1.50 per 1,000 changed pages on Apify's free plan plus a $0.003 start fee per run, and needs no login.
```

MCP server URL for this Actor: `https://mcp.apify.com/?tools=aurelix/website-change-monitor` (setup for each client in the section above).

#### API

Start a run and read the dataset with Apify's REST API or the Python and JavaScript clients:

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("aurelix/website-change-monitor").call(run_input={'startUrls': ['https://docs.apify.com/academy'], 'corpusId': 'apify-academy', 'maxPages': 300, 'webhookUrl': 'https://example.com/hooks/rag-sync'})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

#### No-code: schedule it in 3 clicks

Send the results to Google Sheets, Zapier, Make, n8n, Slack or any webhook URL.

1. Fill in the input, then click **Save as a new task**.
2. In the task, click **Schedule** and pick daily or weekly (Apify Schedules).
3. Paste a webhook URL into **Webhook URL for the delta** to get what changed after every run (each run already reports only what is new).

Prefer n8n or Make? Ask in the **Issues** tab and we will share a ready-made workflow for it.

### FAQ

#### Can I track website changes without login?

Yes. It never logs in, so you need no website account, password or cookies; only a free Apify account to run it.

#### Is there a website change detection API?

You can use this Actor as one. Start it with Apify's REST API or the Python and JavaScript clients and read the results as JSON (see the code above), or call it from an AI agent through MCP.

#### How much does it cost to track website changes?

About $1.50 for 1,000 changed pages on Apify's free plan and $1.05 on paid plans, start fee included. The free $5 monthly credit covers about 3,300 changed pages (or, with checks at $0.50 per 1,000, a 250-page docs site checked daily for a month).

#### Can I export website changes to Excel or Google Sheets?

Yes. Download the results as Excel, CSV, JSON, XML or HTML from the run's **Storage** tab, or send them to Google Sheets with Apify's Google Sheets integration or the API.

#### Is it legal to monitor websites for changes?

It reads only public data any logged-out visitor can see: the website you choose (track only sites you have the right to crawl). It goes at a polite pace, respects robots.txt, never logs in and never solves CAPTCHAs. It returns each page's main content as Markdown chunks with title and headings; the content is returned as the site published it; copyright in it stays with its owner. To have a record removed, open an issue on this Actor's Issues tab; if a site owner formally asks us to stop, this Actor will stop. You are responsible for using the data lawfully, incl. anti-spam and privacy laws; this is not legal advice.

#### How are changes detected?

Each page's main content is converted to Markdown, split into chunks by heading and paragraph, and each chunk gets an ID from the page's canonical URL, its heading path and a hash of its normalized text. A run compares the page's chunk IDs with the ones stored for your corpus: new IDs are upserts, missing IDs are deletions, and a page whose set of IDs is unchanged is not returned. Moving sections around or changing whitespace is not a change.

#### Why do unchanged chunks keep their IDs when the text around them changes?

Chunk boundaries are cut at content-defined points (heading boundaries and paragraph or sentence breaks chosen by their own hash), not at fixed character offsets, so inserting or editing a paragraph re-cuts only the chunk it sits in and, rarely, the next one. In our tests an edited paragraph changed 1 of 7 chunk IDs on its page, and a newly inserted section added 1 chunk and kept all 7 existing IDs.

#### What is ignored as boilerplate?

Everything outside the page's main content: navigation, headers, footers, sidebars, cookie and consent banners, share buttons, forms and scripts. Inside the content, lines such as 'Last updated: ...', copyright lines, 'x min read' and 'Was this page helpful?' are dropped before comparison. In our tests, pages whose only differences were a new menu, cookie banner, footer and 'last updated' date were not returned, and re-checking 218 real documentation pages (Python, FastAPI and Apify docs) returned 0 false changes.

#### How is a deleted page different from an extraction failure?

A page is deleted only when the site says so: HTTP 404 or 410, or a redirect to another page or out of scope. A known page that returns an error (403, 429, 5xx), a bot challenge, an empty JavaScript shell, or main content that shrank by more than 90% is returned as 'extraction_failed' with the reason, free of charge, and its previous version stays in the corpus, so a temporary outage never wipes your index.

#### Where is the state kept?

In a named key-value store 'aurelix-website-change-tracker' in your own Apify account, one record per corpus ID ('corpus-<corpusId>'). Named stores do not expire. Delete the record (or use a new corpus ID) to start a fresh baseline.

#### What does the webhook send?

If you set 'webhookUrl', the run POSTs JSON in batches of 25 records: {corpus_id, run_id, batch, final, changes: \[...]}; every run ends with a batch where final is true and 'summary' holds the run's counts. A webhook error is retried 3 times and never fails the run; the delta is always in the dataset too.

#### Which IDs do I use in my vector database?

Use 'chunk_id' as the point or vector ID (a UUID, accepted as-is by Qdrant, Weaviate, Pinecone, Chroma and pgvector) and store 'doc_id', 'url' and 'citation' as metadata. Upsert every item of 'upsert_chunks' and delete every ID in 'delete_chunk_ids'; skip 'extraction_failed' records. In n8n: Schedule Trigger, then an HTTP Request to this Actor's run-sync-get-dataset-items endpoint, a Code node that splits the records into upserts and one delete list, an embeddings call, and the vector database's upsert and delete endpoints. We share a ready import file for OpenAI embeddings and Qdrant on request through the Issues tab.

#### Does it respect robots.txt?

Always, and it cannot be switched off. Disallowed URLs are never requested, Crawl-delay is honoured up to 6 seconds (sites asking for more are skipped and not charged), redirects into disallowed paths are not followed, and links on nofollow pages are not followed.

#### What does a run with no changes cost?

Only the page checks ($0.50 per 1,000) and the $0.003 start fee; nothing is returned, so no change is charged. A 17-page tutorial re-checked with no edits costs 17 checks. Blocked, broken and unreadable pages, robots.txt blocks, fetch errors and duplicates are not charged at all.

#### What happens on the first run?

The first run with a corpus ID is the baseline. With Return the baseline on (the default), every page comes back as added with all its chunks and is charged as a change. Turn it off to record the baseline silently and pay only per page checked.

#### What if Max pages is lower than the number of pages on the site?

Pages already in the corpus but not reached in this run are kept, not deleted, and the status message says how many were not reached. New pages found later come back as added. Set Max pages above the size of the site so every known page is re-checked on each run.

#### What happens when a run hits my maximum charge or its timeout?

Changes that could not be returned are not recorded as seen, so they come back in the next run. A page check is charged only when its change record also fits under the maximum charge. Progress is saved every 200 pages and at the end, and the run stops before its timeout with its results and state saved.

#### Can I track several sites?

Yes. Each corpus ID is a separate state, so use one corpus ID per site or per index. One corpus can also hold several start URLs. Runs with different corpus IDs never affect each other.

#### Can I change the chunk size later?

Not for an existing corpus: the chunk size is fixed at its first run so chunk IDs stay stable. Use a new corpus ID to re-chunk; it starts a new baseline.

#### Where do I find the run's counts?

In the run's status message, in the SUMMARY record of the run's key-value store, and in the final webhook batch when a webhook URL is set: pages checked, added, changed, deleted, unchanged and failed.

### Limits (stated honestly)

- No JavaScript rendering: pages that are an empty app shell are skipped (and flagged if already in the corpus).
- Only HTML pages are tracked; PDFs, images and other files are skipped. HTML pages over 2 MB are skipped.
- Max pages caps the pages checked per run (up to 20,000). Known pages not reached in a run are kept, not deleted, and checked in a later run; set Max pages above the size of the site.
- At most two new requests per second per site (slower with a Crawl-delay up to 6 s; sites asking for more are skipped). Conversion is CPU-bound at 256 MB: about 1,000 large documentation pages an hour; the run stops before its timeout with its results and state saved.
- Chunk size is fixed per corpus at its first run; to re-chunk, use a new corpus ID (which starts a new baseline).
- A page that shrinks by more than 90% (from at least 200 words) is treated as broken, not as a change; if the shrink is real, delete the corpus record or use a new corpus ID.
- rel=canonical is followed only within the same site and scope; a site whose canonical tags all point to one page will collapse to that page.
- If the site stops responding (15 errors in a row), the run stops early and the corpus is kept as it was.
- Pages behind a login, paywall or bot challenge are not accessible.

### Works great with

- **Before this: [Website to Markdown API](https://apify.com/aurelix/website-to-markdown)**: one full Markdown copy of the same `startUrls` to start the corpus

### Related Aurelix tools

Public Aurelix Actors, grouped by the data they collect. All are pay-per-result and need no login.

#### AI, content and SEO

- **[Google Trends by Region](https://apify.com/aurelix/google-trends-scraper)**: Google Trends interest over time, by region and rising queries for any number of keywords, plus trending searches.
- **[Google Autocomplete API](https://apify.com/aurelix/google-autocomplete-scraper)**: Expands seed keywords into hundreds of real Google and YouTube autocomplete phrases, tagged by search intent.
- **[PDF to Markdown OCR & Table Extraction](https://apify.com/aurelix/pdf-to-markdown)**: Turns linked PDF and Word files into Markdown per page, with tables as JSON rows, RAG chunks and OCR.
- **[Broken Link Crawler & SEO Audit](https://apify.com/aurelix/seo-audit-crawler)**: Audits a whole site: one row per page with SEO checks and broken links, plus a ranked issues summary.
- **[Website to Markdown API](https://apify.com/aurelix/website-to-markdown)**: Crawls a site and returns one row of clean main-content Markdown per page, with title, headings and links.

#### Jobs and hiring data

- **[Greenhouse Jobs Scraper](https://apify.com/aurelix/greenhouse-jobs-scraper)**: Live jobs from a company's Greenhouse, Lever, Ashby or SmartRecruiters board, with salary and Canadian province tags.
- **[LinkedIn Jobs Country Filter Scraper](https://apify.com/aurelix/linkedin-jobs)**: Public LinkedIn job postings by keyword and country: title, company, location, seniority, job type, salary text, description.
- **[Multiple Job Boards Scraper](https://apify.com/aurelix/multi-job-board-scraper)**: LinkedIn, about 50 company career sites, Remote OK and We Work Remotely jobs in one deduplicated list.

#### E-commerce and product data

- **[AliExpress Scraper API](https://apify.com/aurelix/aliexpress-product-search)**: AliExpress search results by keyword: price, discount, rating, units sold, ship-from country and sponsored flag.
- **[Amazon New Releases & Best Sellers](https://apify.com/aurelix/amazon-best-sellers)**: Amazon Best Sellers and New Releases top 100 on 10 marketplaces: rank, ASIN, price, rating, rank change.
- **[Shopify Product Export Scraper](https://apify.com/aurelix/shopify-products-scraper)**: Every public product of any Shopify store with variants, SKUs, prices, barcodes, stock counts and collections.

#### Leads and company data

- **[Newly Funded Companies Hiring](https://apify.com/aurelix/funded-companies-hiring)**: Companies that just raised money (SEC Form D), with their open roles and business contacts, as a new-only monitor.
- **[Google Maps Phone Scraper](https://apify.com/aurelix/google-maps-scraper)**: Local business lists from Google Maps: phones, websites, ratings, hours, coordinates; optional website emails and socials.
- **[Recently Funded Companies API](https://apify.com/aurelix/recently-funded-companies)**: US companies that just filed an SEC Form D: amount raised, industry, state, executives, confirmed website.
- **[Tech Stack API](https://apify.com/aurelix/tech-stack-detector)**: CMS, e-commerce platform, analytics, ad pixels, chat, CDN and email provider for each domain you enter.

#### Ad intelligence

- **[Competitor Watch](https://apify.com/aurelix/competitor-watch)**: One report per competitor with only what is new: Meta and TikTok ads, page changes, Shopify products and prices, jobs.
- **[Facebook Ad Library Search & API](https://apify.com/aurelix/facebook-ad-library-scraper)**: Facebook and Instagram business ads from the Meta Ad Library: copy, creatives, landing pages, dates, optional EU reach.

#### Canada and government data

- **[Canada Building Permits](https://apify.com/aurelix/canada-building-permits-scraper)**: Building permits and development applications from 7 Canadian cities: address, type, status, dates, value, applicant, contractor and Toronto planner contact.

#### More data tools

- **[Tennis Betting Odds Scraper](https://apify.com/aurelix/tennis-odds-scraper)**: Tennis results and upcoming matches with opening and closing odds per bookmaker and implied probabilities.

### Recent changes

- **0.1**: Store page corrected: there is no ready-made n8n template; the FAQ describes the workflow and we share an import file on request.

The full history is on the **Changelog** tab.

### Tested daily

An automated health check runs this Actor every day on a fixed input and checks that at least 0 changed pages come back with `corpus_id`, `change`, `doc_id`, `url`, `scraped_at` filled in; a failed check alerts the maintainer straight away.

### Your feedback

Missing a field, found a bug or need another filter? Open an issue on the **Issues** tab; we aim to answer within a day. If this Actor saved you time, a short review helps other people find it.

# Changelog

This Actor's version history is a separate document: https://apify.com/aurelix/website-change-monitor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start from, e.g. \['https://docs.apify.com/academy'], or sitemap URLs ending in .xml, e.g. \['https://example.com/sitemap.xml']. Links are followed within the chosen scope. A URL without http(s):// gets https://.

## `corpusId` (type: `string`):

Name of the tracked corpus, e.g. 'apify-academy'. Runs with the same corpus ID compare against each other; a new ID starts a new baseline. 1-60 letters, digits, '-' or '\_'. Stored in your account's key-value store 'aurelix-website-change-tracker'.

## `maxPages` (type: `integer`):

Stop after this many pages are checked (and charged), e.g. 500. Set it above the size of the site so every known page is re-checked each run; known pages not reached are kept, not deleted. Pages blocked by robots.txt, errors and duplicates do not count.

## `scope` (type: `string`):

'path' = only URLs under the start URL's folder (start https://example.com/docs/ tracks /docs/...), 'domain' = the whole site of each start URL (www. and the bare domain count as one site).

## `includeGlobs` (type: `array`):

Track only links whose full URL matches one of these patterns, e.g. \['https://docs.apify.com/academy/**']. '**' matches anything, '\*' anything except '/'. Empty = every in-scope URL.

## `excludeGlobs` (type: `array`):

Never follow links whose full URL matches one of these patterns, e.g. \['**/changelog/**', '\*\*?page=\*'].

## `useSitemaps` (type: `boolean`):

Also seed the crawl with in-scope URLs from each start site's sitemap (robots.txt Sitemap lines or /sitemap.xml), so pages that no other page links to are tracked too.

## `returnBaseline` (type: `boolean`):

On the first run of a corpus, return every page as 'added' with all its chunks (charged per page) to fill your vector database. Off = record the baseline silently and pay only per page checked; later runs return changes either way.

## `chunkSize` (type: `integer`):

Largest chunk in characters, e.g. 1500 (about 350 tokens). Chunks follow headings and paragraph boundaries. Fixed per corpus at its first run, so chunk IDs stay stable; use a new corpus ID to re-chunk.

## `webhookUrl` (type: `string`):

Optional. The changes are also POSTed here as JSON in batches of 25 ({corpus_id, run_id, batch, final, changes}); the last batch has final=true and the run summary. A failing webhook never fails the run.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "corpusId": "apify-academy",
  "maxPages": 20,
  "scope": "path",
  "includeGlobs": [],
  "excludeGlobs": [],
  "useSitemaps": false,
  "returnBaseline": true,
  "chunkSize": 1500
}
```

# Actor output Schema

## `changes` (type: `string`):

One item per added, changed or deleted page (with the chunks to upsert and the chunk IDs to delete), plus free records for known pages that could not be read this run.

## `summary` (type: `string`):

Counts for this run: pages checked, added, changed, deleted, unchanged, extraction failures, corpus size.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "corpusId": "apify-academy",
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("aurelix/website-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy"],
    "corpusId": "apify-academy",
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("aurelix/website-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "corpusId": "apify-academy",
  "maxPages": 20
}' |
apify call aurelix/website-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aurelix/website-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/c9coCX7iJRFP8kLsX/builds/8d2HPafKnqkJ2mqBQ/openapi.json