# SiteVerdict — Web Change Feed (any page to RSS, Atom, JSON) (`siteverdict/siteverdict-web-change-feed`) Actor

Turn any public page or section into an RSS, Atom or JSON feed of its changes, or of new items on a list page. CSS selectors or automatic main content, noise filters, de-duplicated entries, Slack/webhook alerts and optional AI summaries. Respects robots.txt.

- **URL**: https://apify.com/siteverdict/siteverdict-web-change-feed.md
- **Developed by:** [Aman Thakran](https://apify.com/siteverdict) (community)
- **Categories:** Developer tools, Marketing
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Change Feed: turn any web page into an RSS feed of its changes

**No RSS on the site you care about? Make one.** Point this Actor at any public page, a part of one (CSS selector), or a list page (news, jobs, releases, products). Each scheduled run turns what changed into clean **RSS 2.0, Atom and JSON Feed** entries that feed readers, Slack, Zapier, n8n and AI agents can subscribe to.

- **Two modes:**
  - **section:** a feed entry with added/removed lines whenever the content changes
  - **list:** one entry per new item, keyed by its link
- **Fewer false alarms:** scripts, styles, iframes, form fields (so CSRF tokens), meta tags, hidden elements, navigation, headers, footers and common cookie banners are ignored automatically. Visible text that changes on every load (a "Last updated" time, a rotating ad or "recommended" box) still counts as a change until you add an `ignorePatterns` regex, an `ignoreSelectors` entry or `ignoreNumbers`. Moved lines don't count as changes, and small edits under a threshold accumulate quietly.
- **Honest statuses:** the first run of a page is stored as its `baseline` (never charged as a change). A failed fetch, timeout, HTTP error, `404` or bot-protection page is reported as such, is free and leaves the stored baseline untouched, so it is never reported as "changed" or "removed".
- **Polite by design:** robots.txt is checked before every page, with at most one request per second per host, an honest User-Agent and conditional requests (ETag / Last-Modified). Bot-protected pages are reported, **never bypassed**.

### Who it is for

- **Researchers, journalists and analysts:** follow regulator pages, policy pages and competitor pricing in your feed reader.
- **Ops and growth teams:** get a Slack alert when a supplier changes its terms, a competitor changes prices, or a careers page posts a role.
- **AI agents:** a JSON Feed with structured `added`/`removed` lines is far cheaper than re-reading whole pages.

### Use cases

1. **Competitor pricing page** (`selector: "#pricing"`) → entries like "Pricing: 2 lines added, 1 removed", with the old and new price lines.
2. **Government or regulator updates list** (`itemSelector: "li.gem-c-document-list__item"`) → one feed item per new publication.
3. **Release notes / changelog / status page** → a Slack message with an AI one-liner on your own key: "Version 4.2 adds SSO and drops IE11 support."

### Example (live run, 11 Oct 2026)

Input: GOV.UK HMRC organisation page, item selector `li.gem-c-document-list__item`, plus the HMRC "about" page in section mode.

- **Run 1:** both pages were stored as the baseline, and the list page published its 15 current items as `initial` entries (not charged as changes).
- **Run 2:** both pages were `unchanged`, with 0 new entries (the list page got `304 Not Modified`).

Feed item from run 1:

```xml
<item>
  <title>Apply for business rates for a self-catering property in Wales</title>
  <link>https://www.gov.uk/guidance/apply-for-business-rates-for-a-self-catering-property-in-wales</link>
  <guid isPermaLink="false">urn:sv:fe1f0c195b1663ff6058e5b3c3275238</guid>
  <category>initial</category>
  <description><![CDATA[Apply for business rates for a self-catering property in Wales — 9 October 2026 — Guidance]]></description>
</item>
```

### Where is my feed?

Feeds live in a **named key-value store** in your Apify account (`feedName`, default `siteverdict-change-feed`) as `feed.xml` (RSS), `feed.atom` and `feed.json`. Their URLs are in the run's `OUTPUT` record and stay the same between runs, so subscribe once. If your account restricts storage access, make that store's records readable or fetch them with your API token. **Never put your token in a public feed reader.**

### Output

- **Dataset:** one row per page check with `status` (baseline / changed / unchanged / robots_disallowed / blocked / selector_not_found / http_error / not_html / error / skipped_charge_limit / invalid_input), `addedLines`, `removedLines`, `newEntries` and the new entries.
- **Feed entries:** `id` (stable, used for de-duplication), `kind` (changed / new_item / updated_item / removed_item / initial), `title`, `link`, `published`, `summary`, `added[]`, `removed[]` and HTML content with `<ins>`/`<del>`.
- **Webhook:** Slack gets a message; other HTTPS endpoints get a JSON payload with the entries.

### Pricing

Pay per event:

- **`page-checked`: $0.0005** per page successfully checked (including "unchanged")
- **`change-detected`: $0.003** per page that changed in a run (the first run's baseline and initial items are not charged as changes)
- **`ai-change-summary`: $0.002** per AI summary on your own LLM key (optional)

Example: 20 pages checked hourly for a month (14,400 checks) ≈ **$7.20**, plus $0.003 per actual change. Blocked, disallowed and failed pages are free.

**Maximum total charge:** the run never charges more than your maximum. Before a page's result is saved, the page check plus its change (and any AI summary) must fit in what is left. If a detected change does not fit, nothing is charged for that page, its row says `skipped_charge_limit` and its baseline is not moved, so the change is reported on the next run. Pages that are not processed are listed as `skipped_charge_limit` and the run's status message says "N not processed (max total charge reached)". An AI summary that does not fit is skipped without calling your LLM.

### FAQ

**JavaScript-heavy pages?** This Actor reads the server HTML (fast and cheap). If the content is rendered in the browser, you will see `selector_not_found`. Use our Website Change & Uptime Monitor for browser rendering.

**How do I stop noise from dates or counters?** Use `ignoreSelectors` for widgets, `ignorePatterns` for text such as `Last updated .*`, and `ignoreNumbers` for counters. Raise `minChangeChars` for small edits.

**Will it re-announce items that flicker in and out?** No. Items are remembered by link, and those that disappear and come back are not reported as new.

### Limits

- Up to 500 pages per run and 5 MB per page (larger pages are truncated). HTML and text only.
- Each run checks once. Use Apify Schedules for monitoring (e.g. hourly).
- The feed window is up to 500 most recent entries.
- The first 5,000 non-empty text lines of a page are stored and compared; changes further down a very long page are not reported (use a `selector`).
- If a run is migrated or restarted by the platform, it continues at the next unfinished page: finished pages are not checked or charged again, and their feed entries are kept. A finished run that is resurrected does not re-check its pages; start a new run instead.
- Changing `ignorePatterns`, `ignoreNumbers` or a selector starts a new baseline for that page (the first run after the change is `baseline`).

### Sources & compliance

- **Public pages only.** robots.txt is honoured for our token `SiteVerdict` (RFC 9309; a robots.txt server error counts as "disallow"). Redirect targets are checked too.
- We fetch at most one request per second per host, identify ourselves (`SiteVerdict/web-change-feed (+hello@webrevenuescan.com)`) and send conditional requests. We never get around CAPTCHAs, logins, paywalls or bot protection; those pages are reported as `blocked`.
- **Security:** an SSRF guard blocks private, internal and metadata addresses on every redirect hop. Webhooks must use HTTPS. The LLM key and webhook URL are secret inputs that are never logged.
- **Copyright and terms:** feeds contain short excerpts of changed text for monitoring. Respect each site's terms, and don't republish full content without permission. GOV.UK content used in the example is under the Open Government Licence v3.0.
- **Personal data (UK GDPR):** don't monitor pages about private individuals. Not legal advice.

Built by SiteVerdict (BharatFare Travel and Technologies Ltd, UK). Contact: hello@webrevenuescan.com.

# Changelog

This Actor's version history is a separate document: https://apify.com/siteverdict/siteverdict-web-change-feed/changelog.md

# Actor input Schema

## `urls` (type: `array`):

Public pages. Each run compares them with the previous run and adds changes to your feed. Schedule the Actor (e.g. hourly/daily) to keep the feed fresh.

## `selector` (type: `string`):

Watch only this part of each page, e.g. main, #pricing, .release-notes. Empty = automatic main-content detection.

## `itemSelector` (type: `string`):

Turn a listing into feed items, e.g. article, li.job, .post-card. Each item becomes one feed entry when it first appears (keyed by its link).

## `pages` (type: `array`):

Optional objects {"url", "name", "selector", "itemSelector", "ignoreSelectors": \["…"]} to override the global settings per page.

## `ignoreSelectors` (type: `array`):

CSS selectors removed before comparing (ads, 'last updated' widgets, counters). Navigation, header, footer and cookie banners are ignored by default.

## `ignorePatterns` (type: `array`):

Regular expressions removed from the text before comparing, e.g. "Last updated .\*" or "\d+ people viewing".

## `ignoreNumbers` (type: `boolean`):

Treat changes in digits only (prices, counters, dates) as no change.

## `minChangeChars` (type: `integer`):

Smaller edits are not reported (they accumulate until they cross the threshold).

## `feedName` (type: `string`):

Named key-value store holding state and the feed files (feed.xml, feed.atom, feed.json). Use one name per feed; keep it the same between runs.

## `feedTitle` (type: `string`):

Shown in feed readers.

## `maxEntries` (type: `integer`):

Rolling window of most recent entries.

## `emitInitialItems` (type: `boolean`):

With an item selector, the first run publishes the items already on the page (not charged as changes).

## `trackUpdatedItems` (type: `boolean`):

List pages: add an entry when an existing item's text changes.

## `trackRemovedItems` (type: `boolean`):

List pages: add an entry when an item disappears.

## `webhookUrl` (type: `string`):

Called when new changes are found (Slack incoming webhook gets a message, other URLs a JSON payload).

## `aiSummary` (type: `boolean`):

Uses YOUR OWN LLM key to summarise what changed (max 3 summaries per page per run).

## `aiProvider` (type: `string`):

LLM provider for summaries.

## `aiApiKey` (type: `string`):

Your LLM key; never logged or output.

## `aiModel` (type: `string`):

Defaults: gpt-4.1-mini, claude-sonnet-5-5, gemini-2.5-flash, mistral-small-latest.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://www.gov.uk/government/organisations/hm-revenue-customs/about"
    }
  ],
  "ignoreNumbers": false,
  "minChangeChars": 10,
  "feedName": "siteverdict-change-feed",
  "maxEntries": 100,
  "emitInitialItems": true,
  "trackUpdatedItems": false,
  "trackRemovedItems": false,
  "aiSummary": false,
  "aiProvider": "openai"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `output` (type: `string`):

No description

## `files` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://www.gov.uk/government/organisations/hm-revenue-customs/about"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("siteverdict/siteverdict-web-change-feed").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [{ "url": "https://www.gov.uk/government/organisations/hm-revenue-customs/about" }] }

# Run the Actor and wait for it to finish
run = client.actor("siteverdict/siteverdict-web-change-feed").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://www.gov.uk/government/organisations/hm-revenue-customs/about"
    }
  ]
}' |
apify call siteverdict/siteverdict-web-change-feed --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,siteverdict/siteverdict-web-change-feed"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fKfwbBYBe3VMnkz0g/builds/7mcBG6qoDRCYAIpxa/openapi.json