# Actor Website Change Monitor (`anyhowmarketer/actor-website-change-monitor`) Actor - **URL**: https://apify.com/anyhowmarketer/actor-website-change-monitor.md - **Developed by:** [Egor Kaleynik](https://apify.com/anyhowmarketer) (community) - **Categories:** Automation, Lead generation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website Change Detection & Monitoring (Apify Actor) Stateful website monitoring actor for long-running cron use. Unlike one-shot extractors, this actor stores URL history, compares snapshots between runs, and emits change events. ### Modes - `monitor`: user provides explicit URL list (`startUrls`), actor watches for changes. - `discover`: actor discovers URLs from a domain (or category URL seed) and then monitors them. - `promo-detect`: actor generates promo candidates from a pattern library, probes them in batch, then monitors live pages. Amazon rule: - `platform=amazon` + `mode=promo-detect` is blocked by input validation. ### Scheduler Guidance - Default: every `24h` - Recommended: every `12h` - Hard minimum: `6h` - Maximum interval: `168h` (7 days) Input guardrail: - `checkIntervalHours` must be `6..168`. ### Tiering Model - Tier 1: checked every run. - Tier 2: checked on modulo rotation (`runCounter % N === tierOffset`), where `N=tier2Modulo`. - Tier 3: baseline capture tier (new URLs). - Quarantine: after `5` consecutive fetch errors. Transitions: - Tier 3 -> Tier 2 after baseline captured. - Tier 2 -> Tier 1 after detected change. - Tier 1 -> Tier 2 after `3` stable runs. - Any -> Quarantine after `5` consecutive errors. ### Platform Support - Detection: Shopify, WooCommerce, Magento, Amazon, BigCommerce, PrestaShop, Generic. - Platform extraction: Shopify/Woo/Magento/Amazon selector packs + JSON/API-like parsing + JSON-LD fallback. ### Discovery Strategies #### Mode `discover` 1. Sitemap recursion (`/sitemap.xml`, `/sitemap_index.xml`, `robots.txt` references) 2. Shopify API pagination (`/products.json?limit=250&since_id=...`) for Shopify platforms 3. Category/listing crawl fallback (depth and page-limit configurable) #### Amazon `discover` - Strategy 1: `amazonAsins` list (build `/dp/{ASIN}` URLs) - Strategy 2: keyword search crawl (`/s`), parse `data-asin` - Strategy 3: browse node crawl (`/b?node=...`), parse `data-asin` ### Mode C Probe Engine Batch probe engine (before registry merge): - Concurrency control (`probeConcurrency`) - Per-domain throttle (`probeDelayMs`) - HEAD-first probing with GET fallback for HEAD-blocking responses - Redirect capture (`MODE_C_REDIRECTS` KV key) - Probe lifecycle events: - `page_appeared` for newly live probed URLs - `page_disappeared` for previously known URLs that fail probe Localization: - Promo URL candidates include static locales and dynamic prefixes extracted from homepage `hreflang` links. ### Amazon Hardening Implemented safeguards: - Forces JS rendering (`renderJs=true` semantics) - Randomized delay profile (`3-8s`) - Rotating desktop user agents - Session cookies persisted in-run - Residential proxy enforcement (`proxyConfig.apifyProxyGroups` must include `RESIDENTIAL`) ### Stateful Storage (KV) - `urlRegistry` - `urlRegistryByDomain` - `platformCache` - `runCounter` - `failedWebhooks` - `deadWebhooks` - `MODE_C_REDIRECTS` ### Webhook Reliability - Failed sends buffered into `failedWebhooks` - Retry on next run before dispatching new events - Retry cap: `20` attempts - Exceeded entries moved to `deadWebhooks` dead-letter queue Run output includes webhook retry/dead-letter metrics. ### Input Highlights Core: - `mode`, `startUrls`, `domain`, `platformHint` - `monitorSelectors`, `changeThreshold` - `maxUrls`, `tier2Modulo`, `checkIntervalHours` - `notificationWebhook` Mode B: - `categoryCrawlDepth`, `maxCategoryPages` Mode C: - `probeOnlyMode`, `probeConcurrency`, `probeDelayMs` - `userPatterns` Amazon: - `proxyConfig` - `amazonAsins`, `amazonSearchKeywords`, `amazonSearchCategoryId`, `amazonBrowseNodeId` Scale: - `checkConcurrency` (parallel URL checks) `domain` input accepts either: - host/domain, e.g. `www.reserved.com` - full URL seed, e.g. `https://www.reserved.com/pl/pl/kobieta` When a full URL seed is provided in `discover` mode, category crawl starts from that URL first. ### Local Development Install/build/run: ```bash npm install npm run build npm start ```` Run tests: ```bash npm run test ``` Smoke inputs are in `smoke-tests/`. ### Deployment Sanity Checklist 1. `npm run build` passes. 2. `npm run test` passes. 3. Verify `.actor/actor.json` points to `INPUT_SCHEMA.json` and `Dockerfile`. 4. For Amazon runs, set proxy group to `RESIDENTIAL`. 5. Create Apify Scheduler task with interval `>= 6h`. 6. Validate webhook endpoint availability and idempotency handling. ### Cost Model Notes - HTML fetch path (Cheerio/gotScraping) is the cheap default. - JS rendering is more expensive and should be used only where needed. - Tiering and modulo rotation are the primary cost controls at scale. - Probe-first Mode C avoids expensive full checks for dead URLs. # Actor input Schema ## `mode` (type: `string`): monitor: explicit URLs, discover: auto-discover product URLs, promo-detect: generate and probe promo candidates. ## `startUrls` (type: `array`): Start URLs input parameter. ## `domain` (type: `string`): Supports host (e.g. www.example.com) or full URL seed (e.g. https://www.example.com/women/dresses). ## `platformHint` (type: `string`): Platform Hint input parameter. ## `monitorSelectors` (type: `array`): Monitor Selectors input parameter. ## `changeThreshold` (type: `number`): Change Threshold (%) input parameter. ## `checkIntervalHours` (type: `integer`): Check Interval (Hours) input parameter. ## `notificationWebhook` (type: `string`): Notification Webhook input parameter. ## `maxUrls` (type: `integer`): Max URLs input parameter. ## `probeOnlyMode` (type: `boolean`): Probe-only Mode input parameter. ## `renderJs` (type: `boolean`): Render JS input parameter. ## `userPatterns` (type: `array`): User Patterns input parameter. ## `tier2Modulo` (type: `integer`): Tier 2 Rotation N input parameter. ## `categoryCrawlDepth` (type: `integer`): Category Crawl Depth input parameter. ## `maxCategoryPages` (type: `integer`): Max Category Pages input parameter. ## `probeConcurrency` (type: `integer`): Probe Concurrency input parameter. ## `probeDelayMs` (type: `integer`): Probe Delay Per Domain (ms) input parameter. ## `proxyConfig` (type: `object`): Proxy Config input parameter. ## `amazonAsins` (type: `array`): Amazon ASINs input parameter. ## `amazonSearchKeywords` (type: `array`): Amazon Search Keywords input parameter. ## `amazonSearchCategoryId` (type: `string`): Amazon Search Category ID input parameter. ## `amazonBrowseNodeId` (type: `string`): Amazon Browse Node ID input parameter. ## `checkConcurrency` (type: `integer`): Check Concurrency input parameter. ## `debugLogging` (type: `boolean`): Debug Logging input parameter. ## Actor input object example ```json { "mode": "monitor", "startUrls": [], "monitorSelectors": [], "changeThreshold": 0, "checkIntervalHours": 24, "maxUrls": 5000, "probeOnlyMode": false, "renderJs": false, "userPatterns": [], "tier2Modulo": 5, "categoryCrawlDepth": 2, "maxCategoryPages": 250, "probeConcurrency": 30, "probeDelayMs": 500, "amazonAsins": [], "amazonSearchKeywords": [], "checkConcurrency": 20, "debugLogging": false } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("anyhowmarketer/actor-website-change-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("anyhowmarketer/actor-website-change-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call anyhowmarketer/actor-website-change-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=anyhowmarketer/actor-website-change-monitor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Actor Website Change Monitor", "version": "0.1", "x-build-id": "22ZScVjA8Sriztgaz" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/anyhowmarketer~actor-website-change-monitor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-anyhowmarketer-actor-website-change-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/anyhowmarketer~actor-website-change-monitor/runs": { "post": { "operationId": "runs-sync-anyhowmarketer-actor-website-change-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/anyhowmarketer~actor-website-change-monitor/run-sync": { "post": { "operationId": "run-sync-anyhowmarketer-actor-website-change-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "mode": { "title": "Mode", "enum": [ "monitor", "discover", "promo-detect" ], "type": "string", "description": "monitor: explicit URLs, discover: auto-discover product URLs, promo-detect: generate and probe promo candidates.", "default": "monitor" }, "startUrls": { "title": "Start URLs", "type": "array", "description": "Start URLs input parameter.", "default": [], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "domain": { "title": "Domain or Category URL", "type": "string", "description": "Supports host (e.g. www.example.com) or full URL seed (e.g. https://www.example.com/women/dresses)." }, "platformHint": { "title": "Platform Hint", "enum": [ "shopify", "woocommerce", "magento", "amazon", "bigcommerce", "prestashop", "generic" ], "type": "string", "description": "Platform Hint input parameter." }, "monitorSelectors": { "title": "Monitor Selectors", "type": "array", "description": "Monitor Selectors input parameter.", "items": { "type": "string" }, "default": [] }, "changeThreshold": { "title": "Change Threshold (%)", "type": "number", "description": "Change Threshold (%) input parameter.", "default": 0 }, "checkIntervalHours": { "title": "Check Interval (Hours)", "minimum": 6, "maximum": 168, "type": "integer", "description": "Check Interval (Hours) input parameter.", "default": 24 }, "notificationWebhook": { "title": "Notification Webhook", "type": "string", "description": "Notification Webhook input parameter." }, "maxUrls": { "title": "Max URLs", "minimum": 1, "type": "integer", "description": "Max URLs input parameter.", "default": 5000 }, "probeOnlyMode": { "title": "Probe-only Mode", "type": "boolean", "description": "Probe-only Mode input parameter.", "default": false }, "renderJs": { "title": "Render JS", "type": "boolean", "description": "Render JS input parameter.", "default": false }, "userPatterns": { "title": "User Patterns", "type": "array", "description": "User Patterns input parameter.", "items": { "type": "string" }, "default": [] }, "tier2Modulo": { "title": "Tier 2 Rotation N", "minimum": 2, "maximum": 30, "type": "integer", "description": "Tier 2 Rotation N input parameter.", "default": 5 }, "categoryCrawlDepth": { "title": "Category Crawl Depth", "minimum": 1, "maximum": 2, "type": "integer", "description": "Category Crawl Depth input parameter.", "default": 2 }, "maxCategoryPages": { "title": "Max Category Pages", "minimum": 10, "maximum": 1000, "type": "integer", "description": "Max Category Pages input parameter.", "default": 250 }, "probeConcurrency": { "title": "Probe Concurrency", "minimum": 1, "maximum": 100, "type": "integer", "description": "Probe Concurrency input parameter.", "default": 30 }, "probeDelayMs": { "title": "Probe Delay Per Domain (ms)", "minimum": 0, "maximum": 30000, "type": "integer", "description": "Probe Delay Per Domain (ms) input parameter.", "default": 500 }, "proxyConfig": { "title": "Proxy Config", "type": "object", "description": "Proxy Config input parameter." }, "amazonAsins": { "title": "Amazon ASINs", "type": "array", "description": "Amazon ASINs input parameter.", "items": { "type": "string" }, "default": [] }, "amazonSearchKeywords": { "title": "Amazon Search Keywords", "type": "array", "description": "Amazon Search Keywords input parameter.", "items": { "type": "string" }, "default": [] }, "amazonSearchCategoryId": { "title": "Amazon Search Category ID", "type": "string", "description": "Amazon Search Category ID input parameter." }, "amazonBrowseNodeId": { "title": "Amazon Browse Node ID", "type": "string", "description": "Amazon Browse Node ID input parameter." }, "checkConcurrency": { "title": "Check Concurrency", "minimum": 1, "maximum": 200, "type": "integer", "description": "Check Concurrency input parameter.", "default": 20 }, "debugLogging": { "title": "Debug Logging", "type": "boolean", "description": "Debug Logging input parameter.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```