JSON & Visual Content Monitor

Monitor JSON endpoints and web pages for changes with automatic snapshots, AI-powered summaries, email notifications, and multi-format exports. Perfect for tracking API changes, pricing updates, and visual website modifications.

Why Use This Actor?

Stop manually checking websites and APIs for changes. This Actor automatically monitors your endpoints and pages, detects any modifications, and notifies you instantly. Whether you're tracking competitor prices, monitoring API schemas, or watching for policy changes, this Actor has you covered.

Features

• 🔍 Dual Monitoring Modes - Monitor JSON data, visual screenshots, or both simultaneously
• 📊 Detailed Diffs - Get path-level changes with old and new values for precise tracking
• 📸 Visual Screenshots - Capture before/after screenshots automatically with hash-based comparison
• 📄 HTML Diff Reports - Beautiful HTML reports showing before/after JSON with highlighted changes
• 🤖 AI-Powered Summaries - Use OpenAI or Anthropic to generate human-readable change summaries
• 📧 Multiple Notification Channels - Email (Apify/SendGrid/Mailgun), Slack, and custom webhooks
• 💾 Smart Snapshots - Automatically stores and compares against previous versions
• 📤 Multi-Format Export - Export to Dataset, CSV, JSON, JSONL, SQLite, PostgreSQL, MySQL, MongoDB, Google Sheets
• 🔑 Entity Tracking - Track changes in arrays using primary keys (e.g., product IDs)
• 🚨 Error Handling - Graceful handling of network errors with automatic retries
• ⚙️ Highly Configurable - Control depth, ignore paths, viewport size, headers, and more

Use Cases

E-commerce & Retail

• Price Monitoring - Track competitor pricing across multiple stores
• Inventory Tracking - Monitor stock levels and availability
• Product Changes - Detect when product descriptions or specifications change

SaaS & Technology

• API Monitoring - Track REST API changes and schema updates
• Feature Flags - Monitor configuration and feature flag changes
• Breaking Changes - Get alerted to API breaking changes before they affect your app

Compliance & Legal

• Terms of Service - Monitor changes to terms, policies, and legal documents
• Regulatory Compliance - Track updates to compliance requirements
• Privacy Policies - Stay informed about privacy policy modifications

Finance & Trading

• Exchange Rates - Monitor currency exchange rate changes
• Stock Prices - Track stock price movements
• Crypto Prices - Monitor cryptocurrency price changes

Input

Required Fields

• targetUrls (array) - List of URLs to monitor (JSON APIs or web pages)

  ["https://api.example.com/products", "https://example.com/pricing"]
  

Optional Fields

• monitoringMode (string) - Choose monitoring type:

  • json - Monitor JSON data only (default)
  • visual - Capture visual screenshots only
  • both - Monitor both JSON and visual changes

• requestHeaders (object) - HTTP headers for API requests

  {
    "Authorization": "Bearer YOUR_TOKEN",
    "Accept": "application/json"
  }
  

• primaryKeyPath (string) - JSONPath for primary key in array responses

  "$[*].id"
  

• maxDepth (integer) - Maximum depth for JSON tree traversal (default: 5)

• ignorePaths (array) - JSON paths to ignore in diffs

  ["$.timestamp", "$.lastModified", "$[*].updatedAt"]
  

• notificationEmail (string) - Simple email notification address (uses Apify's built-in email)

  "admin@example.com"
  

• emailNotifications (object) - Advanced email configuration (SendGrid/Mailgun)

  {
    "enabled": true,
    "recipients": ["admin@example.com"],
    "provider": "sendgrid",
    "apiKey": "SG.xxx",
    "fromEmail": "alerts@example.com"
  }
  

• slackWebhookUrl (string) - Slack Incoming Webhook URL for notifications

  "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
  

• webhookUrl (string) - Generic webhook URL to POST change data

  "https://your-server.com/webhook"
  

• enableAiSummary (boolean) - Generate AI summaries of changes (default: false)

• aiApiKey (string, secret) - API key for OpenAI or Anthropic

• aiProvider (string) - AI provider: "openai" or "anthropic" (default: "openai")

• exportFormats (array) - Export destinations

  [
    {"type": "dataset"},
    {"type": "csv"},
    {"type": "json"},
    {"type": "jsonl"},
    {"type": "sqlite"}
  ]
  

• screenshotOptions (object) - Screenshot configuration

  {
    "fullPage": true,
    "viewportWidth": 1920,
    "viewportHeight": 1080,
    "waitForSelector": "#content"
  }
  

Output

Dataset Records

Each detected change is saved as a structured record:

{
  "url": "https://api.example.com/products",
  "detectedAt": "2025-01-15T10:30:00.000Z",
  "changeType": "modified",
  "path": "$.items[123].price",
  "oldValue": 29.99,
  "newValue": 24.99,
  "meta": {
    "reason": "field_changed",
    "primaryKey": "123"
  }
}

Change Types:

• added - New field or entity added
• removed - Field or entity removed
• modified - Existing value changed
• error - Fetch or parsing error occurred

Key-Value Store

For each monitored URL, the Actor stores:

1. Snapshots (SNAPSHOT-<url-slug>) - Latest JSON response for comparison
2. Screenshots (SCREENSHOT-<url-slug>) - Latest visual screenshot (PNG)
3. Screenshot Hashes (SCREENSHOT-HASH-<url-slug>) - MD5 hash for comparison
4. Summaries (SUMMARY-<url-slug>) - Human-readable change summary
5. HTML Diff Reports (DIFF-REPORT-<url-slug>-<timestamp>.html) - Beautiful HTML diff visualization
6. Report Screenshots (DIFF-REPORT-<url-slug>-<timestamp>.png) - Screenshot of HTML report
7. Exports (changes-<timestamp>.csv/json/jsonl) - Exported change data
8. SQLite Database (changes-<runId>.sqlite) - SQLite database with all changes (if enabled)

Dataset Views

The Actor provides multiple views of your data:

• All Changes - Complete change history
• Added Items - Only new additions
• Removed Items - Only deletions
• Modified Items - Only modifications
• Errors - Failed requests and errors

Examples

Example 1: Monitor JSON API

{
  "targetUrls": [
    "https://api.github.com/repos/apify/apify-sdk-js"
  ],
  "monitoringMode": "json",
  "maxDepth": 5
}

Example 2: Track Product Prices

{
  "targetUrls": [
    "https://api.example.com/products"
  ],
  "primaryKeyPath": "$[*].id",
  "ignorePaths": ["$[*].lastModified"],
  "notificationEmail": "team@example.com"
}

Example 3: Visual Website Monitoring

{
  "targetUrls": [
    "https://competitor.com/pricing"
  ],
  "monitoringMode": "visual",
  "screenshotOptions": {
    "fullPage": true,
    "viewportWidth": 1920,
    "viewportHeight": 1080
  }
}

Example 4: Monitor with AI Summary and Notifications

{
  "targetUrls": [
    "https://api.example.com/v1/products"
  ],
  "enableAiSummary": true,
  "aiApiKey": "sk-...",
  "aiProvider": "openai",
  "notificationEmail": "admin@example.com",
  "slackWebhookUrl": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
  "exportFormats": [
    {"type": "dataset"},
    {"type": "jsonl"},
    {"type": "sqlite"}
  ]
}

Example 5: Track Array Changes

{
  "targetUrls": [
    "https://jsonplaceholder.typicode.com/users"
  ],
  "primaryKeyPath": "$[*].id",
  "maxDepth": 3,
  "ignorePaths": ["$[*].company.catchPhrase"]
}

How It Works

1. First Run - Fetches data/screenshots and stores as baseline snapshot
2. Subsequent Runs - Compares new data against previous snapshot
3. Diff Generation - Produces detailed change records with paths and values
4. Notification - Sends email alerts if changes detected (optional)
5. Export - Saves changes to configured formats (Dataset, CSV, JSON, etc.)
6. Storage Update - Updates snapshot for next comparison

Scheduling

For continuous monitoring, schedule the Actor to run periodically:

• Every hour - For frequently changing data
• Every 6 hours - For moderate monitoring
• Daily - For slow-changing content
• Weekly - For rarely updated pages

Set up schedules in the Apify Console under the "Schedules" tab.

Integration

Webhooks

Connect to other services when changes are detected:

• Slack notifications
• Discord alerts
• Custom webhooks
• Zapier/Make integrations

API Access

Access your data programmatically:

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const dataset = await client.dataset('DATASET_ID').listItems();

Database Export

Export directly to databases:

{
  "exportFormats": [
    {
      "type": "postgresql",
      "config": {
        "connectionString": "postgresql://user:pass@host:5432/db"
      }
    }
  ]
}

Running Locally

1. Install dependencies:

   $npm install
   

2. Create .actor/INPUT.json with your configuration

3. Run the actor:

   $apify run
   

4. View results in ./storage/datasets/default/

Deploying to Apify

1. Login to Apify CLI:

   $apify login
   

2. Push to Apify platform:

   $apify push
   

Limitations

• Maximum JSON depth is configurable (default: 5 levels)
• Very large JSON responses (>10MB) may impact performance
• Screenshot comparison uses MD5 hashing (pixel-perfect comparison not available)
• AI summaries require OpenAI or Anthropic API key (paid service)
• SQLite export requires better-sqlite3 package (framework ready)
• Database exports (PostgreSQL, MySQL, MongoDB) require additional packages
• Rate limiting applies based on your Apify plan

Error Handling

• 5xx Server Errors - Automatic retry with exponential backoff (3 attempts)
• Invalid JSON - Logged as error record, previous snapshot preserved
• Network Failures - Retried up to 3 times before marking as error
• Non-JSON Content - Detected and logged with content-type information
• Screenshot Failures - Logged as error, JSON monitoring continues (if in "both" mode)

Performance Tips

1. Use ignorePaths - Skip dynamic fields like timestamps to reduce noise
2. Set appropriate maxDepth - Limit depth for large nested objects
3. Use primaryKeyPath - For efficient array diffing with large datasets
4. Schedule wisely - Don't over-monitor; match frequency to change rate
5. Filter exports - Only export to formats you need

Privacy & Security

• API keys and tokens are stored securely
• Screenshots are stored in your private Key-Value Store
• Data is never shared with third parties
• Use Apify's secret input fields for sensitive data
• All data is encrypted at rest and in transit

Version History

0.0 (Current)

• Initial release
• JSON monitoring with detailed diffs
• Visual screenshot monitoring
• Email notifications (Apify built-in, SendGrid, Mailgun)
• Slack and webhook notifications
• AI-powered summaries (OpenAI, Anthropic)
• HTML diff reports with screenshots
• Multi-format export (Dataset, CSV, JSON, JSONL, SQLite)
• Database export support (PostgreSQL, MySQL, MongoDB, Google Sheets)
• Entity tracking with primary keys
• Configurable depth and ignore paths
• Automatic retries and error handling

License

Apache-2.0

Author

Built with ❤️ for the Apify community

---

Ready to start monitoring? Click "Try for free" to run your first monitoring job!