Incremental Sync Engine

Sync data changes from websites incrementally. Detect added, modified, and deleted items between runs. Perfect for monitoring product catalogs, listings, and any structured data.

🎯 Features

• Change Detection: Track added, modified, and deleted items
• Multiple Data Types: JSON API, HTML tables, HTML lists, XML feeds
• Field Tracking: Monitor specific fields for changes
• State Persistence: Automatically save state between runs
• Flexible Output: Changes only, incremental, or full sync

📥 Input

{
    "sourceUrls": [{ "url": "https://api.example.com/products" }],
    "dataType": "json",
    "identifierField": "id",
    "trackFields": ["price", "status", "quantity"],
    "syncMode": "changes-only",
    "jsonPath": "data.products",
    "maxItems": 10000
}

📤 Output (changes-only mode)

{
    "id": "SKU-123",
    "changeType": "modified",
    "changedFields": [
        {
            "field": "price",
            "oldValue": 99.99,
            "newValue": 89.99
        }
    ],
    "oldItem": { "id": "SKU-123", "price": 99.99 },
    "newItem": { "id": "SKU-123", "price": 89.99 },
    "detectedAt": "2024-01-15T10:30:00Z"
}

🔄 Sync Modes

Mode	Description
changes-only	Only output changes (added/modified/deleted)
incremental	Output all items with change metadata
full	Output all items (like regular scraper)

💡 Use Cases

1. Price Monitoring: Track product price changes
2. Stock Tracking: Monitor inventory changes
3. Content Updates: Detect new/updated articles
4. Competitor Analysis: Track listing changes
5. Data Warehousing: Sync only deltas to reduce storage

📊 Pricing

Suggested: $0.0001 per change detected

⚙️ How It Works

1. Fetch data from source URLs
2. Compare with previous run's data
3. Detect added/modified/deleted items
4. Output changes based on sync mode
5. Save current data for next run

The previous data is kept in your account's named key-value store incremental-sync-state, one record per set of URLs and settings, so scheduled runs (or tasks) with the same input compare against the last run automatically. To compare against a specific earlier run instead, pass its dataset ID as previousDatasetId.

How to use Incremental Sync Engine

1. Open Incremental Sync Engine on Apify Store and click Try for free.
2. Fill in the input (the form is prefilled with a working example) and click Start.
3. Watch results appear in the Output tab, then download them as JSON, CSV, Excel or HTML, or read them through the API.
4. Optional: save the input as a task and add a schedule to run it automatically, or connect it to your tools with integrations and webhooks.

Use Incremental Sync Engine with AI agents (MCP)

Give Claude, ChatGPT, Cursor, VS Code or any MCP client this Actor as a tool through the hosted Apify MCP server. The agent sees the input fields with descriptions and example values, runs the Actor, and reads the results.

MCP server URL (this Actor only):

https://mcp.apify.com/?tools=technicaldost/incremental-sync-engine

Claude Code:

$claude mcp add --transport http apify "https://mcp.apify.com/?tools=technicaldost/incremental-sync-engine" --header "Authorization: Bearer YOUR_APIFY_TOKEN"

Cursor, VS Code, Claude Desktop and other clients (mcp.json or the client's MCP settings):

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=technicaldost/incremental-sync-engine",
      "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }
    }
  }
}

Clients that support sign-in (such as Claude.ai connectors) can use the URL without a token and log in to Apify instead. Get your token in Apify Console → Settings → API & Integrations, or generate a ready config for your client at mcp.apify.com. In the agent the tool is named technicaldost--incremental-sync-engine; ask it, for example, to "run Incremental Sync Engine with {"startUrls": [{"url": "https://jsonplaceholder.typicode.com/posts"}], "dataType": "json", "identifierField": "id", "syncMode": "full"}".

AI agents without an Apify account can also run and pay for this Actor per request with x402 agentic payments.

Run Incremental Sync Engine via API

Run the Actor and get its results in one call (replace YOUR_APIFY_TOKEN):

curl -X POST "https://api.apify.com/v2/acts/technicaldost~incremental-sync-engine/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://jsonplaceholder.typicode.com/posts"}], "dataType": "json", "identifierField": "id", "syncMode": "full"}'

Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("technicaldost/incremental-sync-engine").call(run_input={"startUrls": [{"url": "https://jsonplaceholder.typicode.com/posts"}], "dataType": "json", "identifierField": "id", "syncMode": "full"})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript / TypeScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('technicaldost/incremental-sync-engine').call({"startUrls": [{"url": "https://jsonplaceholder.typicode.com/posts"}], "dataType": "json", "identifierField": "id", "syncMode": "full"});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Long runs: start them with POST /v2/acts/technicaldost~incremental-sync-engine/runs and fetch the dataset when the run finishes, or receive a webhook.

Integrations

Incremental Sync Engine works with Make, Zapier, n8n, LangChain, Google Sheets, Slack, GitHub and webhooks. Schedule runs in Apify Console and send new results straight to your apps, spreadsheets or AI pipelines.

FAQ

Can I use Incremental Sync Engine with AI agents like Claude or ChatGPT?

Yes. Connect the Apify MCP server with ?tools=technicaldost/incremental-sync-engine (see above) and the agent can run the Actor and read its results on its own.

Can I call Incremental Sync Engine from my own code?

Yes. Use the REST API, or the official Python and JavaScript clients shown above. Results are available as JSON, CSV, Excel, XML or HTML.

How do I run it on a schedule?

Save your input as a task in Apify Console and add a schedule (for example, daily). Use webhooks or integrations to send each run's results onward.

Is it legal to use this Actor?

It works with publicly available data. Make sure your use complies with the source website's terms and with data protection laws such as GDPR, especially for personal data.

Something doesn't work or I need a feature. What should I do?

Open an issue in the Issues tab on this Actor's page with an example input, and we'll look into it.