# Only New Items | Cross-Run Dedupe Ledger for Any Dataset (`broomwagon/only-new-items`) Actor

Run your scraper on a schedule and get back only what you have never seen before. Name the field that identifies an item, pipe any dataset in, and this actor holds back everything already delivered. Records pass through untouched. The ledger is yours, not a database you rent.

- **URL**: https://apify.com/broomwagon/only-new-items.md
- **Developed by:** [Brandon Mensing](https://apify.com/broomwagon) (community)
- **Categories:** Developer tools, Automation, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 record checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Only New Items

<img align="right" src="https://broomwagon.vercel.app/icons/brand-mark.png" width="64" alt="broomwagon.">

> Run your scraper on a schedule and get back only what you have never seen before. A cross-run ledger you hold, not a database someone rents you.

Scheduled scrapers re-deliver the same records every run. Yesterday's 4,000 jobs come
back today with 40 new ones buried inside, and every downstream step, the alert, the
enrichment call, the LLM summary, the row in your warehouse, pays for all 4,000 again.
This actor sits between the scrape and everything after it and passes through **only the
records that are actually new**.

Part of the **Broomwagon** family: deterministic tools that watch scraped and AI data
change. Records come out exactly as they went in, minus the ones you already had.

### The pipeline: scrape → subtract → act

1. **Scrape (any actor).** Your scheduled scraper runs and produces a dataset, the same
   full listing it produces every time.
2. **Subtract (this actor).** Point `datasetId` at that run's dataset and name the field
   that identifies an item, e.g. `["url"]` or `["jobId"]`. Everything already in your
   ledger is held back.
3. **Act.** The output dataset holds only new items, so the expensive stage downstream,
   notification, enrichment, ingestion, only ever touches the delta.

Chaining is zero-glue: create one key-value store in your account, put its ID in
`memoryStoreId`, and schedule this actor right after your scraper. The ledger accumulates
itself, run after run.

### What counts as the same item

Exact matching, deliberately. Values are trimmed and compared as they are: no case
folding, no punctuation stripping, no fuzzy scoring. That is a different job, and doing
it quietly here would merge records you meant to keep apart. If you need fuzzy identity
or within-run merging of near-duplicates, use **Dataset Deduper** instead.

- **With `keyFields`:** only those fields decide identity. A job whose title, price, or
  `scrapedAt` changed but whose `jobId` did not is *not* new, which is usually what you
  want from a "notify me about new listings" pipeline.
- **Without `keyFields`:** the whole record decides. Any changed field makes it new,
  which is what you want when you are watching for *any* change at all.

### Stateless, with memory

The ledger lives in the `MEMORY` key-value record that every run returns: a compressed
Bloom filter holding fingerprints, never your records. Pass it back next run (`memory`),
or point the actor at a key-value store in **your** account (`memoryStoreId`) and
scheduled runs chain themselves. Broomwagon stores nothing.

Its size depends on how full it is, not on how much data you fed it. At the default
capacity of 100,000 items it is about 2 KB while lightly loaded and grows toward roughly
150 KB as it fills. Small enough for a key-value record throughout, which is why
`memoryStoreId` is the path to use past the two-minute demo: nobody wants to paste 150 KB
into a form.

A Bloom filter trades a little accuracy for that size, and it trades it in the safe
direction: it can occasionally withhold a new item it believes it has already seen, and
it can **never** deliver something twice. Every run reports the current estimate. Set
`ledgerCapacity` to roughly the number of distinct items you expect to accumulate and
the estimate stays around 1%.

### Try it in two minutes (no scraper needed)

**Run 1.** Leave the three sample records in **Inline records** and start the run. All
three come out, because the ledger is empty, along with a `MEMORY` record: your ledger,
held by you.

**Run 2.** Copy the `MEMORY` object into **Previous memory**, add a fourth record to the
samples, and run again. Only the fourth comes out. In real use, replace the pasted
records with your scraper run's dataset as described above.

### Input

- `datasetId` | `items` | `fileUrl`: the records to filter. A picked dataset wins.
- `keyFields`: what identifies an item. Empty means the whole record.
- `ledgerCapacity`: how many distinct items to size the ledger for (default 100,000).
- `memory` / `memoryStoreId`: the ledger, as above.

### Output

- Dataset: the new records, byte-for-byte as they arrived. This actor subtracts; it
  never edits.
- `OUTPUT`: run report, with counts for checked, delivered, already delivered, repeated
  within this run, and the ledger's current false-positive estimate.
- `MEMORY`: your ledger, to send back next run.

### Pricing (pay-per-event)

| Event | What you pay for | Price |
| --- | --- | --- |
| `record-processed` | Per record checked | $0.0005 |
| `apify-actor-start` | Run start, per GB of run memory | $0.005 |

**Worked example:** a 5,000-listing scrape checked daily: 5,000 × $0.0005 = $2.50 per
run, about **$75/month**, and every downstream step after it only processes the handful
that changed. If your per-record cost downstream is an LLM call or an enrichment API,
this pays for itself in the first run. Priced below the vertical dedupers on purpose:
this is the volume tool.

Set a max charge on any run (`Maximum cost per run` in Console, or
`ACTOR_MAX_TOTAL_CHARGE_USD` via API) and the actor stops cleanly at your budget.

### Integrations

- **Apify Schedule (zero glue):** create a key-value store once, put its ID in
  `memoryStoreId`, and schedule this actor right after your scraper.
- **API:** `POST https://api.apify.com/v2/acts/broomwagon~only-new-items/runs` with
  `{"datasetId": "<scrape run's dataset>", "keyFields": ["url"], "memory": <last MEMORY record>}`.
- **n8n / Make:** scraper node → this actor → your notification or enrichment branch.
  The branch now fires once per genuinely new item instead of once per row per day.
- **MCP / AI agents:** callable as a tool via the Apify MCP server. An agent that polls a
  source can round-trip the `MEMORY` object as tool context and only reason about deltas,
  which is the difference between a cheap agent and an expensive one.

### Roadmap

Optional time-to-live on ledger entries (for sources that legitimately re-list), and a
"changed since last time" mode that emits the diff rather than the record.

### The Broomwagon family

This actor is one of nine deterministic post-processing tools from [Broomwagon](https://apify.com/broomwagon): the layer that follows your scrapers and agents, cleaning and watching what they produce. Same input, same output, every time.

- [Scraper Output Monitor](https://apify.com/broomwagon/scraper-output-monitor) — catch the day your scrape silently breaks.
- [Dataset Deduper](https://apify.com/broomwagon/dataset-deduper) — exact and fuzzy dedupe for any dataset, with an audit trail.
- [CSV Doctor](https://apify.com/broomwagon/csv-doctor) — repair broken CSV files, with a report of every fix.
- [LLM Output Guard](https://apify.com/broomwagon/llm-output-guard) — validate LLM and agent JSON against your schema.
- [PII Redactor](https://apify.com/broomwagon/pii-redactor) — strip emails, phones, SSNs, cards, and addresses, deterministically.
- [Record Linker](https://apify.com/broomwagon/record-linker) — fuzzy join two datasets that share no key.
- [Google Maps Scraper Deduper](https://apify.com/broomwagon/google-maps-deduper) — merge duplicate places across Google Maps scrapes.
- [AI Search Visibility Monitor](https://apify.com/broomwagon/ai-search-visibility-monitor) — track your brand's share of voice in AI answers.

# Actor input Schema

## `datasetId` (type: `string`):

This run's scraper output. Pipe any actor's dataset straight in; a picked dataset always wins over the sample records below.

## `items` (type: `array`):

Records provided inline as a JSON array, for trying the actor out before wiring a scraper up. Ignored when a dataset is picked above.

## `fileUrl` (type: `string`):

URL of a JSON or NDJSON file of records.

## `keyFields` (type: `array`):

The fields that identify an item, e.g. \["url"] or \["jobId"]. Matching is exact after trimming: no case folding, no fuzzy matching. Leave empty to key on the whole record, which makes any changed field a new item.

## `ledgerCapacity` (type: `integer`):

How many distinct items the ledger is sized for. It stays a few kilobytes either way; going far past this raises the rate at which a new item is mistaken for an old one. The run report always tells you where you stand.

## `memory` (type: `object`):

An object beginning with "memoryVersion", copied from the MEMORY record in your last run's key-value store (not your records, those go in the fields above). It holds the ledger of everything already delivered.

## `memoryStoreId` (type: `string`):

Optional: a key-value store in YOUR account. The actor reads the ledger from it before running and writes the update back after, so scheduled runs chain automatically with nothing to copy by hand.

## Actor input object example

```json
{
  "items": [
    {
      "jobId": "a1",
      "title": "Senior Data Engineer",
      "company": "Northwind",
      "url": "https://example.com/jobs/a1"
    },
    {
      "jobId": "b2",
      "title": "Product Designer",
      "company": "Northwind",
      "url": "https://example.com/jobs/b2"
    },
    {
      "jobId": "c3",
      "title": "Support Analyst",
      "company": "Contoso",
      "url": "https://example.com/jobs/c3"
    }
  ],
  "keyFields": [
    "jobId"
  ],
  "ledgerCapacity": 100000
}
```

# Actor output Schema

## `newItems` (type: `string`):

Only the records this run had never delivered before, passed through untouched.

## `memory` (type: `string`):

The ledger of everything delivered so far. Send it back as `memory` on the next run, or skip this by using `memoryStoreId`.

## `report` (type: `string`):

Records checked, delivered, and held back, plus the ledger's current false-positive estimate.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "jobId": "a1",
            "title": "Senior Data Engineer",
            "company": "Northwind",
            "url": "https://example.com/jobs/a1"
        },
        {
            "jobId": "b2",
            "title": "Product Designer",
            "company": "Northwind",
            "url": "https://example.com/jobs/b2"
        },
        {
            "jobId": "c3",
            "title": "Support Analyst",
            "company": "Contoso",
            "url": "https://example.com/jobs/c3"
        }
    ],
    "keyFields": [
        "jobId"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("broomwagon/only-new-items").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "items": [
        {
            "jobId": "a1",
            "title": "Senior Data Engineer",
            "company": "Northwind",
            "url": "https://example.com/jobs/a1",
        },
        {
            "jobId": "b2",
            "title": "Product Designer",
            "company": "Northwind",
            "url": "https://example.com/jobs/b2",
        },
        {
            "jobId": "c3",
            "title": "Support Analyst",
            "company": "Contoso",
            "url": "https://example.com/jobs/c3",
        },
    ],
    "keyFields": ["jobId"],
}

# Run the Actor and wait for it to finish
run = client.actor("broomwagon/only-new-items").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    {
      "jobId": "a1",
      "title": "Senior Data Engineer",
      "company": "Northwind",
      "url": "https://example.com/jobs/a1"
    },
    {
      "jobId": "b2",
      "title": "Product Designer",
      "company": "Northwind",
      "url": "https://example.com/jobs/b2"
    },
    {
      "jobId": "c3",
      "title": "Support Analyst",
      "company": "Contoso",
      "url": "https://example.com/jobs/c3"
    }
  ],
  "keyFields": [
    "jobId"
  ]
}' |
apify call broomwagon/only-new-items --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,broomwagon/only-new-items"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Kuh7Pkm77hzQVILzy/builds/jEkfm7fPHUHPKbkCZ/openapi.json