Only New Items

Run your scraper on a schedule and get back only what you have never seen before. A cross-run ledger you hold, not a database someone rents you.

Scheduled scrapers re-deliver the same records every run. Yesterday's 4,000 jobs come back today with 40 new ones buried inside, and every downstream step, the alert, the enrichment call, the LLM summary, the row in your warehouse, pays for all 4,000 again. This actor sits between the scrape and everything after it and passes through only the records that are actually new.

Part of the Broomwagon family: deterministic tools that watch scraped and AI data change. Records come out exactly as they went in, minus the ones you already had.

The pipeline: scrape → subtract → act

1. Scrape (any actor). Your scheduled scraper runs and produces a dataset, the same full listing it produces every time.
2. Subtract (this actor). Point datasetId at that run's dataset and name the field that identifies an item, e.g. ["url"] or ["jobId"]. Everything already in your ledger is held back.
3. Act. The output dataset holds only new items, so the expensive stage downstream, notification, enrichment, ingestion, only ever touches the delta.

Chaining is zero-glue: create one key-value store in your account, put its ID in memoryStoreId, and schedule this actor right after your scraper. The ledger accumulates itself, run after run.

What counts as the same item

Exact matching, deliberately. Values are trimmed and compared as they are: no case folding, no punctuation stripping, no fuzzy scoring. That is a different job, and doing it quietly here would merge records you meant to keep apart. If you need fuzzy identity or within-run merging of near-duplicates, use Dataset Deduper instead.

• With keyFields: only those fields decide identity. A job whose title, price, or scrapedAt changed but whose jobId did not is not new, which is usually what you want from a "notify me about new listings" pipeline.
• Without keyFields: the whole record decides. Any changed field makes it new, which is what you want when you are watching for any change at all.

Stateless, with memory

The ledger lives in the MEMORY key-value record that every run returns: a compressed Bloom filter holding fingerprints, never your records. Pass it back next run (memory), or point the actor at a key-value store in your account (memoryStoreId) and scheduled runs chain themselves. Broomwagon stores nothing.

Its size depends on how full it is, not on how much data you fed it. At the default capacity of 100,000 items it is about 2 KB while lightly loaded and grows toward roughly 150 KB as it fills. Small enough for a key-value record throughout, which is why memoryStoreId is the path to use past the two-minute demo: nobody wants to paste 150 KB into a form.

A Bloom filter trades a little accuracy for that size, and it trades it in the safe direction: it can occasionally withhold a new item it believes it has already seen, and it can never deliver something twice. Every run reports the current estimate. Set ledgerCapacity to roughly the number of distinct items you expect to accumulate and the estimate stays around 1%.

Try it in two minutes (no scraper needed)

Run 1. Leave the three sample records in Inline records and start the run. All three come out, because the ledger is empty, along with a MEMORY record: your ledger, held by you.

Run 2. Copy the MEMORY object into Previous memory, add a fourth record to the samples, and run again. Only the fourth comes out. In real use, replace the pasted records with your scraper run's dataset as described above.

Input

• datasetId | items | fileUrl: the records to filter. A picked dataset wins.
• keyFields: what identifies an item. Empty means the whole record.
• ledgerCapacity: how many distinct items to size the ledger for (default 100,000).
• memory / memoryStoreId: the ledger, as above.

Output

• Dataset: the new records, byte-for-byte as they arrived. This actor subtracts; it never edits.
• OUTPUT: run report, with counts for checked, delivered, already delivered, repeated within this run, and the ledger's current false-positive estimate.
• MEMORY: your ledger, to send back next run.

Pricing (pay-per-event)

Event	What you pay for	Price
record-processed	Per record checked	$0.0005
apify-actor-start	Run start, per GB of run memory	$0.005

Worked example: a 5,000-listing scrape checked daily: 5,000 × $0.0005 = $2.50 per run, about $75/month, and every downstream step after it only processes the handful that changed. If your per-record cost downstream is an LLM call or an enrichment API, this pays for itself in the first run. Priced below the vertical dedupers on purpose: this is the volume tool.

Set a max charge on any run (Maximum cost per run in Console, or ACTOR_MAX_TOTAL_CHARGE_USD via API) and the actor stops cleanly at your budget.

Integrations

• Apify Schedule (zero glue): create a key-value store once, put its ID in memoryStoreId, and schedule this actor right after your scraper.
• API: POST https://api.apify.com/v2/acts/broomwagon~only-new-items/runs with {"datasetId": "<scrape run's dataset>", "keyFields": ["url"], "memory": <last MEMORY record>}.
• n8n / Make: scraper node → this actor → your notification or enrichment branch. The branch now fires once per genuinely new item instead of once per row per day.
• MCP / AI agents: callable as a tool via the Apify MCP server. An agent that polls a source can round-trip the MEMORY object as tool context and only reason about deltas, which is the difference between a cheap agent and an expensive one.

Roadmap

Optional time-to-live on ledger entries (for sources that legitimately re-list), and a "changed since last time" mode that emits the diff rather than the record.

The Broomwagon family

This actor is one of nine deterministic post-processing tools from Broomwagon: the layer that follows your scrapers and agents, cleaning and watching what they produce. Same input, same output, every time.

• Scraper Output Monitor — catch the day your scrape silently breaks.
• Dataset Deduper — exact and fuzzy dedupe for any dataset, with an audit trail.
• CSV Doctor — repair broken CSV files, with a report of every fix.
• LLM Output Guard — validate LLM and agent JSON against your schema.
• PII Redactor — strip emails, phones, SSNs, cards, and addresses, deterministically.
• Record Linker — fuzzy join two datasets that share no key.
• Google Maps Scraper Deduper — merge duplicate places across Google Maps scrapes.
• AI Search Visibility Monitor — track your brand's share of voice in AI answers.