Scraper Output Monitor

Catch the day your scrape silently breaks: field drift, completeness decay, and volume anomalies, judged against your own rolling baseline.

Scrapers rarely fail loudly. The run goes green, the dataset has rows, and only weeks later does someone notice that phone has been empty since the site redesign, or that price started arriving as a string, or that 4,000 listings quietly became 400. This actor is the check that runs right after your scrape and answers one question every time: is this output normal for this scrape?

Part of the Broomwagon family: deterministic tools that watch scraped and AI data change. No LLM guesses at your data quality. Same records in, same verdicts out.

The pipeline: scrape → check → alert

1. Scrape (any actor). Your scheduled scraper runs and produces a dataset. Nothing about it changes.
2. Check (this actor). Point datasetId at that run's dataset. Every field is profiled: how often it carries a value, what types those values are, whether it is there at all, and how many records came back.
3. Alert. Each run emits one run-health verdict plus one field-health row per field, with the delta and z-score against your baseline and anomaly: true when the change is statistically real rather than Tuesday noise. Filter on verdict: "alert" in n8n, Make, or a webhook and route it to Slack. That is the whole loop.

Chaining is zero-glue: create one key-value store in your account, put its ID in memoryStoreId, and schedule this actor right after your scraper. Baselines accumulate themselves, run after run.

What it watches

Signal	What breaking looks like	Verdict
Record volume	4,000 listings become 400, or zero	volume-drop / volume-spike
Field completeness	phone was 94% filled, now 3%	fill-drop / fill-spike
Schema drift	phone stops appearing in records at all	missing-field
Schema drift	a field you have never seen shows up	new-field
Type drift	price arrives as "19.99" instead of 19.99	type-shift

A field counts as filled only when it carries something usable. null, "", [], and {} are all gaps, because a "reviews": [] from a broken selector is not a value.

Stateless, with memory

Every run returns a compact aggregate (the MEMORY key-value record) holding your rolling per-field baselines: sufficient statistics, never your data. Pass it back next run (memory), or point the actor at a key-value store in your account (memoryStoreId) and scheduled runs chain automatically. Broomwagon stores nothing; your history is yours, and the memory stays a few kilobytes no matter how big the scrape.

First run establishes the baseline. Deltas begin on run two. Alerts engage after five runs of baseline, so a pipeline that is still settling stays quiet. A field that vanishes for good stops alerting after three runs and drops out of the baseline: the monitor accepts the new normal rather than nagging forever.

Try it in two minutes (no scraper needed)

Run 1. Leave the sample records in Inline records and start the run. You get a run-health verdict, one field-health row per field, and a MEMORY record in the run's key-value store: your baselines, held by you. Everything reads baseline, which is correct, there is nothing to compare to yet.

Run 2. Copy the MEMORY object into Previous memory, delete a field from one of the sample records, and run again. That field's row now carries previousFillRate, a negative delta, and after five runs of baseline, anomaly: true. In real use, replace the pasted records with your scraper run's dataset as described above; a picked dataset always wins over the samples in the form.

Input

• datasetId | items | fileUrl: the scrape output to check. A picked dataset wins.
• fields: limit the watch list, e.g. ["name", "price", "phone"]. Default: every field, which is also how new fields get noticed.
• memory / memoryStoreId: the memory, as above.

Output

• Dataset row 1, kind: "run-health": itemCount, previousItemCount, delta, zScore, status, fieldsTracked, fieldsNew, fieldsMissing, anomalies, and verdict (ok, baseline, or alert).
• Then one kind: "field-health" row per field: fillRate, previousFillRate, delta, zScore, dominantType, previousDominantType, status, anomaly.
• OUTPUT: run report. MEMORY: your rolling baselines, to send back next run.

Pricing (pay-per-event)

Event	What you pay for	Price
run-report	Per scrape checked	$0.02
anomaly-alert	Per real problem detected	$0.05
apify-actor-start	Run start, per GB of run memory	$0.005

Worked example: one scraper, checked daily: 30 × $0.025 = $0.75/month, plus a nickel per genuine alert (a healthy month has none). Ten scrapers checked daily: $7.50/month for continuous coverage of the whole pipeline. Note that the price does not move with dataset size, a 500K-row scrape costs the same to check as a 50-row one.

Set a max charge on any run (Maximum cost per run in Console, or ACTOR_MAX_TOTAL_CHARGE_USD via API) and the actor stops cleanly at your budget.

Integrations

• Apify Schedule (zero glue): create a key-value store once, put its ID in memoryStoreId, and schedule this actor right after your scrape. Baselines, deltas, and alerts accumulate automatically.
• API: POST https://api.apify.com/v2/acts/broomwagon~scraper-output-monitor/runs with {"datasetId": "<scrape run's dataset>", "memory": <last MEMORY record>}. Read MEMORY back from the run's key-value store for the next call.
• n8n / Make: scraper node → this actor → filter on verdict: "alert" → Slack or PagerDuty. Four nodes and your scrapes stop failing silently.
• MCP / AI agents: callable as a tool via the Apify MCP server; agents round-trip the MEMORY object as tool context between calls, so an agent that scrapes on a schedule can tell when its own source went bad.

Roadmap

Per-field value distributions (catching "the price column is now all zeros" when the fill rate never moved), cross-field consistency rules, and drift attribution against a named previous run. Built by an ex-Elastic engineer who spent years on log analytics: this is time-series observability pointed at your scrape output.

The Broomwagon family

This actor is one of nine deterministic post-processing tools from Broomwagon: the layer that follows your scrapers and agents, cleaning and watching what they produce. Same input, same output, every time.

• Only New Items — deliver only records you have never delivered before.
• Dataset Deduper — exact and fuzzy dedupe for any dataset, with an audit trail.
• CSV Doctor — repair broken CSV files, with a report of every fix.
• LLM Output Guard — validate LLM and agent JSON against your schema.
• PII Redactor — strip emails, phones, SSNs, cards, and addresses, deterministically.
• Record Linker — fuzzy join two datasets that share no key.
• Google Maps Scraper Deduper — merge duplicate places across Google Maps scrapes.
• AI Search Visibility Monitor — track your brand's share of voice in AI answers.