Datasource Pulse

Know when any data connector fails silently — even when everything looks "up."

Datasource Pulse monitors your API credentials, usage quotas, and scraper output quality across your whole pipeline, then sends one alert when something degrades and one notice when it recovers. It runs on a schedule you choose, needs nothing but the keys you want watched, and is billed per connector it checks.

---

For agents & programmatic use

Datasource Pulse is agent-first: discoverable and callable as a tool via the Apify API, MCP, and x402 agentic payments (USDC on Base — no Apify account required). Everything an agent needs to decide, invoke, and pay is here:

• What it does: takes a list of data connectors, checks each one's health, returns one record per connector.

• Input (INPUT_SCHEMA) — two mutually exclusive modes; clientToken takes precedence:

  // Mode A (self-serve) — supply checks inline:
  {
    "checks": ConnectorCheck[],   // REQUIRED in Mode A (non-empty); billed $0.02 each
    "onlyConnectorIds"?: string[],// subset recheck — probe & bill ONLY these ids (see bullet below)
    "webhookUrl"?: string,        // any HTTP endpoint; Slack-compatible card body by default
    "webhookSecret"?: string,     // optional — adds X-DP-Signature (HMAC-SHA256) to webhook POSTs
    "webhookFormat"?: "slack"|"json", // default "slack" (chat card); "json" = raw structured event object (see Alerting)
    "webhookExpectJsonOk"?: boolean, // when true, a 2xx must carry {"ok":true} to count as delivered
    "emailTo"?: string,           // email via your webhook relay (see Alerting)
    "quotaWarningPct"?: number    // default 80
  }
  
  // Mode B (managed) — supply only a provisioned token; the manifest loads server-side:
  { "clientToken": string, "onlyConnectorIds"?: string[] }
  

  If clientToken is present, Mode B is used and checks is ignored. Otherwise Mode A requires a non-empty checks array.

  ConnectorCheck is either { "type":"credential", "id", "label", "vendor": "serpapi|apify|hunter|openai|anthropic|twitter|reddit|github|stripe|resend|producthunt|gemini|groq|mistral|cohere|perplexity|together|openrouter|firecrawl|exa|tavily|brightdata|scrapingbee|zenrows|elevenlabs|deepgram|assemblyai|replicate|huggingface|pinecone|supabase|airtable|stackexchange|google_oauth|youtube|serper|brave|generic_http", ...credential fields } Credential fields: every vendor takes apiKey except reddit (clientId+clientSecret), google_oauth (clientId+clientSecret+refreshToken), twitter (bearerToken), supabase project checks (url+apiKey; apiKey alone = management token), and generic_http (url+authHeader+successStatusCode+optional successJsonPath/successJsonValue). or { "type":"actor", "id", "label", "actorId", "apifyToken", "enableVolumeBaseline":bool, "enableCostBaseline":bool, "stalenessThresholdDays"?, "volumeDropPct"?, "costSpikePct"? }. Optional on both shapes: impact (string ≤ 200 chars) — what breaks downstream when this connector fails; echoed verbatim in every alert, so the operator's first question ("what breaks?") is answered in the alert itself. Optional on credential checks: quotaSoftCap (number) — your own budget line in the vendor's native usage unit; quota warnings and the 100%-crossed warning state are evaluated against your cap (the incident that matters happens at your budget line, not the vendor's billing wall), with the vendor-account % always reported alongside in metadata.quota_used_pct vs metadata.quota_soft_cap_used_pct. Units: searches (serpapi), USD (apify, openrouter), requests (hunter), credits (firecrawl, scrapingbee), characters (elevenlabs), API calls (tavily, stackexchange). If the vendor probe exposes no raw usage number, the record says so explicitly (quota_soft_cap_status: "unavailable") — never silently ignored.

• Output — one dataset record per connector, read from the run's default dataset (.../datasets/{id}/items):

  { "connector_id", "connector_label", "check_type": "credential|actor",
    "state": "healthy|warning|failing", "failure_class": string|null,
    "event_source": "scheduled_probe", "message": string, "metadata": {} }
  

• Cost is computable: total = checks.length × $0.02, charged on attempt (a failing connector still counts). No start fee, no per-run minimum.

• Subset recheck (preflight): onlyConnectorIds probes and bills ONLY the listed ids — recheck one credential immediately before an expensive job (1 × $0.02) instead of paying for the full roster. Unknown ids fail the run before anything is charged. Worst-case latency per check is the probe's hard timeout — 15 s (credential) / 30 s (actor) — plus ~5–10 s platform boot; typical healthy checks answer in 1–3 s. Note: actor checks read run metadata — Datasource Pulse never executes your Actors, so slow multi-phase scrapers do not slow the probe.

• Stable failure_class enum (values an agent can actually receive): dead_token, quota_exhausted, rate_limited, timeout, unexpected_response, actor_failed, volume_drop, schema_drift, cost_spike, actor_stale.

• One-time account approval: the very first run on a new Apify account must approve the Actor's full permissions (its named key-value store is the alert-dedup state memory). Via API/MCP this surfaces as full-permission-actor-not-approved — open the Actor once in Apify Console, approve, retry; every later run (API, MCP, scheduled) proceeds without interaction. Billing note: the run's chargedEventCounts can settle a few minutes after the run finishes — re-read before treating a fresh 0 as a billing failure.

---

Why you need it

Any product built on third-party data degrades silently. A dead API token, an exhausted quota, a scraper whose maintainer pushed a breaking schema change — none of these crash your app. They just make your output worse, and you find out weeks later from a confused customer.

Apify's own built-in monitoring is free and covers your Actors' run status — but it knows nothing about your SerpAPI quota, your GitHub PAT, or whether the TikTok or LinkedIn Actor your pipeline depends on just had its output schema changed by its maintainer.

Datasource Pulse fills the gap: cross-vendor credential health + dependency-watching on Actors you don't own, unified into one alert stream.

• ✅ Catch credential expiration before production breaks — rotate a key on your schedule, not after a customer complains
• 📉 Track API quota usage proactively — know when you're at 80% before you hit 100%
• 🔎 Detect scraper output drift early — field names change silently; you'll know the run it happens
• ⚡ Reduce incident response time — one Slack message the moment something degrades, not hours later
• 🛡️ Maintain SLAs for downstream consumers — your data products stay reliable even when vendors aren't
• 👀 Gain visibility across your entire vendor stack — one dashboard instead of logging into every vendor console separately

---

How Datasource Pulse works

Run flow (text, for agents):

trigger (scheduled run OR x402 agent)
  └─ for each connector:
       ├─ probe:  credential/quota health  (Probe Type 1, dozens of vendors)
       │          OR scraper output quality (Probe Type 2, Actors you depend on)
       ├─ compare result to previous state (Apify key-value store)
       └─ on state change:
            ├─ send ONE deduplicated alert  (Slack-compatible webhook / email via your relay)
            └─ write ONE dataset record per connector (health history)
billing: $0.02 per connector checked, charged on attempt (failures still count)

Each run probes every connector you configured, compares the result to the state stored from the previous run, and fires exactly one alert the moment a connector changes state — then stays quiet until it recovers. Every run also writes a health-history record per connector to the Apify dataset, so you can build your own dashboard on top.

---

What Datasource Pulse monitors

Probe Type 1 — Credential & quota health

The cheapest authenticated call per vendor — enough to prove the token is alive and read current usage.

Looking for TikTok, Google Maps, or LinkedIn? Those are scrapers, not API keys — Datasource Pulse watches them with Probe Type 2 just below: output schema drift, failure spikes, staleness, volume drops. Platforms like LinkedIn and TikTok keep their official APIs locked down, so pipelines run on scrapers that change constantly — which is exactly why they need watching.

Vendor	What it checks
SerpAPI	Token validity + remaining monthly searches
Apify	Account token validity + monthly usage vs. plan cap
Hunter.io	Token validity + requests remaining
OpenAI	Key validity (no tokens consumed)
Anthropic	Key validity (no tokens consumed)
X/Twitter API	Bearer token validity + rate-limit window
Reddit OAuth	Client credentials flow success
GitHub	PAT validity + core rate-limit remaining
ProductHunt	Developer token validity
Stripe	Restricted read-only key validity
Resend	API key validity
Google Gemini (AI Studio)	Key validity via free models list
Groq	Key validity via free models list
Mistral AI	Key validity via free models list
Cohere	Key validity via dedicated check-api-key endpoint
Perplexity	Key validity — authenticated pre-generation rejection, zero tokens billed
Together AI	Key validity via free models list
OpenRouter	Key validity + credit usage vs. limit
Firecrawl	Key validity + remaining credits
Exa (Metaphor)	Key validity — authenticated pre-search rejection, zero credits used
Tavily	Key validity + plan usage vs. limit
Bright Data	API token validity + active zone count
ScrapingBee	Key validity + credit usage vs. plan
ZenRows	Key validity + subscription details
ElevenLabs	Key validity + character quota usage
Deepgram	Key validity via token introspection
AssemblyAI	Key validity (no audio processed)
Replicate	API token validity
Hugging Face	Access token validity + token role
Pinecone	Key validity + index count
Supabase	Project key acceptance, or management token validity
Airtable	Personal access token validity + scope count
Stack Exchange	Key validity + shared daily quota + backoff signal
Google OAuth (refresh token)	Refresh-token validity via a free token mint — catches revoked/expired Google-API access
YouTube Data API	Key validity + quota-exhaustion detection (1 unit/check; Google exposes no usage readout API)
Serper	Key validity — authenticated pre-search rejection, zero searches billed
Brave Search	Token validity — auth-coded rejection, zero queries billed
Generic HTTP	Your URL + auth header + success condition

Failure classifications (named in every alert so you know the fix): dead_token, quota_exhausted, rate_limited, timeout, unexpected_response.

Probe Type 2 — Scraper output quality

Monitors Apify Actors you depend on but did not build — the TikTok, Google Maps, and LinkedIn scrapers that power most Apify pipelines, Instagram and YouTube scrapers, or any other Actor in the Store, by actorId.

Signal	Toggle
Last successful run + timestamp	Always on
7-day and 30-day success/failure rate	Always on
Actor lastModifiedAt staleness	Always on
Output schema drift (field names changed)	Always on
Actor build ID (recorded in metadata for context; not a standalone alert)	Always on
Output record volume vs. rolling baseline	Per-check toggle
Compute cost per run vs. rolling baseline	Per-check toggle

Volume and cost baselines assume the same query runs repeatedly. If your Actor runs use different inputs each time (different search terms, different targets), set enableVolumeBaseline: false and enableCostBaseline: false to avoid false alarms.

---

Start from a ready-made example

Every example below is a one-click task: open it, plug in your own keys or Actor ID, and hit Run.

• Validate OpenAI and Anthropic API keys and rate limits
• Monitor Clockworks TikTok Scraper for failures and drift
• Monitor Compass Google Maps Scraper for errors and staleness
• Alert when LinkedIn Scraper cookie or session expires
• Watch a third-party Apify Actor for breaking changes
• Monitor SerpAPI quota and alert on Slack when credits run low

Browse the full set on the Examples tab — TikTok, Google Maps, Instagram, LinkedIn, YouTube, schema drift, credential health, and quota alerts.

How to use Datasource Pulse

Start here — the 2-cent smoke test (no real keys required)

You shouldn't have to hand a monitoring product your API keys just to find out whether you trust it. So don't. Your first run can watch a public URL instead of a real credential — the full pipeline (probe → state memory → dataset record → billing) for exactly $0.02, without pasting a single key.

Step 1. Open the Actor in Apify Console and paste this as the input:

{
  "checks": [
    {
      "type": "credential",
      "id": "public-ping",
      "label": "Public ping — GitHub Zen",
      "vendor": "generic_http",
      "url": "https://api.github.com/zen",
      "successStatusCode": 200
    }
  ]
}

Step 2. In the run options, set Max cost per run to $0.50. Wild overkill for a 2-cent run — that's the point: the cap is a hard ceiling Datasource Pulse can never bill past, and you should see that ceiling work before you scale up.

Step 3. Hit Run. First run on your account only: Apify asks you to approve the Actor with account-level permissions. That approval powers the state memory that makes alerts fire once per problem instead of once per run — one click, once per account, never again (details in the First-run FAQ).

Step 4. When the run finishes, open the Dataset tab: one health record, "state": "healthy", and exactly one 2-cent connector-checked event billed. Probe, memory, record, price — the entire product in miniature, for less than a stick of gum.

Step 5. Make it yours: swap the ping for your real connectors (ready-made examples above, input reference below), add your Slack webhook, set a 6-hour schedule. From here on, silence means your stack is healthy — and the first message you do get is the one that saves you a debugging afternoon.

Agents: this smoke test doubles as the cheapest possible integration test of your Datasource Pulse wiring — same input contract, same output schema, same billing event as production runs. Run it once after wiring, assert state == "healthy" on the dataset record, proceed.

Mode A — Self-serve (paste keys, pick a schedule)

1. Open the Actor input in Apify Console
2. In the Connector checks field, add your checks as JSON (see examples below)
3. Add a Slack-compatible webhook URL (optionally with a signing secret) and/or an email-relay recipient
4. Set a schedule — run every 1–6 hours depending on how quickly you need to know about failures
5. Save and run — first alert within one schedule interval if anything is wrong

First run on this account? Apify will ask for a one-time permission approval before the run starts — expected, one click, explained in the First-run FAQ.

Mode B — Managed (enterprise, team accounts)

1. Contact the Actor operator to provision a client token for your account
2. In the Actor input, set only the Client Token field
3. Your full connector manifest is loaded from the operator's private store — no keys in the input UI

Operators: the full provisioning runbook — intake template, scoped-key requirements, live validation at onboarding, token rotation/revocation — is provided during onboarding.

Input examples

Credential check — SerpAPI

{
  "type": "credential",
  "id": "serpapi-main",
  "label": "SerpAPI — main account",
  "vendor": "serpapi",
  "apiKey": "YOUR_SERPAPI_KEY"
}

Credential check — GitHub PAT

{
  "type": "credential",
  "id": "github-pat",
  "label": "GitHub — main PAT",
  "vendor": "github",
  "apiKey": "ghp_YOUR_TOKEN"
}

Credential check — Reddit OAuth

{
  "type": "credential",
  "id": "reddit-oauth",
  "label": "Reddit — production app",
  "vendor": "reddit",
  "clientId": "YOUR_CLIENT_ID",
  "clientSecret": "YOUR_CLIENT_SECRET"
}

Credential check — Stripe (restricted read-only key only)

{
  "type": "credential",
  "id": "stripe-restricted",
  "label": "Stripe — restricted balance key",
  "vendor": "stripe",
  "apiKey": "rk_live_YOUR_RESTRICTED_KEY"
}

⚠️ Stripe security: Never use a live secret key (sk_live_...). Create a restricted key in Stripe Dashboard → Developers → Restricted keys with Balance: Read permission only. Datasource Pulse will refuse to probe a live secret key.

Credential check — Google OAuth (refresh token)

Validates that a Google OAuth refresh token can still mint access tokens (the token endpoint is free — no API quota consumed). Use it for any Google-API-scoped application (Sheets, Drive, Search Console, YouTube-with-OAuth):

{
  "type": "credential",
  "id": "google-oauth-sheets",
  "label": "Google Sheets — pipeline refresh token",
  "vendor": "google_oauth",
  "clientId": "YOUR_CLIENT_ID.apps.googleusercontent.com",
  "clientSecret": "YOUR_CLIENT_SECRET",
  "refreshToken": "YOUR_REFRESH_TOKEN"
}

An expired or revoked refresh token surfaces as dead_token (invalid_grant) — the failure mode that otherwise goes unnoticed until a nightly Google-API job breaks.

Credential check — single-API-key vendors (most vendors)

Every vendor not shown in a dedicated example above takes just apiKey — that includes gemini, groq, mistral, cohere, perplexity, together, openrouter, firecrawl, exa, tavily, brightdata, scrapingbee, zenrows, elevenlabs, deepgram, assemblyai, replicate, huggingface, pinecone, airtable, stackexchange, youtube, serper, and brave:

{
  "type": "credential",
  "id": "openrouter-main",
  "label": "OpenRouter — agent key",
  "vendor": "openrouter",
  "apiKey": "sk-or-YOUR_KEY"
}

Credential check — Supabase (two shapes)

With url, Datasource Pulse verifies your project's REST endpoint accepts the key (anon, service_role, or new sb_... keys):

{
  "type": "credential",
  "id": "supabase-prod",
  "label": "Supabase prod — service key",
  "vendor": "supabase",
  "url": "https://YOURPROJECT.supabase.co",
  "apiKey": "YOUR_SERVICE_OR_ANON_KEY"
}

Omit url to instead validate an account-level management token (sbp_...) against the Supabase Management API.

Credential check — Generic HTTP

{
  "type": "credential",
  "id": "my-internal-api",
  "label": "Internal API — health endpoint",
  "vendor": "generic_http",
  "url": "https://api.example.com/v1/status",
  "authHeader": "Bearer YOUR_TOKEN",
  "successStatusCode": 200,
  "successJsonPath": "status",
  "successJsonValue": "ok"
}

Scraper quality check — any Actor you depend on (TikTok, Google Maps, LinkedIn, …)

{
  "type": "actor",
  "id": "tiktok-scraper-watch",
  "label": "clockworks TikTok Scraper",
  "actorId": "clockworks~tiktok-scraper",
  "apifyToken": "YOUR_APIFY_TOKEN",
  "enableVolumeBaseline": false,
  "enableCostBaseline": false,
  "stalenessThresholdDays": 30,
  "volumeDropPct": 30,
  "costSpikePct": 50
}

Full Mode A input example

{
  "checks": [
    {
      "type": "credential",
      "id": "serpapi-main",
      "label": "SerpAPI — main account",
      "vendor": "serpapi",
      "apiKey": "YOUR_SERPAPI_KEY"
    },
    {
      "type": "credential",
      "id": "github-pat",
      "label": "GitHub — main PAT",
      "vendor": "github",
      "apiKey": "ghp_YOUR_TOKEN"
    },
    {
      "type": "actor",
      "id": "my-scraper-watch",
      "label": "My dependency Actor",
      "actorId": "some-author~some-actor",
      "apifyToken": "YOUR_APIFY_TOKEN",
      "enableVolumeBaseline": true,
      "enableCostBaseline": true
    }
  ],
  "webhookUrl": "https://hooks.slack.com/services/YOUR/WEBHOOK",
  "emailTo": "alerts@yourcompany.com",
  "quotaWarningPct": 80
}

---

How much does Datasource Pulse cost?

Datasource Pulse uses pay-per-event pricing: you pay $0.02 for every connector it checks — nothing else. No monthly subscription, no per-run start fee, no charge for runs that check nothing.

You are billed for the check itself, not the result. A dead token, an exhausted quota, or a rate-limited vendor still bills, because catching that failure is exactly the value you're paying for.

Your cost is easy to predict:

monthly cost  ≈  connectors  ×  runs per day  ×  30  ×  $0.02

You monitor	Schedule	Checks / day	≈ Cost / month
3 connectors	every 6h (4×/day)	12	~$7
5 connectors	every 3h (8×/day)	40	~$24
10 connectors	hourly (24×/day)	240	~$144
1 connector	hourly (24×/day)	24	~$14

Tune the cost to your needs: monitor fewer connectors, or lower the schedule frequency, and the bill drops proportionally. You can also cap spend with the max cost per run setting in the Actor's run options.

Recommended cadence (defensible defaults)

• Credential checks: 2–4×/day (every 6–12 hours) — catches dead keys within hours without surprise bills
• Hourly only for connectors already in warning — escalate frequency while quota creep is underway, drop back after recovery
• Scraper quality checks (Probe Type 2): once daily — dependency Actors change on release cadence, not by the hour

At these defaults a 5-connector stack runs $6–12/month; hourly-everything monitoring is almost never necessary.

Pay-as-an-agent (x402)

Datasource Pulse is enabled for agentic payments via the x402 protocol. An AI agent can discover, run, and pay for it in USDC on the Base network — no Apify account required. The agent calls the Actor, receives an HTTP 402, its wallet signs the payment, and the run proceeds. This makes Datasource Pulse a drop-in reliability check inside autonomous data pipelines.

---

First-run FAQ

Four things every new user — human or agent — trips over in the first ten minutes, answered up front. Surprise is the one thing a monitoring product should never produce.

Why does Apify ask me to approve extra permissions on the first run? Datasource Pulse remembers each connector's last known state between runs — that memory is exactly what makes it fire one alert per state change instead of spamming you every run. The memory lives in a named Apify key-value store, and Apify requires a one-time, account-level approval for any Actor that uses named storage. Approve once and every later run — manual, scheduled, or API-triggered — starts instantly. The permission is used for precisely what the run-flow diagram shows: the Actor's own state store and your run's dataset. Your input stays private to your account either way (see Security guidelines). Calling via API or MCP? A pending approval surfaces as full-permission-actor-not-approved — open the Actor once in Console, approve, retry.

The run finished but the charged-events counter shows 0 — did billing break? No. Apify's charged-events counter on the run page can lag a few minutes behind the run itself; it settles at exactly $0.02 × connectors checked, and your account usage page catches up quickly. Agents: don't assert a billing failure from an immediate post-run read — re-read after a short delay.

What happens if my "Max cost per run" cap is set too low? Datasource Pulse stops cleanly instead of half-working. A connector is either checked and billed and written to the dataset — or not touched at all; there is no state where you paid for a check that didn't happen. If the cap runs out mid-run, the remaining connectors are skipped for that run (the run log says so explicitly) and picked up on the next run. Size the cap at connectors × $0.02 plus headroom and it will never trigger.

My first run produced no alerts — is it even working? Silence is the product working. The first run establishes each connector's baseline; alerts fire only on state changes. Proof of life is in the Dataset tab — one row per connector, every run. If something is already broken on day one, you'll hear about it within your first schedule interval: exactly one message per problem, and one recovery notice when it's fixed.

---

What you get back

Every run writes one record per connector to the Apify dataset:

{
  "run_at": "2026-07-13T18:00:00.000Z",
  "connector_id": "serpapi-main",
  "connector_label": "SerpAPI — main account",
  "check_type": "credential",
  "state": "healthy",
  "failure_class": null,
  "event_source": "scheduled_probe",
  "message": "SerpAPI healthy — 4,200 searches left (58.0% used).",
  "metadata": {
    "searches_left": 4200,
    "searches_used": 5800,
    "quota_used_pct": 58.0
  }
}

Use the Apify dataset view, export it, or connect it to your own dashboard via the Apify API.

---

Alerting

Every alert payload includes:

• Connector label — the human-readable name you set
• Failure classification — dead_token, quota_exhausted, etc. — tells you what to fix
• Last healthy timestamp — how long the issue has been going
• Recommended action — specific to the failure type
• Recovery notice — one notification when the issue is resolved

Deduplication: the Actor fires exactly one alert per state change. If a connector is still failing on the next run, no second alert is sent until it recovers and fails again.

Escalation: warning → failing fires its own alert, marked [ESCALATED → ...]. Quota creep's endgame — warning at 80%, exhausted at 100% — is never silent. Escalation is a new state change, not a duplicate: still exactly one alert per transition, never one per run.

Webhook format: two body formats, chosen by the optional webhookFormat input field:

• slack (default) — Slack-compatible JSON (attachments array). Works with Slack, Discord (Slack-compatible mode), and any endpoint that accepts POST application/json. Built for humans reading a chat channel.
• json — the raw structured event object, for programmatic receivers that index alerts by field instead of displaying them. Top-level fields, exactly: transition, connector_id, connector_label, failure_class, message, last_healthy_at, occurred_at, recommended_action, impact (present when the check declares one), metadata.

Backward-compatible by construction: with webhookFormat absent, the wire format is exactly what it always was. X-DP-Signature signs the raw body in both formats.

Delivery verification (optional): set webhookExpectJsonOk: true and a 2xx webhook response must carry the JSON body {"ok":true} to count as delivered — guards against catch-all 200 pages (framework fallback routes) acknowledging alerts they never processed.

Webhook signing (optional): set webhookSecret and every webhook POST carries X-DP-Signature: sha256=<HMAC-SHA256 hex of the raw request body>. Verify before trusting an alert:

import { createHmac, timingSafeEqual } from "node:crypto";
const sig = req.headers["x-dp-signature"] ?? "";
const expected = "sha256=" + createHmac("sha256", process.env.DP_WEBHOOK_SECRET).update(rawBody).digest("hex");
const authentic = sig.length === expected.length && timingSafeEqual(Buffer.from(sig), Buffer.from(expected));

Email: there is no SMTP send inside the Actor. emailTo marks the recipient in a structured EMAIL_ALERT log event designed to be relayed — attach an Apify webhook to an email API (Resend, Postmark), or point webhookUrl at Zapier/Make. Native email delivery is on the roadmap; until then, treat email as webhook-relay powered.

---

Scheduling recommendations

• Credential health checks: 2–4×/day is the right default; go hourly only while a connector sits in warning
• Scraper quality checks: once daily (Actor releases move slowly)
• Time-sensitive pipelines (production traffic depends on them): every 30–60 minutes
• Schedule at your peak traffic hours — failures are most likely when your system is under load, not at quiet hours

---

Security guidelines

1. Use scoped / read-only keys everywhere possible. Stripe's restricted read-only key is the mandatory example; apply the same principle to every vendor that supports key scoping.
2. Mode A privacy: Apify stores all Actor input values privately. Only field names (the schema) are public on the Store listing. Your API keys are never visible to other users.
3. Mode B privacy: Your full connector list and credentials live only in the operator's private KV store, keyed by your client token. No configuration is exposed in the Actor input UI.
4. Never share your client token or API keys in run logs, dataset records, or Actor output. Datasource Pulse itself guarantees no credential material ever appears in messages, metadata, alerts, or dataset records — enforced by an automated redaction check in its test suite.
5. Verify webhook authenticity — set webhookSecret so your receiver can check the X-DP-Signature header (HMAC-SHA256 of the raw body) before acting on an alert.

---

Failure classification reference

Classification	Meaning	Fix
dead_token	401/403 — key invalid or revoked	Rotate the API key or re-authorise the OAuth app
quota_exhausted	Usage limit hit	Upgrade plan or wait for quota reset
rate_limited	429 — too many requests	Reduce polling frequency
timeout	Request timed out	Check vendor status page; retry
unexpected_response	2xx but unexpected body	Check vendor API changelog
actor_failed	Actor run status != SUCCEEDED	Review Actor run logs on Apify Console
volume_drop	Output record count below baseline	Check upstream data source access
schema_drift	Output field names changed	Review Actor changelog; update downstream consumers
cost_spike	Per-run cost above baseline	Check for infra issues or large result sets
actor_stale	Actor not updated recently	Check for a maintained fork

---

Related Actors

• apify/monitoring — Apify's own monitoring suite for the run status of Actors in your account. Datasource Pulse is the cross-vendor complement: it watches the API credentials and quotas that live outside Apify, plus the output quality of Actors you depend on but don't own.
• Any Actor your pipeline depends on — point a Probe Type 2 check at it (by actorId) to get schema-drift, volume, staleness, and cost alerts the moment its maintainer ships a breaking change.

Pair Datasource Pulse with your existing scrapers and API integrations to turn a fragile multi-vendor pipeline into one you can actually trust.

Datasource Pulse is from the maker of Demand Discovery AI™ — validate a startup idea with real behavioral demand signals before building it, right inside your IDE via the Demand Discovery MCP.