Dataset Egress Firewall — Allowlist, Leak Report, Provenance

In plain words: you've collected some data, and you're about to hand it to someone. Is there anything inside that shouldn't leave with it — a password, a key, someone's private details buried three layers deep? This tool stands at the door. You name what's allowed out, and only that gets through; everything else is scrubbed, however deep it hides, and you get a plain receipt of exactly what was caught. If the door itself were ever broken, it refuses to open at all rather than let one record slip. That's the whole idea. The rest of this page is the same promise in engineering terms.

Govern what leaves your datasets. Point this Actor at any records — pasted inline or straight from another Actor's dataset — give it an allowlist, and it emits only the projection: unknown fields are dropped, forbidden-named fields (passwords, tokens, PII, debug machinery) are scrubbed at any nesting depth, every clean record carries a tamper-evident provenance tag, and the run produces a leak report you can hand to a client or an auditor.

Deterministic. LLM-free. Fail-closed. The same input always yields the same output, and if the firewall itself is ever broken, the Actor refuses to run rather than emit a single record.

What you can rely on:

• Fail-closed is not a setting — it's the construction. The egress invariants run inside the Docker build itself: a build that could leak fails and never deploys. The firewall then re-proves itself at startup before reading a single record. You are not trusting a config; you're trusting arithmetic that has already refused to ship broken. In plain words: a version of this tool that could leak cannot even be built.
• Your data never leaves. The engine is pure Python standard library — no AI calls, no external requests of any kind. Records are processed entirely in-container.
• Works behind anything (universal middleware). Point it at any Actor's dataset (or inline records) — it chains behind every scraper in the Store, and it's cheap enough to chain everywhere.
• The leak report is a deliverable. Hand it to a client or an auditor as-is: exactly what was caught, where, and what was dropped — receipts for the claim that nothing else left the building.

Why allowlist, not blocklist

A blocklist enumerates what you fear; an allowlist names what you mean to ship. Everything not named simply does not exist in the output — absence, not exposure. The forbidden-substring tripwire runs behind the allowlist as defense-in-depth, catching machinery that hides inside allowed structures (rows[3].api_key, meta.session_token, …).

Two modes

Two ways to stand at the door: clean whatever passes through, or refuse to open the door at all until what's behind it is fixed.

The window (live view)

Watertight is not lightproof. While a run walks, the Actor serves a read-only live view — the leak report as it forms: records seen, records emitted, leaks caught so far, updating every two seconds. Open the run's live-view URL in Console and watch the receipt being written.

It is a window, not a hand: GET-only, no control endpoints, nothing that can touch the run. And it obeys the same law as the door — the page is an allowlist projection of aggregate tallies, so no record content, no forbidden value, and no internal machinery can ever appear in it. That's not a policy; it's a build-gate invariant (a build whose window could leak fails and never deploys).

Input

What you tell it, in plain words: where the records are, what's allowed to leave, which names are forbidden, and how strict to be at the door. The same, as configuration:

{
  "datasetId": "abc123DatasetId",            // clean another actor's output…
  "items": [{ "url": "…", "…": "…" }],       // …and/or inline records
  "allowFields": ["url", "title", "text", "meta"],
  "forbiddenKeySubstrings": ["api_key", "token", "password", "_internal"],  // empty = built-in defaults
  "allowedNestedKeys": [],                   // optional strict mode for nested objects
  "mode": "scrub",                           // or "fail-closed"
  "provenanceSecret": "…",                   // optional — upgrades tags to HMAC (fw1:)
  "maxItems": 10000
}

Output

What comes back: your records wearing only their allowed fields, each stamped with a verifiable seal — one clean record per input record, allowlist-only, plus a provenance tag:

{
  "url": "https://example.com/pricing",
  "title": "Pricing",
  "text": "Start on the free plan…",
  "meta": { "lang": "en" },
  "provenance": "fw1:9f2c41d0a7b3…"
}

And a LEAK_REPORT in the run's key-value store:

{
  "mode": "scrub",
  "failed": false,
  "records_seen": 1240,
  "records_emitted": 1240,
  "records_with_leaks": 37,
  "leaks_total": 41,
  "leak_paths": { "rows[].api_key": 22, "meta.session_token": 15, "user.password": 4 },
  "dropped_fields": { "internal_notes": 1240, "_debug": 96 }
}

Verifying provenance

The seal, in plain words: every clean record carries a stamp that can be re-computed from the record itself — proof it passed this firewall and hasn't been touched since. In engineering terms:

fw0: tags are a SHA-256 over the record's clean fields (sorted-key JSON, provenance excluded) — anyone can recompute them. fw1: tags are an HMAC-SHA-256 with a key derived from your provenanceSecret (PBKDF2-HMAC-SHA256, salt equinox-egress-v1, 200k iterations, 32 bytes) — only holders of the secret can verify, and no tag reveals anything about what was removed.

import hashlib, hmac, json
body = {k: v for k, v in record.items() if k != "provenance"}
payload = json.dumps(body, sort_keys=True, ensure_ascii=False, default=str).encode()
assert record["provenance"] == "fw0:" + hashlib.sha256(payload).hexdigest()[:32]

Great for

• Shipping scraped data to clients — prove the internal fields never left the building.
• Meeting privacy law (GDPR/CCPA egress) — the fields that must never leave, enforced and receipted.
• AI-pipeline hygiene — strip debug machinery, raw traces, and prompt internals from model outputs before they reach consumers.
• Actor chaining — put it between any scraper and its consumers; it's deterministic and cheap, so it costs almost nothing to be safe.

Provable vs. promised — a structural difference, not a slogan

Most tools that clean or redact data work one of three ways, and each asks for a trust this one doesn't:

• Blocklists / pattern rules enumerate what to hide — so anything you didn't think to name walks straight through. An allowlist inverts it: only the fields you named survive, and everything else simply doesn't exist in the output. Absence, not omission.
• ML / heuristic redaction is probabilistic — it usually catches the secret and can't promise it always will. This engine is deterministic and rule-exact: the same input yields the same output, every time.
• Policy you configure is a promise that the config held at runtime. Here, fail-closed isn't a setting — the egress invariants run inside the Docker build, so a version that could leak fails to build and never ships. You're not trusting a policy; you're trusting arithmetic that already refused to compile broken.

None of that calls another tool careless. It's the difference between promised and provable — and it hands you the proof: read the build gate, plant a forbidden field and watch the run refuse, recompute any provenance tag yourself. You don't have to believe the door holds. You can check it.

Pricing

You pay only for what verifiably happened — every charge has a receipt, nothing else is ever billed. Pay-per-event, three meters that add together on every run:

The single "from $0.10 / 1,000" figure Apify shows in the header is just the cheapest of these three meters — not the price of a run. Your total is

run-started + (records × record-cleaned) + (leaks × leak-detected)

What a real run costs

Concrete, in scrub mode (the common case):

• 1,000 records, spotless — 1 run + 1,000 records + 0 leaks ≈ $0.10
• 10,000 records, spotless — 1 run + 10,000 records + 0 leaks ≈ $1.00
• 10,000 records, 300 leaks caught — that same $1.00, plus the 300 catches ≈ $1.15

record-cleaned is the predictable meter; leak-detected is a small add-on that fires only on a genuine catch, so a clean dataset never pays for leaks it doesn't have. (In fail-closed mode a run that trips aborts and emits nothing — you pay for the startup and the catches that stopped it, not for records never shipped.)

How this price was derived

Priced by derivation, not by market — no reference to anyone else's prices. The formula: measured cost ÷ platform share + a stated stewardship wage. From a real measured run (2026-07-22): startup ≈ $0.0017/run — so run-started is set at exactly that — and the marginal per-record cost is microscopic (pure stdlib), so record-cleaned sits at $0.0001/record. The launch is priced once as itself so small runs and huge runs each pay the true shape of their own cost — no cross-subsidy in either direction. Every charged event corresponds to a verifiable occurrence with a receipt (the record itself, or its line in the LEAK_REPORT); nothing is ever charged for what did not happen. The margin above cost is a stewardship wage, not what-the-market-bears; when the meters change, the price is re-derived and this section updated.

Servicing of terms

We flipped the label on purpose: not terms that govern the service — a service that keeps its terms. Here they are, short enough to actually read: your data stays yours (processed in-container, never retained after the run, never sold, never trained on, never sent to any external service — this engine makes no outbound calls at all); no rights are claimed over your inputs or outputs; you pay only for receipted events; you can leave anytime with nothing held. We ask one term back: don't use the firewall as a laundry for taking — it exists to help you ship data cleanly, to people who deserve clean data.

Notes

• The firewall governs by field names, so a record with no named fields — a bare string, number, or list where a {...} object should be — gives the allowlist nothing to hold. Such records are never passed through raw: they are dropped and counted in the report (non_dict_records_skipped). What can't be governed doesn't get emitted.
• Matching is case-insensitive substring, over-broad on purpose: a tripwire should catch API_Key, apiKeyOld, and legacy_api_key_backup alike. Tune the vocabulary to your domain.
• The firewall runs a self-check before reading a single record and the build itself fails if any egress invariant breaks — fail-closed is not a setting here; it's the construction.