Silverset | SQL, REST & Sheets to Analysis-Ready Data

What does Silverset do?

Silverset reads your tables, APIs, sheets, files, or pasted JSON, keeps every raw row in Bronze, runs named stages, and writes Silver rows that passed. A stage reject goes to quarantine. Missing-drop and dedup remove rows without a quarantine item (counted as dropped on MANIFEST). Fail fast writes every row the failing batch rejected to quarantine, then aborts (rows written in earlier batches stay written; any trailing flush is batch-local). Bronze is always raw, including when pii redacts Silver and quarantine.

It is not a scraper and not a warehouse. You point it at data you already have. Delivery is at-least-once with idempotency, not exactly-once: a retried run can append a second copy of the same row. Silver lands on an Apify Dataset (or key-value store).

Why use Silverset?

• Nine sources, one pipeline. SQL, REST, Google Sheets, CSV/JSON files, another Actor's dataset, a key-value record, or paste.
• Raw is never thrown away. Bronze is the replay source. Silver is what you analyse. Quarantine is rejects; missing-drop and dedup are counted as dropped on MANIFEST.
• Start without writing a pipeline. The Console default is Analysis-ready: profile the data, change no values, do not coerce CSV strings to numbers.
• Schedule it, call it from n8n or the API, export CSV/Excel — same Actor, same stores.

See the Input tab for every field. This page is the walkthrough.

How to run Silverset

1. Open the Actor and leave Pipeline preset on Analysis-ready (under Advanced).
2. Pick a Recipe.
3. Fill Where — the field's help text says what to type for that recipe (for PostgreSQL/MySQL that is user@host/database, not a driver URL).
4. If the recipe needs a password or token, put it in Secret. Leave Secret empty for files, datasets, paste, and SQLite.
5. Click Start. Wait until the run succeeds.
6. Open the default dataset — those are your Silver rows. The run report is key-value store SUMMARY, not a dataset trailer.
7. Open the default key-value store → PROFILE for the schema/profile reports, MANIFEST for counts (including dropped) and charges, SUMMARY for the run report.

A rejected input (exit 2) is not billed. Tick Dry run (Advanced) to check config and list tables with no writes and no charges.

The Console form is recipe + Where + Secret. n8n and the API keep sending source_type + source_config as before — that path ignores Recipe. Everything below is a complete API/n8n payload.

Notes before your first run

• Console: Recipe selects the connector; Where tells it where to look. You do not paste JSON for a first run.
• API / n8n: only source_config changes between sources. source_type selects the connector; the object tells that one connector where to look. Keys from another source are rejected, not ignored — file_url will not accept tables.
• Credentials never go in JSON. Secret (Console) and the hidden credential fields (API) are marked secret, so Apify encrypts them. The same value typed into source_config is stored in plain text in the run's INPUT record.
• Destination defaults to this run's Dataset. Open Advanced to write Silver to a key-value store instead.
• Leave the rest of Advanced alone until a run tells you otherwise. Its defaults are the documented behaviour.

Sources

On the Console, pick a Recipe and fill Where:

On n8n and the API, send source_type + source_config as below. destination_type still works; if you omit it the Actor writes to this run's Dataset.

A complete run, for each source

Every block below is a whole INPUT for the API or n8n. On the Console, pick the matching Recipe and fill Where instead.

SQL — one table, incremental. Put the URL in sql_connection_string (encrypted), not in source_config.

{
    "source_type": "sql",
    "source_config": {
        "tables": ["customers"],
        "cursor_fields": { "customers": "updated_at" }
    },
    "destination_type": "apify_dataset",
    "streams": [
        {
            "name": "customers",
            "sync_mode": "incremental",
            "cursor_field": "updated_at"
        }
    ]
}

A CSV or JSON file over HTTP.

{
    "source_type": "file_url",
    "source_config": { "url": "https://example.com/data.csv" },
    "destination_type": "apify_dataset"
}

A REST endpoint with paging. records_path is where the array sits in the response body; next_url_path is where the next-page link sits in that same body. cursor_field is a separate thing — incremental sync between runs, not paging within one.

{
    "source_type": "rest_api",
    "source_config": {
        "url": "https://api.example.com/v1/items",
        "records_path": "data",
        "pagination": "next_url",
        "next_url_path": "next",
        "cursor_field": "updated_at"
    },
    "destination_type": "apify_dataset"
}

Google Sheets. sheets is required; send the OAuth token as google_sheets_access_token (or an API key as google_sheets_api_key), never both.

{
    "source_type": "google_sheets",
    "source_config": { "spreadsheet_id": "1AbC...", "sheets": ["Sheet1"] },
    "destination_type": "apify_dataset"
}

Paste records straight in — the fastest way to see what the Actor does to your data.

{
    "source_type": "inline",
    "source_config": { "records": [{ "id": 1, "email": "A@Example.COM " }] },
    "destination_type": "apify_dataset"
}

Clean another Actor's output, then write it to a named dataset of your own.

{
    "source_type": "dataset",
    "source_config": { "datasetId": "aBcD1234" },
    "destination_type": "apify_dataset",
    "destination_config": { "dataset_name": "my-silver" }
}

SQL — use a read-only database user. Paste a SQLAlchemy async URL into the connection-string box: postgresql+asyncpg://USER:PASSWORD@HOST:5432/DB, mysql+aiomysql://USER:PASSWORD@HOST:3306/DB, or sqlite+aiosqlite:///data.db for a file. Optional source_config keys: schema_name, tables, cursor_fields, query. Leave Streams empty to sync every discovered table in full_refresh. To limit tables, set Streams to [{"name": "customers"}]. An empty list means zero tables, not all of them.

REST — pages are read in order. The cursor is committed only at the end of the stream, so an interrupted run re-reads rather than skipping.

Paging. pagination defaults to none — one request, one page. page fits APIs that take a page-number parameter (?page=1, ?page=2, ...); next_url fits APIs that return a link to the next page in the response body. Offset-style APIs ($offset, skip, offset) aren't supported yet — page steps a raw parameter by 1, not by page size, so a second page overlaps and duplicates most of the first instead of advancing to new rows. That makes an offset-style API safe to read for exactly one page, not a slow-but-correct multi-page one — paging past the first page returns duplicates, not new data. To page, pass source_config alongside the recipe; your keys win over the recipe's:

{"recipe": "rest_api", "locator": "https://api.example.com/items", "source_config": {"pagination": "page", "page_param": "page", "max_pages": 50}}

Google Sheets — sheets is required. Credential is exactly one of access_token or api_key, with read scope on that sheet only.

File / JSON / JSONL / KVS — format is detected from the extension, then Content-Type; set "format" when neither is reliable. Add "repair": true to let the Actor fix malformed JSON (trailing commas, single quotes) instead of failing. Repair is off by default: it guesses, and a quiet wrong guess is worse than a stopped run.

Incremental vs full refresh. Only sql and rest_api can resume. The other seven re-read everything every run — Reset cursor does nothing for them. On an incremental stream, a second run that reads 0 rows usually means the cursor is current; tick Reset cursor to re-read from scratch.

Destinations

destination_type is apify_dataset (default) or apify_kv. Rows cost the same either way.

Apify Dataset is what the Console previews and what Export downloads (JSON, CSV, Excel, HTML). Leave Destination configuration empty unless you want a named Silver dataset — {"dataset_name": "my-silver"} — or a different stream-name field, {"stream_field": "_source"} (default _stream).

Key-value store — set destination_type to apify_kv, optionally with {"key_prefix": "run1-"}. It writes one JSON key per batch: {key_prefix}{stream}-{batch:06d}. A re-run overwrites those keys; it does not delete leftover higher keys. If the first run wrote 20 rows (several keys) and the next wrote 4, the store still holds the old higher keys. Use a fresh key_prefix when the row count can shrink, or stay on Dataset.

Output — what lands where

You can download the Silver dataset as JSON, CSV, Excel, or HTML from the run's dataset tab.

Bronze and quarantine are named datasets: they persist and accumulate across runs. Filter by run_id to see one run. KV records (MANIFEST, SUMMARY, PROFILE) are overwritten each run.

A named Silver dataset (destination_config.dataset_name) is the same kind of shared bucket: it accumulates across runs. Full refresh does not empty it. Silver rows have no run_id. Leave destination unnamed to get this run's fresh default dataset. Filter Bronze and quarantine by run_id to see one run.

Bronze is not redacted — it is the replay source, including when pii redacts Silver and quarantine. Silver carries your fields plus _stream. The engine record_id used for replay sits on Bronze and quarantine items, not on Silver. After a fan-out (normalize with record_path), each child also carries _record_id ({parent}#{index}) so dedup can keep siblings distinct; payload id is shared, so key dedup on _record_id for one Silver row per child. A second fan-out extends rather than restarts the id ({parent}#{index}#{index}), so leaves of a chained fan-out stay distinct. Non-fan-out Silver omits _record_id.

PROFILE is one record, nested by stream then report. A run with schema, profile, and dedup stores PROFILE["<stream>"]["SCHEMA_REPORT"], ["PROFILE_REPORT"], and ["DEDUP_REPORT"].

Example: before and after

Input row, as read:

{
    "id": "1",
    "e_mail": "ADA@Example.com ",
    "signup": "2026-01-02",
    "notes": null
}

Bronze stores that row under data. Query Bronze for data.email, not email.

Custom pipeline (only when preset is Custom):

[
    {
        "stage": "map",
        "config": {
            "field_map": [["id"], ["email", "e_mail"], ["signup"], ["notes"]]
        }
    },
    { "stage": "types", "config": { "types": { "id": "int" } } },
    {
        "stage": "pii",
        "config": {
            "fields": ["email"],
            "strategy": "hash",
            "salt": "example-salt"
        }
    }
]

Silver row:

{
    "id": 1,
    "email": "501821b0a30aeede2d2b66d8c2949c137a023cd1c70e7bab8a6b656aa416d760",
    "signup": "2026-01-02",
    "notes": null,
    "_stream": "people"
}

map renamed e_mail → email, types turned "1" into 1, pii hashed the address, the destination added _stream. pii hashes the value as it arrived — capitals and the trailing space are in the digest. Normalise before you redact if you will join on the hash.

How much does it cost?

Pay per event. You are not billed for Bronze rows.

Examples: 100 Analysis-ready rows (start + Silver + one PROFILE) → $0.42. 10,000 Silver rows, no PROFILE → $4.30.

What is free. A run rejected for bad input is not charged. Dry run is free of every charge.

What is billed twice. Rows are billed as they are written. If a batch is retried after a network failure, those rows are billed again — they appear again in the dataset (at-least-once). MANIFEST.charges records what this run billed and anything it failed to bill.

Set these same event names and prices in Apify Console → Monetization. Do not also enable apify-actor-start or apify-default-dataset-item, or buyers are double-charged.

Pipeline presets

Analysis-ready (pre_analysis) is the Console default. It runs pii → schema → validate → profile.

It profiles; it does not coerce CSV strings to numbers. It changes no values. It drops no records, with one exception: profile cannot key a value nested past roughly 3,000 levels, and that record goes to quarantine (STAGE_UNCAUGHT_ERROR) — quarantined, never silently lost; the rest of the run continues.

Under this preset, pii and validate get empty config, so nothing is redacted and nothing is rejected. You get a SCHEMA_REPORT over up to sample_size records (1000 by default) and a PROFILE_REPORT over every record, with the redaction and validation slots already in a legal order.

To map columns, convert types, dedup, or redact: set the preset to Custom and fill Transform stages. Setting a preset and a pipeline is rejected (PIPELINE_PRESET_CONFLICT).

Neither schema nor profile ever modifies a record. Profiling is evidence, not permission to silently coerce types.

Custom pipeline

Only when Pipeline preset is Custom. Ordered list of {"stage", "config"}. Empty list is a passthrough.

types targeting date/datetime becomes a Python date/datetime; the Dataset stores the string form. A date is "2026-01-02"; a datetime is "2026-01-02 09:00:00+00:00" (a space, not T). Convert to str if you need strict ISO-8601.

format: is enforced (email, date, date-time, uri, uuid, ipv4, ipv6, and the other checkers this build ships). A schema naming a format this build cannot check is rejected at startup rather than silently ignored — before this change every format: was annotation-only, so a schema that looked like it validated emails did not.

Input

See the Input tab for the full form. The fields that surprise people:

• Recipe / Where / Secret — the Store form. Recipe names the connector; Where is the locator; Secret is the password or token when that recipe needs one.
• source_type / source_config — what n8n and the API send. When source_type is set, Recipe and Where are ignored. source_config ships no prefill: a prefilled example would be right for one source and rejected by the other eight.
• Secret vs JSON — a password typed into source_config is stored in the run's INPUT record in the clear. Secret is encrypted.
• Streams — omit to sync every table. [] means zero tables.
• Sync key — optional cursor namespace. Set it to keep incremental cursors stable across config edits.
• Quality policy — Fail fast writes every row the failing batch rejected to quarantine, then aborts (earlier batches stay written; trailing flush is batch-local). Quarantine continues and marks MANIFEST partial. Warn quarantines and counts those rows on MANIFEST.counts.quarantined, but MANIFEST status stays succeeded. Missing-drop and dedup are dropped, not quarantine.
• Fail on partial — Exit non-zero when quarantined records made the run partial. Default off: Apify can show SUCCEEDED while MANIFEST.status is partial. Turn on if a scheduler should fail on rejects. Fail fast already aborts. Warn never counts as partial.
• On operational failure — legacy; governs destination/stream failures, not bad rows. Quality policy is the control for bad records.

A local INPUT.json is not the Console. The Console defaults pipeline_preset to pre_analysis. A hand-written file that omits the field falls back to custom with an empty pipeline (passthrough, no reports). Set it explicitly when you run locally.

FAQ

Why are my rows in quarantine, not Silver?

A stage rejected them. Open datapipeline-quarantine and read error_code and stage.

Why did the run exit 2 immediately?

Input was rejected before any read. The status message names the field. Common cases: PIPELINE_PRESET_CONFLICT (preset and custom pipeline both set — set preset to custom or clear the pipeline); PII_STAGE_ORDER_INVALID (pii after schema/profile — move pii first); unknown stage name (the log lists registered stages).

Why PII_DEDUP_ORDER_INVALID?

A pii with strategy: mask or remove runs before a dedup that keys on a field it redacts. Move pii after dedup, switch to strategy: hash, or key dedup on a field pii does not touch. The check follows the value through rename / map / derive / missing (flag mode), and out of scope through drop or a map that projects it away. A normalize with record_path between the two cannot be traced, so that order is rejected.

Why SYNC_ABORTED?

A stage aborted the stream. Fail fast writes every row that batch rejected to quarantine first; rows from earlier batches stay written, and any trailing flush is batch-local. The status only says SYNC_ABORTED; the stage's code is in streams[].error_codes in MANIFEST and in the log. For DEDUP_KEY_LIMIT_EXCEEDED: raise max_keys, narrow keys, or set on_limit: quarantine.

Why SOURCE_CONFIG_INVALID?

The connection string could not be parsed. This is found at connect time, not at input validation.

Will a retried run duplicate Silver rows?

Yes, on Dataset: it is append-only. Deduplicate downstream on your own key, or join Silver back to Bronze on record_id. Silver itself has no engine record_id (fan-out children carry _record_id so dedup can key on it).

Does this Actor scrape websites?

No. It reads sources you configure (databases, APIs, sheets, files, Apify storage, pasted JSON). You are responsible for having a lawful basis to process that data. Bronze stores unredacted raw rows; use pii on a Custom pipeline if Silver must not contain personal data. Personal data is protected by the GDPR and similar laws — do not process it unless you have a legitimate reason.

Where do I report a bug?

Use this Actor's Issues tab. For programmatic runs, see the API tab.

Automate it

• n8n — two importable workflows, core nodes only: docs/n8n/README.md
• Agents — operation patterns over the Actor and storage APIs: docs/agents/ops-patterns.md