Apify Dataset Replicator – PostgreSQL, Supabase & S3

What does Apify Dataset Replicator do?

Apify Dataset Replicator automatically syncs the results of your Apify Actors to PostgreSQL or S3-compatible storage — continuously, incrementally, and without ever starting from scratch. Point it at an Actor (or a specific dataset) on the Apify platform, give it a connection string, and every new successful run lands in your database as clean rows, ready for your application, BI dashboard, or AI pipeline.

Unlike one-time exporters, the Replicator remembers what it already copied: it discovers new runs, skips runs it has processed, resumes interrupted transfers from a checkpoint, and upserts records so re-scraped data updates existing rows instead of duplicating them. Combined with Apify's scheduling, API access, and monitoring, it turns any scraper into a live data feed for your systems.

Why use Apify Dataset Replicator?

• Keep a database in sync with a scraper — schedule a scraper hourly and have fresh rows in Supabase, Neon, or RDS minutes later, with no glue code.
• Incremental & resumable — only new items are replicated; a crashed or aborted run resumes from the last committed batch. Datasets with millions of items are handled in bulk batches.
• Idempotent UPSERT — define a primary key (url, placeId, asin…) and re-scraped records update in place. At-least-once delivery + upsert = effectively exactly-once for your table.
• Schema-change proof — the default JSON mode stores items in a JSONB column and never breaks; the infer mode creates typed SQL columns and can add new columns automatically as the scraper evolves.
• Cheap archival — the S3 destination writes compressed JSONL part files plus a manifest, giving you a permanent, queryable backup of every run on S3, Cloudflare R2, Backblaze B2, or MinIO.

Supported destinations

Destination	Works with
PostgreSQL	Supabase, Neon, Railway, Render, AWS RDS, Google Cloud SQL, Azure Database, self-hosted
S3-compatible storage	AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO, and others

How to use Apify Dataset Replicator

1. Open the Actor and go to the Input tab.
2. Pick a source: an Actor whose new successful runs should be replicated (recommended), its last successful run, or a specific dataset ID.
3. Pick a destination: paste a PostgreSQL connection string and table name, or an S3 bucket with credentials. Secrets are stored encrypted.
4. Choose the write mode (Upsert is recommended) and the primary key field that uniquely identifies a record.
5. Click Start. The Replicator creates the table if needed, copies everything in batches, and saves its position.
6. Create a Task from this configuration and add an Apify Schedule (e.g. hourly). Each scheduled run replicates only new data — already-processed runs are skipped automatically.

Tip: run the Replicator right after your scraper finishes by scheduling both, or trigger it from the scraper's webhook for near-real-time sync.

Input

The most important fields (see the Input tab for the full list):

{
    "sourceType": "newRuns",
    "actorId": "apify/google-maps-scraper",
    "destinationType": "postgres",
    "connectionString": "postgresql://user:password@host:5432/db",
    "tableName": "google_places",
    "writeMode": "upsert",
    "primaryKey": ["placeId"],
    "schemaMode": "json",
    "batchSize": 1000,
    "includeMetadata": true,
    "rememberCheckpoint": true
}

• Source — newRuns (every new successful run, exactly once), lastRun (most recent run), or dataset (a specific, possibly growing dataset).
• Write mode — upsert (insert or update by primary key), append, ignoreDuplicates, or replace (truncate and reload).
• Schema mode — json stores each item in a JSONB data column with lineage columns (safest); infer creates typed SQL columns from your data, with automatic ALTER TABLE … ADD COLUMN when new fields appear (schemaEvolution: "add-columns").
• Nested objects — keep as JSONB, flatten (seller.name → seller_name column), or stringify.
• Transformation — replicate only selected fields (pickFields) and/or rename them (fieldMapping, e.g. {"video_url": "url", "channel": "channel.name"}).

Output

PostgreSQL (JSON mode — default)

CREATE TABLE google_places (
    id BIGSERIAL PRIMARY KEY,
    item_key TEXT,               -- primary-key value used for upsert
    apify_actor_id TEXT,
    apify_run_id TEXT,
    apify_dataset_id TEXT,
    apify_item_offset BIGINT,
    data JSONB NOT NULL,         -- the full item
    replicated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

PostgreSQL (infer mode)

Item fields become typed columns (TEXT, BIGINT, DOUBLE PRECISION, BOOLEAN, JSONB). With includeMetadata on, every record also carries an _apify JSONB column:

{
    "title": "MacBook Pro",
    "price": 1999,
    "_apify": {
        "actorId": "apify/amazon-scraper",
        "runId": "xyz",
        "datasetId": "123",
        "itemOffset": 4301,
        "replicatedAt": "2026-08-23T04:10:00Z"
    }
}

S3-compatible storage

my-bucket/
  apify/
    apify-google-maps-scraper/
      2026-08-23/
        run-ABC123/
          manifest.json
          part-000001.jsonl.gz
          part-000002.jsonl.gz

The Replicator's own dataset contains one summary row per replicated dataset (run ID, item counts, destination, timing) — you can download it in JSON, HTML, CSV, or Excel formats.

Data the Replicator tracks for you

Field	Description
itemsReplicated	Items copied in this run per source dataset
resumedFromOffset	Checkpoint offset the run resumed from
runId / datasetId / actorId	Lineage of the replicated data
destination	Target table or bucket
startedAt / finishedAt	Replication timing

How much does it cost to replicate Apify datasets?

Pricing is pay-per-event: a small flat fee per started run plus a per-record fee for the data actually replicated — so cost scales with the amount of data moved, not with how often you run it. Because only new items are replicated on each scheduled run, a typical hourly sync that finds nothing new costs just the start fee and finishes in seconds. Replication is batched (bulk multi-row inserts, gzipped S3 parts), so even datasets with hundreds of thousands of items replicate quickly.

You stay in control of spend: set a maximum cost per run and the Replicator stops cleanly when it is reached — everything already replicated is checkpointed, and the next run resumes the remainder without paying for the same records twice. Max items per run offers the same cap in item terms.

Tips and advanced options

• Big initial loads: raise batchSize (up to 50,000) for the first full sync, then leave the schedule to handle small increments.
• Memory vs. batch size: one batch lives in memory a few times over while being transformed and written, so scale run memory with batchSize — 512–1024 MB covers the default 1,000-item batches, 2048 MB suits batches of 5,000–10,000, and 4096 MB is plenty for the maximum. If a run hits an out-of-memory error, lower batchSize or raise the run's memory.
• Aborting is safe: the checkpoint is committed after every batch, so aborting a run loses nothing — the next run resumes where it stopped.
• Type drift: if a scraper changes a field's type over time, prefer schemaMode: "json"; infer mode fails with an actionable error rather than silently corrupting data.
• Primary keys refer to transformed fields — the field names after pick/mapping/flatten are what upsert matches on (e.g. after flattening, use seller_name, not seller.name).
• Concurrency protection: a job lock prevents two Replicator runs from writing the same table simultaneously; stale locks from crashed runs expire after 30 minutes.
• Local development: apify run works locally, but local storage/ is not synced to the Apify Console — deploy with apify push to verify cloud behavior.

FAQ, disclaimers, and support

What are the delivery guarantees? At-least-once. A crash between a database commit and the checkpoint save may replay one batch; with upsert or ignoreDuplicates this is invisible in the destination.

Does it store my credentials safely? Connection strings and S3 keys are input schema secrets — encrypted by the Apify platform and censored from logs.

Does it modify my source data? No. The Replicator only reads datasets via the Apify API; it never alters your Actors or their storages.

Legal note: this Actor moves data you already collected on the Apify platform. You are responsible for ensuring that storing and processing that data complies with the target website's terms and applicable law (including personal-data regulations).

Found a bug or need another destination (BigQuery, Snowflake, MongoDB, webhooks…)? Open an issue on the Actor's Issues tab — feedback drives the roadmap, and custom replication solutions are available on request.