# Apify Dataset Replicator (`dtrungtin/apify-dataset-replicator`) Actor

Sync the results of your Apify Actors to PostgreSQL or S3-compatible storage - continuously, incrementally, and without ever starting from scratch. Give it a connection string and every new successful run lands in your database as clean rows, ready for your application, BI dashboard, or AI pipeline.

- **URL**: https://apify.com/dtrungtin/apify-dataset-replicator.md
- **Developed by:** [Tin](https://apify.com/dtrungtin) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 replicated records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Apify Dataset Replicator – PostgreSQL, Supabase & S3

### What does Apify Dataset Replicator do?

**Apify Dataset Replicator** automatically **syncs the results of your Apify Actors to PostgreSQL or S3-compatible storage** — continuously, incrementally, and without ever starting from scratch. Point it at an Actor (or a specific dataset) on the [Apify platform](https://apify.com), give it a connection string, and every new successful run lands in your database as clean rows, ready for your application, BI dashboard, or AI pipeline.

Unlike one-time exporters, the Replicator **remembers what it already copied**: it discovers new runs, skips runs it has processed, resumes interrupted transfers from a checkpoint, and **upserts** records so re-scraped data updates existing rows instead of duplicating them. Combined with Apify's scheduling, API access, and monitoring, it turns any scraper into a live data feed for your systems.

### Why use Apify Dataset Replicator?

- **Keep a database in sync with a scraper** — schedule a scraper hourly and have fresh rows in Supabase, Neon, or RDS minutes later, with no glue code.
- **Incremental & resumable** — only new items are replicated; a crashed or aborted run resumes from the last committed batch. Datasets with millions of items are handled in bulk batches.
- **Idempotent UPSERT** — define a primary key (`url`, `placeId`, `asin`…) and re-scraped records update in place. At-least-once delivery + upsert = effectively exactly-once for your table.
- **Schema-change proof** — the default JSON mode stores items in a JSONB column and never breaks; the infer mode creates typed SQL columns and can add new columns automatically as the scraper evolves.
- **Cheap archival** — the S3 destination writes compressed JSONL part files plus a manifest, giving you a permanent, queryable backup of every run on S3, Cloudflare R2, Backblaze B2, or MinIO.

### Supported destinations

| Destination               | Works with                                                                              |
| ------------------------- | --------------------------------------------------------------------------------------- |
| **PostgreSQL**            | Supabase, Neon, Railway, Render, AWS RDS, Google Cloud SQL, Azure Database, self-hosted |
| **S3-compatible storage** | AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO, and others                          |

### How to use Apify Dataset Replicator

1. Open the Actor and go to the **Input** tab.
2. Pick a **source**: an Actor whose *new successful runs* should be replicated (recommended), its *last successful run*, or a *specific dataset ID*.
3. Pick a **destination**: paste a PostgreSQL connection string and table name, or an S3 bucket with credentials. Secrets are stored encrypted.
4. Choose the **write mode** (Upsert is recommended) and the **primary key** field that uniquely identifies a record.
5. Click **Start**. The Replicator creates the table if needed, copies everything in batches, and saves its position.
6. Create a **Task** from this configuration and add an [Apify Schedule](https://docs.apify.com/platform/schedules) (e.g. hourly). Each scheduled run replicates only new data — already-processed runs are skipped automatically.

> Tip: run the Replicator right after your scraper finishes by scheduling both, or trigger it from the scraper's webhook for near-real-time sync.

### Input

The most important fields (see the Input tab for the full list):

```json
{
    "sourceType": "newRuns",
    "actorId": "apify/google-maps-scraper",
    "destinationType": "postgres",
    "connectionString": "postgresql://user:password@host:5432/db",
    "tableName": "google_places",
    "writeMode": "upsert",
    "primaryKey": ["placeId"],
    "schemaMode": "json",
    "batchSize": 1000,
    "includeMetadata": true,
    "rememberCheckpoint": true
}
```

- **Source** — `newRuns` (every new successful run, exactly once), `lastRun` (most recent run), or `dataset` (a specific, possibly growing dataset).
- **Write mode** — `upsert` (insert or update by primary key), `append`, `ignoreDuplicates`, or `replace` (truncate and reload).
- **Schema mode** — `json` stores each item in a JSONB `data` column with lineage columns (safest); `infer` creates typed SQL columns from your data, with automatic `ALTER TABLE … ADD COLUMN` when new fields appear (`schemaEvolution: "add-columns"`).
- **Nested objects** — keep as JSONB, flatten (`seller.name` → `seller_name` column), or stringify.
- **Transformation** — replicate only selected fields (`pickFields`) and/or rename them (`fieldMapping`, e.g. `{"video_url": "url", "channel": "channel.name"}`).

### Output

#### PostgreSQL (JSON mode — default)

```sql
CREATE TABLE google_places (
    id BIGSERIAL PRIMARY KEY,
    item_key TEXT,               -- primary-key value used for upsert
    apify_actor_id TEXT,
    apify_run_id TEXT,
    apify_dataset_id TEXT,
    apify_item_offset BIGINT,
    data JSONB NOT NULL,         -- the full item
    replicated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```

#### PostgreSQL (infer mode)

Item fields become typed columns (`TEXT`, `BIGINT`, `DOUBLE PRECISION`, `BOOLEAN`, `JSONB`). With `includeMetadata` on, every record also carries an `_apify` JSONB column:

```json
{
    "title": "MacBook Pro",
    "price": 1999,
    "_apify": {
        "actorId": "apify/amazon-scraper",
        "runId": "xyz",
        "datasetId": "123",
        "itemOffset": 4301,
        "replicatedAt": "2026-08-23T04:10:00Z"
    }
}
```

#### S3-compatible storage

```text
my-bucket/
  apify/
    apify-google-maps-scraper/
      2026-08-23/
        run-ABC123/
          manifest.json
          part-000001.jsonl.gz
          part-000002.jsonl.gz
```

The Replicator's own dataset contains one summary row per replicated dataset (run ID, item counts, destination, timing) — you can download it in JSON, HTML, CSV, or Excel formats.

### Data the Replicator tracks for you

| Field                             | Description                                 |
| --------------------------------- | ------------------------------------------- |
| `itemsReplicated`                 | Items copied in this run per source dataset |
| `resumedFromOffset`               | Checkpoint offset the run resumed from      |
| `runId` / `datasetId` / `actorId` | Lineage of the replicated data              |
| `destination`                     | Target table or bucket                      |
| `startedAt` / `finishedAt`        | Replication timing                          |

### How much does it cost to replicate Apify datasets?

Pricing is **pay-per-event**: a small flat fee per started run plus a per-record fee for the data actually replicated — so cost scales with the amount of data moved, not with how often you run it. Because only **new** items are replicated on each scheduled run, a typical hourly sync that finds nothing new costs just the start fee and finishes in seconds. Replication is batched (bulk multi-row inserts, gzipped S3 parts), so even datasets with hundreds of thousands of items replicate quickly.

You stay in control of spend: set a **maximum cost per run** and the Replicator stops cleanly when it is reached — everything already replicated is checkpointed, and the next run resumes the remainder without paying for the same records twice. **Max items per run** offers the same cap in item terms.

### Tips and advanced options

- **Big initial loads**: raise `batchSize` (up to 50,000) for the first full sync, then leave the schedule to handle small increments.
- **Memory vs. batch size**: one batch lives in memory a few times over while being transformed and written, so scale run memory with `batchSize` — 512–1024 MB covers the default 1,000-item batches, 2048 MB suits batches of 5,000–10,000, and 4096 MB is plenty for the maximum. If a run hits an out-of-memory error, lower `batchSize` or raise the run's memory.
- **Aborting is safe**: the checkpoint is committed after every batch, so aborting a run loses nothing — the next run resumes where it stopped.
- **Type drift**: if a scraper changes a field's type over time, prefer `schemaMode: "json"`; infer mode fails with an actionable error rather than silently corrupting data.
- **Primary keys refer to transformed fields** — the field names after pick/mapping/flatten are what upsert matches on (e.g. after flattening, use `seller_name`, not `seller.name`).
- **Concurrency protection**: a job lock prevents two Replicator runs from writing the same table simultaneously; stale locks from crashed runs expire after 30 minutes.
- **Local development**: `apify run` works locally, but local `storage/` is not synced to the Apify Console — deploy with `apify push` to verify cloud behavior.

### FAQ, disclaimers, and support

**What are the delivery guarantees?** At-least-once. A crash between a database commit and the checkpoint save may replay one batch; with `upsert` or `ignoreDuplicates` this is invisible in the destination.

**Does it store my credentials safely?** Connection strings and S3 keys are input schema secrets — encrypted by the Apify platform and censored from logs.

**Does it modify my source data?** No. The Replicator only reads datasets via the Apify API; it never alters your Actors or their storages.

**Legal note**: this Actor moves data you already collected on the Apify platform. You are responsible for ensuring that storing and processing that data complies with the target website's terms and applicable law (including personal-data regulations).

Found a bug or need another destination (BigQuery, Snowflake, MongoDB, webhooks…)? Open an issue on the Actor's **Issues** tab — feedback drives the roadmap, and custom replication solutions are available on request.

# Actor input Schema

## `sourceType` (type: `string`):

Where the data comes from. <b>New successful runs</b> watches an Actor and replicates every successful run exactly once. <b>Last successful run</b> replicates only the most recent one. <b>Specific dataset</b> replicates a single (possibly growing) dataset incrementally.

## `actorId` (type: `string`):

The Actor whose run results to replicate, e.g. <code>apify/google-maps-scraper</code>. Required for the Actor source types.

## `datasetId` (type: `string`):

ID or name of the dataset to replicate. Required when the source type is <b>Specific dataset</b>.

## `destinationType` (type: `string`):

Where to replicate the data. PostgreSQL covers Supabase, Neon, Railway, Render, RDS, Cloud SQL and self-hosted servers. S3 covers AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO and other S3-compatible storage.

## `connectionString` (type: `string`):

Connection string in the form <code>postgresql://user:password@host:5432/database</code>. Stored encrypted.

## `tableSchema` (type: `string`):

Database schema of the destination table.

## `tableName` (type: `string`):

The table to replicate into, e.g. <code>google\_places</code>.

## `createTable` (type: `boolean`):

Create the destination table (and the unique index required for upsert) if it does not exist yet.

## `s3Bucket` (type: `string`):

Name of the destination bucket.

## `s3Prefix` (type: `string`):

Folder prefix for replicated files, e.g. <code>apify</code>. Files are written as <code>{prefix}/{actor}/{date}/run-{runId}/part-000001.jsonl.gz</code> plus a <code>manifest.json</code>.

## `s3Region` (type: `string`):

Region of the bucket (or any value accepted by the provider, e.g. <code>auto</code> for Cloudflare R2).

## `s3Endpoint` (type: `string`):

Custom endpoint for S3-compatible providers, e.g. <code>https://\<account>.r2.cloudflarestorage.com</code> or a MinIO URL. Leave empty for AWS S3.

## `s3AccessKeyId` (type: `string`):

Access key for the bucket. Stored encrypted.

## `s3SecretAccessKey` (type: `string`):

Secret key for the bucket. Stored encrypted.

## `s3ForcePathStyle` (type: `boolean`):

Required by MinIO and some other S3-compatible services. Defaults to on when a custom endpoint is set.

## `writeMode` (type: `string`):

<b>Upsert</b> inserts or updates by primary key (recommended — makes replication idempotent). <b>Append</b> always inserts. <b>Ignore duplicates</b> inserts only unseen keys. <b>Replace</b> truncates the table and reloads everything on every run.

## `primaryKey` (type: `array`):

Field(s) that uniquely identify a record, e.g. <code>url</code>, <code>placeId</code>, <code>asin</code>. Required for upsert and ignore-duplicates modes. Refers to field names after transformation (pick / mapping / flatten).

## `schemaMode` (type: `string`):

<b>JSON</b> (safest) stores each item in a JSONB <code>data</code> column with Apify lineage columns — immune to schema changes. <b>Infer</b> converts item fields into typed SQL columns.

## `schemaEvolution` (type: `string`):

What to do when the source data grows new fields (infer mode only): add columns automatically, ignore the new fields, or fail the run.

## `nestedObjects` (type: `string`):

<b>JSONB</b> keeps nested values as JSON. <b>Flatten</b> turns <code>seller.name</code> into a <code>seller\_name</code> column. <b>Stringify</b> serializes nested values to JSON strings.

## `batchSize` (type: `integer`):

Number of items read and written per batch. Larger batches are faster for big datasets.

## `maxItems` (type: `integer`):

Stop after replicating this many items in one run (0 = unlimited). The checkpoint resumes the rest on the next run — useful for testing and for capping run cost.

## `rememberCheckpoint` (type: `boolean`):

Persist checkpoints and the processed-run registry across runs, so only new data is replicated and interrupted runs resume automatically. Disable to re-replicate everything on each run.

## `includeMetadata` (type: `boolean`):

Add an <code>\_apify</code> field (Actor ID, run ID, dataset ID, item offset, replication time) to every record for lineage and debugging.

## `pickFields` (type: `array`):

Replicate only these fields (dot paths supported, e.g. <code>seller.name</code> becomes <code>seller\_name</code>). Leave empty to replicate all fields.

## `fieldMapping` (type: `object`):

Rename fields: an object of <code>{"destination\_field": "source.path"}</code>, e.g. <code>{"video\_url": "url", "channel": "channel.name"}</code>. When used without "Fields to replicate", only the mapped fields are replicated.

## Actor input object example

```json
{
  "sourceType": "newRuns",
  "actorId": "apify/google-maps-scraper",
  "destinationType": "postgres",
  "tableSchema": "public",
  "createTable": true,
  "s3Prefix": "apify",
  "s3Region": "us-east-1",
  "s3ForcePathStyle": false,
  "writeMode": "upsert",
  "primaryKey": [
    "url"
  ],
  "schemaMode": "json",
  "schemaEvolution": "add-columns",
  "nestedObjects": "jsonb",
  "batchSize": 1000,
  "maxItems": 0,
  "rememberCheckpoint": true,
  "includeMetadata": true
}
```

# Actor output Schema

## `replicationSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "actorId": "apify/google-maps-scraper",
    "primaryKey": [
        "url"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("dtrungtin/apify-dataset-replicator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "actorId": "apify/google-maps-scraper",
    "primaryKey": ["url"],
}

# Run the Actor and wait for it to finish
run = client.actor("dtrungtin/apify-dataset-replicator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "actorId": "apify/google-maps-scraper",
  "primaryKey": [
    "url"
  ]
}' |
apify call dtrungtin/apify-dataset-replicator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dtrungtin/apify-dataset-replicator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/40jPPXf0vgdj5ig3c/builds/MwIk2NnFKSwQGhP3i/openapi.json