Under maintenance

Pricing

from $1.00 / 1,000 replicated records

Try for free

Go to Apify Store

Apify Dataset Replicator

Under maintenance

Try for free

Sync the results of your Apify Actors to PostgreSQL or S3-compatible storage - continuously, incrementally, and without ever starting from scratch. Give it a connection string and every new successful run lands in your database as clean rows, ready for your application, BI dashboard, or AI pipeline.

Pricing

from $1.00 / 1,000 replicated records

Rating

0.0

(0)

Developer

Tin

Actor stats

Bookmarked

Total users

Monthly active users

14 days ago

Last modified

Categories

Developer tools

Source type

sourceType

Optional

Where the data comes from. New successful runs watches an Actor and replicates every successful run exactly once. Last successful run replicates only the most recent one. Specific dataset replicates a single (possibly growing) dataset incrementally.

Type:string

Default:newRuns

Options:

newRunslastRundataset

Actor

actorId

Optional

The Actor whose run results to replicate, e.g. apify/google-maps-scraper. Required for the Actor source types.

Type:string

Dataset ID

datasetId

Optional

ID or name of the dataset to replicate. Required when the source type is Specific dataset.

Type:string

Destination type

destinationType

Optional

Where to replicate the data. PostgreSQL covers Supabase, Neon, Railway, Render, RDS, Cloud SQL and self-hosted servers. S3 covers AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO and other S3-compatible storage.

Type:string

Default:postgres

Options:

postgress3

PostgreSQL connection string

connectionString

Optional

Connection string in the form postgresql://user:password@host:5432/database. Stored encrypted.

Type:string

PostgreSQL schema

tableSchema

Optional

Database schema of the destination table.

Type:string

Default:public

Table name

tableName

Optional

The table to replicate into, e.g. google_places.

Type:string

Automatically create table

createTable

Optional

Create the destination table (and the unique index required for upsert) if it does not exist yet.

Type:boolean

Default:true

S3 bucket

s3Bucket

Optional

Name of the destination bucket.

Type:string

S3 key prefix

s3Prefix

Optional

Folder prefix for replicated files, e.g. apify. Files are written as {prefix}/{actor}/{date}/run-{runId}/part-000001.jsonl.gz plus a manifest.json.

Type:string

Default:apify

S3 region

s3Region

Optional

Region of the bucket (or any value accepted by the provider, e.g. auto for Cloudflare R2).

Type:string

Default:us-east-1

S3 endpoint URL

s3Endpoint

Optional

Custom endpoint for S3-compatible providers, e.g. https://<account>.r2.cloudflarestorage.com or a MinIO URL. Leave empty for AWS S3.

Type:string

S3 access key ID

s3AccessKeyId

Optional

Access key for the bucket. Stored encrypted.

Type:string

S3 secret access key

s3SecretAccessKey

Optional

Secret key for the bucket. Stored encrypted.

Type:string

Use path-style S3 addressing

s3ForcePathStyle

Optional

Required by MinIO and some other S3-compatible services. Defaults to on when a custom endpoint is set.

Type:boolean

Default:false

Write mode

writeMode

Optional

Upsert inserts or updates by primary key (recommended — makes replication idempotent). Append always inserts. Ignore duplicates inserts only unseen keys. Replace truncates the table and reloads everything on every run.

Type:string

Default:upsert

Options:

upsertappendignoreDuplicatesreplace

Primary key field(s)

primaryKey

Optional

Field(s) that uniquely identify a record, e.g. url, placeId, asin. Required for upsert and ignore-duplicates modes. Refers to field names after transformation (pick / mapping / flatten).

Type:array

Table schema mode

schemaMode

Optional

JSON (safest) stores each item in a JSONB data column with Apify lineage columns — immune to schema changes. Infer converts item fields into typed SQL columns.

Type:string

Default:json

Options:

jsoninfer

Schema evolution

schemaEvolution

Optional

What to do when the source data grows new fields (infer mode only): add columns automatically, ignore the new fields, or fail the run.

Type:string

Default:add-columns

Options:

add-columnsignorefail

Nested objects & arrays

nestedObjects

Optional

JSONB keeps nested values as JSON. Flatten turns seller.name into a seller_name column. Stringify serializes nested values to JSON strings.

Type:string

Default:jsonb

Options:

jsonbflattenstringify

Batch size

batchSize

Optional

Number of items read and written per batch. Larger batches are faster for big datasets.

Type:integer

Minimum:1

Maximum:50000

Default:1000

Max items per run

maxItems

Optional

Stop after replicating this many items in one run (0 = unlimited). The checkpoint resumes the rest on the next run — useful for testing and for capping run cost.

Type:integer

Minimum:0

Default:0

Remember sync position

rememberCheckpoint

Optional

Persist checkpoints and the processed-run registry across runs, so only new data is replicated and interrupted runs resume automatically. Disable to re-replicate everything on each run.

Type:boolean

Default:true

Include Apify metadata

includeMetadata

Optional

Add an _apify field (Actor ID, run ID, dataset ID, item offset, replication time) to every record for lineage and debugging.

Type:boolean

Default:true

Fields to replicate

pickFields

Optional

Replicate only these fields (dot paths supported, e.g. seller.name becomes seller_name). Leave empty to replicate all fields.

Type:array

Field mapping

fieldMapping

Optional

Rename fields: an object of {"destination_field": "source.path"}, e.g. {"video_url": "url", "channel": "channel.name"}. When used without "Fields to replicate", only the mapped fields are replicated.

Type:object

PostgreSQL Database Extractor

parseforge/postgresql-database-extractor

Extracts data from PostgreSQL databases. Connects via connection string or individual credentials, supports SSL, table selection, and row limits. Returns each row as a flat JSON object.

ParseForge

5.0

🎉 Apify Actors

prog-party/apify-actors

This Apify Actors Actor retrieves data from Apify, allowing to filter, and returns a list of actors as a Dataset.

Prog Party

Apify Creators · Profiles & Actor Export

corent1robert/apify-users-sitemap-scraper

Extract every public Apify creator profile in one run. Get name, username, published actors, and usage stats — ready to export as a lead list or enrich in your CRM

Corentin Robert

Recall Feed to S3/R2

zentrafoundry/recall-feed-to-s3-r2

Monitor EU Safety Gate alerts, Apify datasets/storage, S3-compatible storage / Cloudflare R2 and return product, brand, risk, authority, market, source URL, and business-action fields.

Zentra

Apify Monitor

coinbase/apify-monitor

Get Your Monetization Stats of Apify

Live

PostgreSQL Insert

petr_cermak/PostgreSQL-insert

This act takes a crawler execution and inserts it's results into a remote PostgreSQL database.

Petr Cermak

Promote Apify Actors

badruddeen/promote-apify-actors

Automatically gathers all your Apify actors and emails them in a beautifully formatted card, making it easy to promote your work and share your automation projects with anyone — one click, zero manual copy-paste.

Badruddeen Naseem

Apify Actor Run Monitor: Empty Output Alerts

davidbenittah/apify-run-output-health-monitor

Check up to 50 Apify Actors and find the ones that are quietly broken: no recent run, a failed or aborted run, a failure rate above your threshold, a successful run with an empty dataset, or output missing a required field. One health row per Actor, plus a summary.

David

Apify

doshikevin361/apify

Kevin Doshi

Apify Portfolio Showcase

marielise.dev/apify-portfolio-showcase

Turn your Apify profile into stunning portfolio widgets. Extract all your actors with reviews, ratings, and stats - ready to embed anywhere in seconds.