Universal Database Gateway

Connect PostgreSQL, MySQL, MariaDB, SQLite, or MongoDB to an Apify Dataset. Universal Database Gateway reads a table, runs one safe read-only query, discovers schema metadata, or syncs new rows on a schedule.

The result is ordinary Dataset data that you can send to another Actor, Google Sheets workflow, forecasting pipeline, AI task, or human review process.

What this Actor does

• Tests the database connection before reading data.
• Exports SQL tables or MongoDB collections in bounded batches.
• Runs parameterized SELECT queries with a read-only safety check.
• Selects columns and excludes sensitive columns from table output.
• Saves incremental cursor state in a named Apify Key-Value Store.
• Incremental sync uses a compound cursor + overlap window and dedupes on PK/tiebreak within a run.
• Normalizes timestamps, big integers, decimals, UUIDs, JSON, and binary values.
• Keeps credentials out of Dataset records and normal logs.
• Works from the Apify Console, API, CLI, schedules, and integrations.

This Actor does not host the DBX web application, provide a public SQL console, modify source databases, or require an AI service.

Supported databases

The current build has native provider implementations for these databases:

Status	Database	Available modes
Native and tested	PostgreSQL	table, query, incremental, discover
Native and tested	MySQL and MariaDB	table, query, incremental, discover
Native and tested	SQLite	table, query, incremental, discover
Native and tested	MongoDB	collection export, incremental, discover
Planned	SQL Server, ClickHouse, DuckDB, Redis, Elasticsearch, Oracle, Snowflake, BigQuery, Databricks, Neo4j, Qdrant	Not available yet

Do not select a planned database type. The Actor rejects database types that are not implemented.

Quick start in Apify Console

1. Open the Actor and select Start.
2. Choose a database type.
3. Paste a connection string into Connection string.
4. Choose table as the mode.
5. Enter the schema and table name.
6. Set Maximum rows and start the run.
7. Open the run's Dataset tab to inspect the exported records.

Use a database account with read-only permissions. Do not use localhost or 127.0.0.1 for a cloud run. The database hostname must resolve and accept connections from Apify's compute environment.

PostgreSQL table extraction

Example input:

{
  "databaseType": "postgresql",
  "connectionString": "postgresql://readonly_user:PASSWORD@db.example.com:5432/app?sslmode=verify-full",
  "mode": "table",
  "schema": "public",
  "table": "customers",
  "columns": ["id", "name", "email"],
  "maxRows": 1000
}

Use the direct database connection string when your provider supplies pooled and direct URLs. For Neon, this is usually the unpooled connection URL because the Actor applies a PostgreSQL statement timeout on the connection.

MySQL or MariaDB table extraction

{
  "databaseType": "mysql",
  "connectionString": "mysql://readonly_user:PASSWORD@db.example.com:3306/shop",
  "mode": "table",
  "table": "customers",
  "columns": ["id", "name", "email"],
  "excludeColumns": ["password_hash"],
  "maxRows": 1000
}

MariaDB uses the same input shape with databaseType: "mariadb".

SQLite table extraction

For a SQLite database available inside the Actor container, set the connection string to the file path:

{
  "databaseType": "sqlite",
  "connectionString": "/data/app.sqlite",
  "mode": "table",
  "table": "events",
  "maxRows": 5000
}

The SQLite file must already be available inside the Actor environment. A cloud Actor cannot read a file that exists only on your computer, and this build does not download local files automatically.

MongoDB collection export

MongoDB uses collection terminology instead of SQL tables:

{
  "databaseType": "mongodb",
  "connectionString": "mongodb+srv://readonly_user:PASSWORD@cluster.example.mongodb.net/app",
  "mode": "table",
  "table": "customers",
  "maxRows": 1000
}

Documents remain nested JSON objects. MongoDB SQL query mode is not available in this build.

Read-only query mode

Query mode is useful when table mode is not enough:

{
  "databaseType": "postgresql",
  "connectionString": "YOUR_CONNECTION_STRING",
  "mode": "query",
  "query": "SELECT id, customer_id, total FROM orders WHERE updated_at >= $1",
  "parameters": ["2026-09-01T00:00:00Z"],
  "maxRows": 10000,
  "batchSize": 1000
}

The Actor accepts one SELECT-style statement. It rejects INSERT, UPDATE, DELETE, MERGE, DDL, transaction control, procedure calls, provider commands, and multiple statements. Parameters are passed through the database driver and are never inserted into the SQL string.

The read-only check is an application safeguard. It cannot replace database permissions. Use a database role that can only read the required schema and tables.

Incremental sync

Incremental mode reads rows in stable (cursor, tiebreak) order. It stores a flat JSON cursor object in the named Key-Value Store:

{
  "databaseType": "postgresql",
  "connectionString": "YOUR_CONNECTION_STRING",
  "mode": "incremental",
  "schema": "public",
  "table": "orders",
  "cursor": {
    "field": "updated_at",
    "type": "timestamp",
    "tiebreakField": "id",
    "overlapSeconds": 5
  },
  "stateStoreName": "orders-sync-state",
  "maxRows": 100000,
  "batchSize": 1000
}

Supported cursor types are timestamp, integer, and lexicographically ordered string. Use a populated, indexed field that moves forward reliably and a unique tiebreakField, usually the primary key. SQL providers auto-detect a single primary key; otherwise incremental mode fails with a recommendation to set cursor.tiebreakField. MongoDB requires it explicitly. Timestamp cursors default to a five-second overlap window; integer and string cursors default to no overlap because subtracting a meaningful window is not defined.

The saved state has this shape and can gain fields in future versions:

{
  "cursor_field": "updated_at",
  "cursor_value": "2026-09-08T20:41:02.496Z",
  "tiebreak_field": "id",
  "tiebreak_value": 10001,
  "overlap_seconds": 5,
  "version": 2
}

The query is equivalent to cursor_field > effective_cursor_value OR (cursor_field = effective_cursor_value AND tiebreak_field > tiebreak_value), ordered ascending by both fields. effective_cursor_value is moved backward only for timestamp overlap. Existing bare v1 cursor values, and the earlier {lastCursorValue} record, are migrated in memory and rewritten in v2 after the next successful Dataset batch. Numeric PKs appear as JSON numbers; string PKs appear as strings.

The Actor writes each Dataset batch first, then updates the cursor. A failed Dataset write does not advance the saved cursor. The state store and stateKey must stay the same between scheduled runs.

Schema discovery

Set mode to discover to inspect database metadata without exporting table rows:

{
  "databaseType": "postgresql",
  "connectionString": "YOUR_CONNECTION_STRING",
  "mode": "discover",
  "schema": "public"
}

The output includes schemas and tables. Provider metadata methods also expose table columns, types, nullability, and primary-key information for future integrations.

Input fields

The Console form groups the common settings first. Most users only need these fields:

Field	What to enter
Database type	postgresql, mysql, mariadb, sqlite, or mongodb
Connection string	The complete URL from your database provider, marked secret in the Console
Mode	table, query, incremental, or discover
Schema	Usually public for PostgreSQL; omit for MongoDB and SQLite defaults
Table	SQL table or MongoDB collection name
Columns	Optional list of columns to export
Query	Required only for query mode
Parameters	Values matching $1, $2 or ? placeholders
Maximum rows	Hard per-run limit; default 100000
Batch size	Rows written per Dataset batch; default 1000
Cursor field	Incremental source field, such as updated_at
Cursor type	timestamp, integer, or string
Cursor tiebreak field	Unique tiebreak, usually the PK (id); required for MongoDB, while SQL auto-detects a single PK when possible
Cursor overlap seconds	Timestamp skew window; default 5; not applied meaningfully to integer or string cursors

Advanced settings control timeouts, binary values, serialization failures, and the persistent state store. stateStoreName and stateKey identify the cursor record and should stay unchanged across scheduled runs. Leave advanced settings at their defaults until the basic extraction works.

Connection options and networking

You can use a connection string or the advanced connection object with host, port, database, username, password, ssl, and provider-specific fields. Connection strings are usually the simplest option.

Use the advanced object when your database provider gives you separate connection fields:

{
  "databaseType": "postgresql",
  "connection": {
    "type": "postgresql",
    "host": "db.example.com",
    "port": 5432,
    "database": "app",
    "username": "readonly_user",
    "password": "YOUR_PASSWORD",
    "ssl": true
  },
  "mode": "table",
  "schema": "public",
  "table": "customers"
}

For cloud runs, the database needs a reachable DNS hostname and open database port. A database running on your laptop, inside a private VPC, or behind a firewall will not be reachable automatically. Use an Apify-supported private networking, VPN, proxy, or secure tunnel arrangement when required. This Actor does not create a tunnel or act as a generic network proxy.

Use TLS when your provider supports it. Do not disable certificate verification to work around a TLS error. For PostgreSQL, sslmode=verify-full is an explicit connection-string setting.

Data types and sensitive values

The Dataset preserves source column names where possible. The serializer uses these rules:

Source value	Dataset value
Timestamp or date	ISO-8601 string
Safe integer or bigint	JSON number
Unsafe bigint	Decimal string, preserving exact digits
Decimal or NUMERIC	Driver string, preserving precision
UUID	String
JSON or nested document	Nested JSON
Binary value	Rejected by default; base64 only with binaryPolicy: "base64"
NULL	JSON null

Use excludeColumns for known secrets such as password hashes or API keys. The Actor does not perform automatic PII detection.

FAQ

Does the Actor modify my database?

No. The current modes are for reading and exporting data. Query mode accepts one read-only SQL statement. Use a database account with read-only permissions as an additional safeguard.

Can I connect to a database on my laptop?

Not directly. Apify must be able to resolve the hostname and reach the database port from the Actor runtime. Use a reachable cloud endpoint or an Apify-supported private networking arrangement.

Does incremental sync replace the Dataset?

Each run writes its rows to the run's default Dataset. Overlap re-reads are deduplicated by the tiebreak key within a run and reported as rowsDeduped. This is an insert/update-only export: it does not update or remove earlier Dataset items, and hard deletes in the source remain in previously exported Datasets. Planned work is a cheap periodic PK-only reconciliation report that diffs source keys against Dataset keys and flags vanished rows; full CDC/change streams are later work.

Does incremental sync detect deletes?

No. Incremental sync is explicitly insert/update-only. A source row deleted after export remains in the earlier Apify Dataset. The roadmap is a periodic, read-only PK reconciliation that reports vanished keys before any future full CDC or MongoDB change-stream implementation.

How do I start with the simplest form input?

Choose the database type, paste the provider's connection string, select table, and enter the schema and table name. Start with a small maxRows value, confirm the output, then increase the limit or schedule an incremental run.

Output

Rows go to the run's default Dataset. The OUTPUT record in the default Key-Value Store contains a summary like this:

{
  "databaseType": "postgresql",
  "mode": "incremental",
  "source": "public.gateway_test",
  "rowsRead": 2,
  "rowsWritten": 2,
  "rowsSkipped": 0,
  "rowsDeduped": 0,
  "batches": 1,
  "cursor": {
    "previous": null,
    "current": {
      "cursor_field": "updated_at",
      "cursor_value": "2026-09-08T20:41:02.496Z",
      "tiebreak_field": "id",
      "tiebreak_value": 10001,
      "overlap_seconds": 5,
      "version": 2
    }
  },
  "completed": true
}

Apify gives the run a Dataset, Key-Value Store, logs, API access, scheduling, monitoring, and integrations. Use the Dataset API or connect the Dataset to downstream Actors and workflows after the run finishes.

Roadmap

• Done/current: compound JSON cursor object v2, timestamp overlap window, in-run PK/tiebreak dedupe, and explicit insert/update-only documentation.
• Next: periodic read-only PK-only reconciliation that flags vanished source keys against Dataset keys; it will not silently delete Dataset items.
• Later: MongoDB change streams and fuller CDC support, plus the additional database providers listed as Planned above.

Limits, retries, and failure behavior

The Actor checks the connection before extraction and applies connection and query timeouts where the provider supports them. maxRows and batchSize prevent an omitted limit from becoming an accidental full-table export.

The default rowErrorPolicy is fail. Set it to skip only when losing an individual serialization-failing row is acceptable; the output reports rowsSkipped. Connection failures, invalid SQL, and permission errors still fail the run. Query replay after a partial network failure is intentionally not automatic because replay can duplicate Dataset rows.

Pricing

The Actor supports an optional pay-per-event event named rows-exported-1000. One event represents up to 1,000 successfully written Dataset rows. The Actor charges after Dataset pushes succeed, never for failed connections or failed queries. The billing count is based on cumulative exported rows, not the configurable batchSize; a final partial export counts as one event. Local and non-PPE runs skip this charge and report chargedBatches: 0.

For an initial beta configuration, set rows-exported-1000 to $0.10 in the Actor's Pricing tab. This is a starting price, not a final cost recommendation. Benchmark 1,000, 10,000, and 100,000 row exports before scheduling large jobs, and include Apify compute and storage costs in the review. See docs/benchmark.md.

Configure the event in Apify Console under Development > My Actors > your Actor > Monetization > Pay per event. Add the event name exactly as rows-exported-1000, give it the description Export up to 1,000 database rows to the default Dataset, and set the price for each subscription tier. Disable the automatic apify-default-dataset-item event, or users may be charged both per Dataset item and per export unit. The source code triggers the custom event after successful Dataset persistence.

Scheduling and integrations

Use an Apify Schedule for recurring table exports or incremental syncs. For incremental runs, keep the same named stateStoreName and stateKey. Each run writes new records to its own default Dataset, so downstream workflows should consume the Dataset linked from that run or use an explicit storage workflow when you need a shared destination.

The output is intentionally generic. No special integration code is needed for AI processing, human review, forecasting, Sheets, or another Actor. Pass the Dataset ID or Dataset URL to the next step in your workflow.

Troubleshooting

getaddrinfo ENOTFOUND

The hostname cannot be resolved. Replace placeholder values such as HOST with the real database hostname from your provider. Do not use localhost for a cloud run.

Authentication failed

Check username, password, database name, port, TLS settings, and the database role's remote-login permissions. Use a read-only role with access to the selected schema and table.

Connected but no rows returned

Check the schema, table or collection name, cursor value, and permissions. In incremental mode, an empty Dataset can be correct when no row is newer than the saved cursor.

PostgreSQL SSL warning

Use sslmode=verify-full in the PostgreSQL connection string. The warning is emitted by the PostgreSQL Node driver and does not by itself mean the connection failed.

Security and privacy

Do not put production credentials in source files, README examples, or logs. Use secret input fields in Apify Console and a least-privilege source account. The Actor never writes credentials to Dataset output, but source column values remain your responsibility. Treat exported data as sensitive if the source database contains personal or confidential information.

Read the full docs/security.md for SQL safety, network access, resource limits, credential handling, and known limitations.