🔥 Power Data Transformer

Quickly select one or more Apify Datasets (must have the same data structure and will be merged) to use as a data source. The source will be referenced as $0 in the 'Transformations' section. For more advanced sources (such as filtered datasets, Key-Value Stores, URLs, or cloud storage), use the 'Sources' field below.

Add data sources to use in your transformation pipeline. Each entry defines a source, referenced as $1, $2, etc., in the 'Transformations' section.

Examples (copy and edit as needed):

• dataset: id='<dataset_id>' or dataset: name='<user/dataset_name>'
• kvstore: id='<store_id>', key='my-key-*'
• http: url='https://example.com/data.json'
• s3: uri='s3://my-bucket/data.csv'

You can use advanced options such as actor_name, last, format, and more. See the README for full details on supported source types and parameters.

Define the data transformation pipeline step-by-step. Each step can be a standard SQL query or a built-in transformation.

• For SQL queries, references must be wrapped in {{...}} (e.g., SELECT * FROM {{$0}}). Bare references like $0 will be treated as plain text.
• For built-in functions (like dedup, ref), the {{...}} wrapper is optional for from and ref_table parameters (e.g., from='#1' and from='{{#1}}' are both valid).

Shorthands:

• # refers to the result of the previous step.
• $ refers to the default data source ($0 or $1).

Examples (copy and edit as needed):

• SELECT id, LOWER(email) FROM {{$0}}
• dedup: from='#', key_fields='id'
• select: from='#', include='id,name'
• ref: from='#', ref_table='$1', key_fields='id', ref_fields='user_id', fields='name'

See the README for all available helper functions (dedup, ref, select, etc.) and their parameters.

Specify where to save the transformed data. By default, it saves the result of the last transformation step. Use from to save an intermediate result (e.g., from='#1').

Examples: (copy and edit as needed):

• dataset: name='my-clean-data'
• kvstore: from='#1', name='my-kv-store', key='output-{{current_date}}', format='json'
• s3: uri='s3://my-bucket/result.csv', write_mode='append', key_fields='id'

See the README for all parameters, including write_mode (overwrite, append, upsert).

Your access key ID for cloud storage.

Your secret access key for cloud storage.

The AWS region of your S3 bucket (e.g., 'us-east-1'). Required for S3.

Your Cloudflare R2 account ID. Required for R2.

Set a default Key-Value Store to be used for any 'sources' or 'outputs' that use a key-value store key but don't specify a key-value store name. This simplifies configurations when you frequently use the same store.

If enabled, the actor will log additional debug messages, including the full SQL query for each transformation step.

Select the timezone for date-related operations, such as parsing dates or using {{current_date}} and {{current_time}} variables in output keys/URIs. This affects functions like now().