Output to Dataset

Apify actor that merges outputs from multiple actors into a single dataset. Execute actors in series or parallel, combine data from datasets, key-value stores, and webhooks, and get ready-made JSON/CSV/XLSX download links straight from the run logs.

📋 Table of Contents

• Features
• Quick Start
• Input Configuration
  • Sources
  • Actor Runs
  • Execution Mode
  • Merge Strategy
  • Transformations
• Complete Example
• Output
• Use Cases

---

✨ Features

Feature	Description
🔗 Multiple Data Sources	Fetch data from existing datasets, key-value stores, actor runs, or webhook URLs
⚡ Actor Execution	Run multiple actors and collect their outputs in parallel or series
🔀 Merge Strategies	Append all data or deduplicate based on specified fields
🔧 Transformations	Filter, remap, pick, or enrich records before merging
📥 Instant Downloads	Every run logs dataset console link plus JSON, CSV, and XLSX download URLs

---

🚀 Quick Start

Run this actor to merge data from multiple sources into a single dataset:

{
  "actorRuns": [
    {
      "actorId": "apify/web-scraper",
      "input": { "startUrls": [{ "url": "https://example.com" }] }
    }
  ],
  "mergeStrategy": "append"
}

After the run completes, check the logs for direct download links in JSON, CSV, and XLSX formats.

---

📝 Input Configuration

Sources

Merge data from existing Apify resources. Specify an array of sources with their type and identifier:

Type	Description	Required Fields
dataset	Read items from an existing dataset	id - Dataset ID
keyValueStore	Read a record from a key-value store	id - Store ID, key - Record name
actorRun	Read output from a previous actor run	id - Run ID
webhook	Fetch data from an external URL	id - Webhook URL

Example:

{
  "sources": [
    { "type": "dataset", "id": "datasetId123" },
    { "type": "keyValueStore", "id": "storeId456", "key": "OUTPUT" },
    { "type": "actorRun", "id": "runId789" },
    { "type": "webhook", "id": "https://api.example.com/data" }
  ]
}

---

Actor Runs

Execute actors before merging their outputs. Each actor run configuration supports:

Field	Type	Description
actorId	string	Required. Actor ID or username/actor-name
input	object	Input object to pass to the actor
outputType	string	"dataset" (default) or "keyValueStore"
outputKey	string	Record name when using keyValueStore output type

Example:

{
  "actorRuns": [
    {
      "actorId": "apify/web-scraper",
      "input": { "startUrls": [{ "url": "https://example.com" }] },
      "outputType": "dataset"
    },
    {
      "actorId": "apify/google-search-scraper",
      "input": { "queries": "apify" },
      "outputType": "keyValueStore",
      "outputKey": "OUTPUT"
    }
  ]
}

💡 Tip: Use outputType: "dataset" (default) when the actor pushes items to its dataset. Use outputType: "keyValueStore" when the actor saves data via Actor.setValue().

---

Execution Mode

Controls how actors are executed:

Mode	Description
parallel	Default. Run all actors simultaneously for faster results
series	Run actors one after another (useful when order matters or for rate limiting)

---

Merge Strategy

Determines how data is combined:

Strategy	Description
append	Default. Combine all items, keeping duplicates
deduplicate	Remove duplicate items based on deduplicateBy fields

Deduplication example:

{
  "mergeStrategy": "deduplicate",
  "deduplicateBy": ["url", "title"]
}

---

Transformations

Apply transformations to each item before merging. Transformations run in the order provided.

Type	Description	Parameters
filter	Keep only items matching a condition	field, operator, value
mapFields	Copy data from one field path to another	mapping, removeOriginal
pickFields	Keep only specified fields	fields, dropUndefined
setField	Set a static value on a field	field, value, overwrite

Filter operators: equals, notEquals, contains, greaterThan, lessThan, exists

Example:

{
  "transformations": [
    {
      "type": "filter",
      "field": "price",
      "operator": "lessThan",
      "value": 50
    },
    {
      "type": "mapFields",
      "mapping": {
        "title": "product.name",
        "price": "product.price"
      },
      "removeOriginal": true
    },
    {
      "type": "pickFields",
      "fields": ["product.name", "product.price", "url"]
    },
    {
      "type": "setField",
      "field": "currency",
      "value": "USD",
      "overwrite": false
    }
  ]
}

---

📦 Complete Example

This example runs two actors in parallel, merges their outputs with an existing dataset, and deduplicates by URL:

{
  "actorRuns": [
    {
      "actorId": "apify/web-scraper",
      "input": {
        "startUrls": [{ "url": "https://apify.com/store" }],
        "pageFunction": "async function pageFunction(context) { return context.request; }"
      }
    },
    {
      "actorId": "apify/google-search-scraper",
      "input": { "queries": "web scraping" }
    }
  ],
  "sources": [
    { "type": "dataset", "id": "existingDatasetId" }
  ],
  "executionMode": "parallel",
  "mergeStrategy": "deduplicate",
  "deduplicateBy": ["url"]
}

---

📤 Output

All merged records are saved to the actor's default dataset. After each run, the logs display:

1. Console link - Direct link to view the dataset in Apify Console
2. Download URLs - Ready-to-use links for JSON, CSV, and XLSX exports

You can also export the dataset manually from the Apify Console in any supported format.

---

💡 Use Cases

1. Merge Multiple Scraping Runs

Run the same scraper with different inputs and merge all results:

{
  "actorRuns": [
    { "actorId": "my-scraper", "input": { "category": "electronics" } },
    { "actorId": "my-scraper", "input": { "category": "books" } },
    { "actorId": "my-scraper", "input": { "category": "clothing" } }
  ],
  "executionMode": "parallel",
  "mergeStrategy": "append"
}

2. Combine Historical Data

Merge data from multiple previous actor runs:

{
  "sources": [
    { "type": "actorRun", "id": "run1" },
    { "type": "actorRun", "id": "run2" },
    { "type": "actorRun", "id": "run3" }
  ],
  "mergeStrategy": "deduplicate",
  "deduplicateBy": ["id"]
}

3. Aggregate Multiple Datasets

Combine existing datasets into one unified dataset:

{
  "sources": [
    { "type": "dataset", "id": "dataset1" },
    { "type": "dataset", "id": "dataset2" },
    { "type": "dataset", "id": "dataset3" }
  ]
}

---

📄 License

This project is licensed under the ISC License.