# Apify Scraper Data Normalizer (`jfinny/research-data-normalizer`) Actor

Combine output from supported Apify scrapers into consistent records for posts, comments, videos, products, property listings, and places, with source references.

- **URL**: https://apify.com/jfinny/research-data-normalizer.md
- **Developed by:** [Joe Finnegan](https://apify.com/jfinny) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 90.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 normalized records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Apify Scraper Data Normalizer

Combine datasets from different Apify scrapers without writing a separate reader for each one.
Give this Actor datasets of posts, comments, videos, products, property listings, or places.
Get back records in a consistent format, ready for your next script or analysis tool.

### See the output

Here is a shortened example using synthetic Instagram data. The full record has
additional fields for dates, links, and checks on the data.

```json
{
  "schema_version": "2.1.0",
  "record_type": "social_post",
  "identity": {
    "namespace": "instagram",
    "source_id": "3000000000000000001"
  },
  "data": {
    "text": "New research mug https://shop.example/products/research-mug?currency=CAD"
  },
  "measurements": [
    {"name": "likes", "value": 10, "source_field": "likesCount", "state": "reported"}
  ],
  "source": {
    "actor": "apify/instagram-scraper",
    "dataset_id": "demo-posts",
    "record_index": 0
  }
}
```

Instagram calls the post text `caption` and its like count `likesCount`. TikTok
uses `text` and `diggCount`. After normalization, your code finds the text in
`data.text` and the like count in a `measurements` entry named `likes` for either
source. `source_field` keeps the original field name visible.

Products and comments use the same places for IDs and source details, with their
own fields inside `data`. For example, you can load posts, comments, and product
listings in one run, then build your own analysis on top of the results.

$0.50 per 1,000 saved records. Upstream scraper costs are separate.

### How to use it

1. Run one or more supported scrapers to collect your data.
2. Remove the prefilled inline `items`. In the JSON editor, omit the `items` key;
   `items: []` still selects inline input and conflicts with dataset input.
3. Choose output version `2.2.0` for all released source modes and input options.
   Existing `2.1.0` workflows remain supported; Maps and custom mappings require 2.2.
4. For one dataset, select `datasetId` and declare its exact `sourceActor` or mode.
   For several datasets, use both the top-level `datasetIds` picker and `sources`.

For one dataset, replace the placeholder below. Leave `datasetIds`, `sources`,
`items` and `csvText` unset. This example accepts ordinary video-section posts,
including slideshows, from the exact TikTok Profile Actor:

```json
{
  "outputVersion": "2.2.0",
  "datasetId": "REPLACE_WITH_YOUR_DATASET_ID",
  "sourceActor": "clockworks/tiktok-profile-scraper",
  "maxRecords": 1000,
  "includeRaw": true
}
```

This example combines Instagram posts and comments. Replace the placeholder IDs
with your dataset IDs and select those same datasets in the picker:

```json
{
  "outputVersion": "2.1.0",
  "datasetIds": ["POST_DATASET_ID", "COMMENT_DATASET_ID"],
  "sources": [
    {
      "datasetId": "POST_DATASET_ID",
      "sourceActor": "apify/instagram-scraper"
    },
    {
      "datasetId": "COMMENT_DATASET_ID",
      "sourceActor": "apify/instagram-comment-scraper"
    }
  ],
  "includeRaw": true,
  "maxRecords": 1000
}
```

The selected dataset IDs must match the IDs in `sources`. Selecting a dataset
grants read access; naming it in `sources` tells the Actor how to read its fields.
Declare each dataset once. For this source-list input, leave top-level `datasetId`,
`payload`, `items`, `csvText`, `customMapping` and `sourceActor` unset, and keep
top-level `sourceType` at `auto`. Put each source declaration inside `sources`.

For an after-run integration, map `datasetId` to
`{{resource.defaultDatasetId}}`, declare the exact scraper, select a matching
output version and trigger only on `ACTOR.RUN.SUCCEEDED`. The picker grants scoped
READ; a dataset reference inside `payload.resource` alone is not a verified grant.
Automatic handoffs process completed scraper runs. Upstream release monitoring,
automatic mapping repairs and automatic deployments are not enabled.

V1 remains the default for the original Instagram and TikTok post format. Choose
version `2.1.0` explicitly for the wider format shown here. Existing `2.0.0` inputs
keep their earlier fields and behavior. You can also supply up to 100 inline
`items` per source. Declare the source Actor or mode for V2 inline input and leave
dataset fields unset. The Actor accepts at most 100 sources, reads them in the
order supplied, and inspects at most 10,000 rows in total. `maxRecords` defaults to
1,000 and includes invalid rows and repeats.

Inspection limits up to 1,000 use 512 MB automatically; larger limits use
2,048 MB. The default timeout is 300 seconds. Let the Actor choose memory for
large runs. A manual low-memory override can require a longer timeout to finish
the reports.

Version `2.2.0` adds Google Maps places: addresses, coordinates, categories, hours,
closure flags, contacts, ratings, and supplied review samples. Existing sources
use their 2.1 fields in this format. Supply Maps and Instagram datasets together
to give your next tool business details alongside social posts. Supplied social
profile links remain evidence; the normalizer does not discover or verify them.
Shared accounts do not merge branches or attribute posts to a particular venue.

### What stays true to the source

- Missing values stay unknown. A missing count does not become zero.
- Counts keep their meaning. A view count and a play count remain separate.
- Prices keep their known currency and price type. An unclear dollar symbol
  does not become USD, and a product's starting price is not a specific variant's price.
- Links come from IDs or URLs in the supplied data. A comment can link to its
  parent post; similar wording alone does not establish a product connection.
- Changed records and records from separate datasets stay separate. V2 removes
  exact repeated source objects within the same declared source.

With `includeRaw: true`, each saved record also includes the complete original
JSON object received by the normalizer. The standard fields cover a supported
subset of that object. Version 2.1 includes supplied Instagram comment previews
and source identifiers such as variant barcodes, with their original field paths.
A preview is not a complete comment collection; an identifier match is not proof
that two listings describe the same product. Nested data stays inside its parent
record rather than creating extra saved records.
Rejected and repeated rows are not copied to the output, so keep your source
datasets or exports if you need every original row.

This Actor structures data already collected by other tools. Analysis, ranking,
and decisions about what the data means are up to your next tool. It does not
scrape websites or send your data to an LLM.

### Other exports and sources

V2 includes four spreadsheet downloads: records, counts and ratings, prices,
and explicit links. They share observation IDs, so your next tool can read them
together. Counts keep their definitions, prices keep their currencies, and
spreadsheet text formatting preserves exact IDs. JSON remains the main output.
The export manifest reports any rows omitted by the spreadsheet size limit.

For an unsupported or private scraper, version `2.2.0` accepts an explicit
`customMapping`. Declare column paths, identity, conversions and measurement
definitions instead of relying on automatic guesses. The first release supports
posts, videos, comments, products, property listings and media titles. It does not
support custom place mappings or arbitrary nested evidence structures. Supplying
a mapping does not mean that scraper has passed our built-in integration tests.

You can also paste CSV into `csvText`, alongside that mapping and a declared
`sourceActor`. CSV cells remain text until a declared conversion applies. Empty
cells stay empty strings, and large IDs keep their digits. CSV accepts comma,
semicolon, tab or pipe delimiters, with up to 8 MiB of text across sources. Clear
the prefilled `items` first. Remote CSV URLs and file uploads are not supported.
With `includeRaw`, CSV records retain parsed string cells, not original file bytes.

### Where to find the results

Open the run's output dataset for normalized records. The run's key-value store
also contains reports and spreadsheet downloads:

- `OUTPUT`: how many rows were read, saved, skipped, or removed as repeats.
- `PRESERVATION`: what happened to each inspected row and which sources were visited.
- `COMPARISON`: available counts and links found in the supplied data.
- `FIELD_COVERAGE`: confirmed field destinations and source fields that need
  inspection in the original data. Untraced does not mean discarded.
- `TABLES`: spreadsheet filenames, coverage limits and write status. The files
  are `TABLE_RECORDS.csv`, `TABLE_MEASUREMENTS.csv`, `TABLE_PRICES.csv` and
  `TABLE_RELATIONSHIPS.csv`.
- `REJECTIONS`: skipped rows explained in plain language, with repeats, unsaved
  records and source-reading failures listed separately. `REJECTED_ROWS.csv`
  lists rejected rows with source references, hashes and suggested next steps.

These additional artifacts are available for V2 runs. They do not create extra
billable records. Rejection reports contain references and reasons, rather than
copies of the rejected content. Keep your sources if you need to revisit a row.

A run can save results before a later source fails. Check its status and reports
before using the output or retrying. Failed or forcibly stopped runs may not have
final reports, so also inspect any records already saved.

### Supported scrapers

Support covers these result types, rather than every option each scraper offers.
Declare the exact Actor name below, or its supported mode.

| Source Actor | Supported data |
| --- | --- |
| `apify/instagram-scraper` | Posts and Reels |
| `apify/instagram-reel-scraper` | Reels |
| `apify/instagram-post-scraper` | Posts, carousels, and Reels |
| `apify/instagram-hashtag-scraper` | Post and Reel results from hashtag searches |
| `apify/facebook-posts-scraper` | Posts from public feeds, with separate reaction and video counters |
| `apify/facebook-comments-scraper` | Comments and separate reply rows with explicit source references |
| `apidojo/tweet-scraper` | Tweets, explicit reply references, and supplied quote/repost link evidence |
| `trudax/reddit-scraper-lite` | Posts and comments, with reported scores and explicit thread references |
| `clockworks/tiktok-scraper` | Video posts and slideshows |
| `clockworks/tiktok-profile-scraper` | Ordinary video-section posts, including slideshows |
| `apify/instagram-comment-scraper` | Comments, including supported nested replies |
| `clockworks/tiktok-comments-scraper` | Comments and separate reply rows |
| `streamers/youtube-scraper` | Videos, Shorts, and streams |
| `streamers/youtube-comments-scraper` | Comments and replies |
| `harvestapi/linkedin-profile-posts` | Posts |
| `harvestapi/linkedin-post-search` | Post results from searches |
| `apify/e-commerce-scraping-tool` | Supported product and offer records |
| `junglee/amazon-crawler` | Products with separate price types |
| `autofacts/shopify` | Products with nested variants |
| `maxcopell/zillow-detail-scraper` | Individual property details |
| `compass/crawler-google-places` | Ordinary place search and detail rows, with embedded review samples; requires 2.2 |

Standalone profiles, channel summaries, search metadata, category-only rows,
standalone reviews, transcripts, external-service hotel entries, and unrelated
custom formats are outside these mappings. IMDb support remains
experimental and outside the released coverage promise.
TikTok profile stories, repost sections, follower lists, and separate LinkedIn
search reaction or comment rows are outside the new post mappings. Facebook
comments keep source IDs and explicit direct parent references. Replies remain
separate records; source context does not replace the reported author.

Counts keep their source definitions. Facebook reaction totals are separate from
LIKE reactions and its video-view thresholds remain separate. Reddit scores are
not likes. X repost counters have an explicit warning when their scope is not
established. Instagram hidden likes remain unknown. Quote and repost links retain
source evidence without claiming a new relationship type. For fields outside
the standard mapping, use `includeRaw: true` to retain the complete source JSON.

Place ratings and review counts remain separate from social engagement. Opening
hours keep their supplied text; missing schedules do not mean closed. Price
brackets stay text, and open status describes the source observation. Supplied
review samples stay inside their place, with dates, ratings, translations, and
owner responses. They do not create extra saved records or establish completeness.

### Pricing

$0.50 per 1,000 normalized records saved to the output dataset, with no Actor
subscription or start fee. Skipped rows, removed repeats, records that were not
saved, and reports have no normalization event charge. Results saved before a
run fails can still be charged. Upstream scrapers are priced separately.

Set a spending limit of at least $0.0005, the price of one record. Apify rejects
smaller limits before starting a run. Allow headroom when testing completion:
10,000 records succeeded with a $5.10 cap. An earlier exact $5 cap ended as platform
`ABORTED` after outputs were saved. Check terminal status and reports rather than
assuming a full dataset means success. Owner-account platform usage does not
verify an actual paid-customer event charge.

### Data handling

The Actor reads only the datasets you select, using limited permissions, and
leaves them unchanged. Inputs, output, reports, and platform logs are stored in
Apify under the running account's settings. Review sharing and retention settings
before submitting personal data. Turning off `includeRaw` does not remove personal
data from other output fields.

You must have the right to process the supplied data and use the results.
Supporting a scraper does not grant rights to its content or imply affiliation
with its platform. Record hashes help trace the input; they are not copies,
anonymization, or proof that the scraped information is true. Supply large IDs as
strings, since precision lost before the Actor receives them cannot be recovered.

### Support

Operated by Joe Finnegan. For private support, data-access, or deletion requests,
contact jaywfinny@gmail.com. Use is governed by
[Apify's Standard Actor Contract](https://docs.apify.com/legal/standard-actor-contract).

For general support, use the Issues tab with the build, run status, scraper name,
and a small synthetic example. Sharing a run link can give the developer access
to its input, logs, and output, so check the data first. Keep tokens, private
dataset IDs, personal information, and raw customer records out of public issues.

# Actor input Schema

## `datasetId` (type: `string`):

Dataset to read. Clear the inline demo records first. Takes precedence over payload. For integrations use {{resource.defaultDatasetId}} here to grant scoped read access.

## `payload` (type: `object`):

Optional ACTOR.RUN.SUCCEEDED event. Clear the inline demo records first. Payload dataset references alone may not grant read permission; prefer datasetId.

## `sourceType` (type: `string`):

V1 auto detects supported social signatures. V2 auto selects a known caller-declared sourceActor, or applies an explicit customMapping in 2.2. No Actor identity is inferred from record fields.

## `sourceActor` (type: `string`):

Optional caller-declared provenance, e.g. apify/instagram-scraper. Never inferred from record fields.

## `maxRecords` (type: `integer`):

Includes invalid and duplicate input rows.

## `includeRaw` (type: `boolean`):

Append raw source records to output; disabled by default.

## `items` (type: `array`):

Two synthetic demo rows are prefilled for a quick test. Replace them with up to 100 scraped JSON rows, or clear this field when using datasetId, payload, csvText, or sources. The demo saves two billable records. An empty array is a successful empty run.

## `outputVersion` (type: `string`):

V1 preserves the existing social contract. V2.0 keeps the original typed records; V2.1 adds comment previews and product identifier evidence. V2.2 adds places and explicit caller-defined mappings. Built-in source mappings remain unchanged.

## `sources` (type: `array`):

V2 only. Supply datasets, inline JSON groups or explicitly mapped CSV text, each with its source declaration. Use instead of top-level datasetId, payload, items, csvText and source declarations. maxRecords applies across all sources in listed order.

## `datasetIds` (type: `array`):

V2 sources only. Select exactly the datasetId values used in sources. This field grants READ access to those datasets; source declarations identify their adapters.

## `customMapping` (type: `object`):

V2.2 only. Declare source fields, record type, custom namespace and conversions using the customMapping guide. Requires sourceType auto and a caller-declared sourceActor. No inferred schema or arbitrary code.

## `csvText` (type: `string`):

V2.2 only. Paste CSV with an explicit customMapping and sourceActor. Clear items first. Cells remain strings until your mapping explicitly converts them; use the CSV guide. Up to 8 MiB of UTF-8 text across sources.

## `csvOptions` (type: `object`):

For csvText only. Optional delimiter: comma (default), semicolon, tab or pipe. hasHeader defaults to true; false uses column keys 0, 1, and so on. No automatic type or delimiter detection.

## Actor input object example

```json
{
  "sourceType": "auto",
  "maxRecords": 1000,
  "includeRaw": false,
  "items": [
    {
      "shortCode": "SyntheticDemoIG",
      "type": "Video",
      "caption": "Synthetic Instagram demo record. Replace this with your scraped data.",
      "ownerUsername": "rdn_synthetic_demo",
      "likesCount": 12,
      "commentsCount": 3,
      "videoViewCount": 120
    },
    {
      "id": "9000000000000000001",
      "webVideoUrl": "https://www.tiktok.com/@rdn_synthetic_demo/video/9000000000000000001",
      "text": "Synthetic TikTok demo record. Replace this with your scraped data.",
      "authorMeta": {
        "name": "rdn_synthetic_demo"
      },
      "diggCount": 24,
      "commentCount": 6,
      "playCount": 240
    }
  ],
  "outputVersion": "1.0.0"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `comparison` (type: `string`):

No description

## `preservation` (type: `string`):

No description

## `fieldCoverage` (type: `string`):

No description

## `tables` (type: `string`):

No description

## `recordsTable` (type: `string`):

No description

## `measurementsTable` (type: `string`):

No description

## `pricesTable` (type: `string`):

No description

## `relationshipsTable` (type: `string`):

No description

## `rejections` (type: `string`):

No description

## `rejectedRowsTable` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "shortCode": "SyntheticDemoIG",
            "type": "Video",
            "caption": "Synthetic Instagram demo record. Replace this with your scraped data.",
            "ownerUsername": "rdn_synthetic_demo",
            "likesCount": 12,
            "commentsCount": 3,
            "videoViewCount": 120
        },
        {
            "id": "9000000000000000001",
            "webVideoUrl": "https://www.tiktok.com/@rdn_synthetic_demo/video/9000000000000000001",
            "text": "Synthetic TikTok demo record. Replace this with your scraped data.",
            "authorMeta": {
                "name": "rdn_synthetic_demo"
            },
            "diggCount": 24,
            "commentCount": 6,
            "playCount": 240
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("jfinny/research-data-normalizer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "items": [
        {
            "shortCode": "SyntheticDemoIG",
            "type": "Video",
            "caption": "Synthetic Instagram demo record. Replace this with your scraped data.",
            "ownerUsername": "rdn_synthetic_demo",
            "likesCount": 12,
            "commentsCount": 3,
            "videoViewCount": 120,
        },
        {
            "id": "9000000000000000001",
            "webVideoUrl": "https://www.tiktok.com/@rdn_synthetic_demo/video/9000000000000000001",
            "text": "Synthetic TikTok demo record. Replace this with your scraped data.",
            "authorMeta": { "name": "rdn_synthetic_demo" },
            "diggCount": 24,
            "commentCount": 6,
            "playCount": 240,
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("jfinny/research-data-normalizer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    {
      "shortCode": "SyntheticDemoIG",
      "type": "Video",
      "caption": "Synthetic Instagram demo record. Replace this with your scraped data.",
      "ownerUsername": "rdn_synthetic_demo",
      "likesCount": 12,
      "commentsCount": 3,
      "videoViewCount": 120
    },
    {
      "id": "9000000000000000001",
      "webVideoUrl": "https://www.tiktok.com/@rdn_synthetic_demo/video/9000000000000000001",
      "text": "Synthetic TikTok demo record. Replace this with your scraped data.",
      "authorMeta": {
        "name": "rdn_synthetic_demo"
      },
      "diggCount": 24,
      "commentCount": 6,
      "playCount": 240
    }
  ]
}' |
apify call jfinny/research-data-normalizer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jfinny/research-data-normalizer"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0fGdsMMDwcxbU4pUz/builds/qqkrNjIkPNg3I3mGY/openapi.json