# Job Dedupe & Merge — Remove Duplicate Jobs Across Boards (`northbell/job-dedupe-merge`) Actor

Merge job data from several scrapers and remove duplicate jobs: the same opening on LinkedIn, Indeed and a company's Greenhouse page becomes one row, with every URL and source kept. Optional memory returns only jobs that are new since the last run. Nothing is scraped.

- **URL**: https://apify.com/northbell/job-dedupe-merge.md
- **Developed by:** [Northbell](https://apify.com/northbell) (community)
- **Categories:** Jobs, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 job returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Job Dedupe & Merge

**You run more than one job scraper — LinkedIn, Indeed, a company's own Greenhouse or Lever page — and the same opening shows up two or three times, spelled differently each time.** `Sr. Software Engineer, Payments` at `Stripe` in `New York, NY` on LinkedIn is `Senior Software Engineer, Payments` at `Stripe, Inc.` in `NYC` on the career page. Give this Actor all the rows — pasted in, or the IDs of your runs' datasets — and get **one row per job**, with **every URL and source it was found on**. Turn on memory and a daily run returns **only the jobs that are new** — or every job with a `newSinceLastRun` flag to filter on. Nothing is scraped: the Actor only reads what you send.

**$0.50 per 1,000 jobs returned**, plus $0.01 per run.

**Try it now — the input is already filled in.** Press **Start**: the four example rows (one Stripe opening found on LinkedIn, on Stripe's career page and on Indeed, plus a different job) come back as **two rows** in about 3 seconds, for $0.011 (measured on 28 Oct 2026):

| `title` | `duplicateCount` | `allSources` |
| --- | --- | --- |
| Senior Software Engineer, Payments | 2 | linkedin.com, stripe.com, indeed.com |
| Software Engineer, Payments | 0 | linkedin.com |

Each row keeps `allUrls` (every link it was found at) and `earliestPostedAt`. Replace the example with your own rows, or put your runs' dataset IDs in **Datasets to merge**.

### Measured on real data

On 11 October 2026, Stripe's postings from two of our scrapers: **510 rows from LinkedIn and 728 from Stripe's Greenhouse board became 1,079 unique jobs. 143 of them were on both.**

| LinkedIn wrote | The career page wrote |
|---|---|
| Product Operations Manager · New York, United States | Product Operations Manager · NYC, SF, Seattle, US |
| Operations Insights, Tax · United States | Operations Insights, Tax · US Remote |
| Product Designer - Dashboard · United States | Product Designer - Dashboard · US / Canada |

Each pair is one row in the output, with both URLs in `allUrls` and `allSources: ["linkedin.com", "greenhouse"]`.

### Run it again and again

- **On a schedule** — save your input as a task and add a **Schedule**: right after your job scrapers' scheduled runs. Send each run to Google Sheets, Slack or email with an integration on the task.
- **Watch for changes** — Turn on **Remember jobs between runs** (with **Return only new jobs** on): each run returns only jobs not seen before, counted once across all your boards.
- **In bulk** — Pick every scraper run's dataset in **Datasets to merge**: one run reads and merges them all.

### What counts as the same job

- **Same company:** `Stripe`, `Stripe, Inc.` and `Stripe Inc` are one company (legal endings, `&`/`and`, accents and capitals are ignored).
- **Same title, written differently:** `Sr.`/`Senior`, `Mgr`/`Manager`, `(Hunter)`/`- Hunter`, extra spaces, `Remote` and a trailing job number are ignored. **No other word is dropped**: `Engineering Manager, ML - Fraud Detection` and `… - Credit Risk` stay two jobs, and so do `Software Engineer` and `Senior Software Engineer`.
- **Same place:** `NYC` = `New York, NY 10001`, `Remote, US` = `United States (Remote)`. When the same title appears **once on each source**, it is one job even if the places are written differently (LinkedIn says `United States`, the career page lists three cities). When one source lists it in several cities, each city stays its own job.
- **Same link:** the same LinkedIn job ID or Indeed `jk` with different tracking parameters is always one job.
- **A row without a place** joins a job only when that company and title have exactly one place. A row without a company or title is never merged except by link.

We tested the rules on that Stripe data before publishing: an earlier version that dropped the words after a dash merged 42 pairs of different jobs; this one merges none.

### What you get

One row per job: **the fullest of its rows, with all its original fields**, plus:

| Field | |
|---|---|
| `duplicateCount` | how many other rows were the same job |
| `allSources`, `allUrls` | every source and link it was found on |
| `earliestPostedAt` | the earliest posting date among them |
| `dedupeKey` | company · title · place, as matched (stays the same between runs) |
| `newSinceLastRun`, `firstSeenOn` | with memory on |
| `title`, `company`, `location`, `url`, `postedAt` | added only when your rows name them differently (`companyName`, `positionName`, `absolute_url`…), so every row has the same columns |

**Every row, marked** (`outputMode: "annotated"`) returns all your rows instead, each with `dedupeKey`, `isDuplicate` and `duplicateOfUrl` — to check the merge, or to keep your own choice of row.

### A daily feed of new jobs only

Turn on **Remember jobs between runs** and schedule the Actor after your scrapers. Each run returns only jobs not seen before (or all of them, flagged, with **Return only new jobs** off). The memory is kept in your key-value store `job-dedupe-history` for 120 days; give each feed its own **Memory name**.

### Input

- `jobs`: rows from any scraper, as they are. Field names read: `title`/`jobTitle`/`positionName`, `company`/`companyName`/`hiringOrganization`, `location`/`jobLocation`, `url`/`jobUrl`/`applyUrl`/`absolute_url`, `postedAt`/`datePosted`/`postedOn`/`first_published` (also `3 days ago`). Summary and error rows are skipped.
- `datasetIds`: datasets from your other runs, read in full — no copying.
- Up to 100,000 rows per run.

### Pricing

**$0.50 per 1,000 jobs returned**, plus $0.01 per run. Duplicates removed and jobs skipped as already seen are free. The Stripe run above (1,238 rows in, 1,079 jobs out) costs $0.55.

### For AI agents

Call with `{"jobs": [...rows from any job scraper...]}` or `{"datasetIds": ["<id>", "<id>"]}` — seconds for a few thousand rows. Each row has `type: "job"`; `duplicateCount` and `allSources` tell what was merged.

### Related

The rows to merge: [LinkedIn Company Jobs Scraper](https://apify.com/northbell/linkedin-company-jobs-scraper) (every open role at a company), [Greenhouse Jobs Scraper](https://apify.com/northbell/greenhouse-jobs-scraper), [Jobs API](https://apify.com/northbell/job-aggregator-scraper) (461 tech companies' own job boards). Pay, skills and remote from the merged postings: [Job Posting Parser](https://apify.com/northbell/job-posting-parser).

# Actor input Schema

## `jobs` (type: `array`):

Rows from any job scraper, as they are. Read from the usual field names: title / jobTitle / positionName, company / companyName / hiringOrganization, location / jobLocation, url / jobUrl / applyUrl, postedAt / datePosted / postedOn. Summary and error rows are skipped. Up to 100,000 rows per run, together with the datasets below.

## `datasetIds` (type: `array`):

Datasets from your other Actor runs: pick them here, or pass their IDs (defaultDatasetId) through the API. The Actor gets read access to the datasets you pick and nothing else. Each is read in full and merged with the rows above, so you do not have to copy anything.

## `outputMode` (type: `string`):

One row per job keeps the fullest row of each group and adds every URL and source. Every row returns all your rows with the group key, so you can check what was merged.

## `rememberAcrossRuns` (type: `boolean`):

Keep the jobs seen so far (in your key-value store job-dedupe-history, for 120 days) and mark each job as new or already seen. Turn on for a daily feed.

## `onlyNew` (type: `boolean`):

With memory on: return only jobs not seen in an earlier run. Turn off to get all jobs with a newSinceLastRun flag.

## `historyName` (type: `string`):

Separate memories for separate feeds (for example "us-engineering" and "uk-sales"). Letters, numbers and dashes.

## Actor input object example

```json
{
  "jobs": [
    {
      "title": "Sr. Software Engineer, Payments",
      "company": "Stripe",
      "location": "New York, NY",
      "jobUrl": "https://www.linkedin.com/jobs/view/4012345678/",
      "postedOn": "2026-10-03"
    },
    {
      "title": "Senior Software Engineer, Payments",
      "companyName": "Stripe, Inc.",
      "location": "NYC",
      "absolute_url": "https://stripe.com/jobs/listing/senior-software-engineer-payments/7001",
      "first_published": "2026-09-28"
    },
    {
      "positionName": "Senior Software Engineer, Payments",
      "company": "Stripe",
      "location": "New York, NY 10001",
      "url": "https://www.indeed.com/viewjob?jk=a1b2c3",
      "postedAt": "3 days ago"
    },
    {
      "title": "Software Engineer, Payments",
      "company": "Stripe",
      "location": "New York, NY",
      "jobUrl": "https://www.linkedin.com/jobs/view/4012345999/"
    }
  ],
  "outputMode": "merged",
  "rememberAcrossRuns": false,
  "onlyNew": true,
  "historyName": "default"
}
```

# Actor output Schema

## `jobs` (type: `string`):

One row per job, duplicates merged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "jobs": [
        {
            "title": "Sr. Software Engineer, Payments",
            "company": "Stripe",
            "location": "New York, NY",
            "jobUrl": "https://www.linkedin.com/jobs/view/4012345678/",
            "postedOn": "2026-10-03"
        },
        {
            "title": "Senior Software Engineer, Payments",
            "companyName": "Stripe, Inc.",
            "location": "NYC",
            "absolute_url": "https://stripe.com/jobs/listing/senior-software-engineer-payments/7001",
            "first_published": "2026-09-28"
        },
        {
            "positionName": "Senior Software Engineer, Payments",
            "company": "Stripe",
            "location": "New York, NY 10001",
            "url": "https://www.indeed.com/viewjob?jk=a1b2c3",
            "postedAt": "3 days ago"
        },
        {
            "title": "Software Engineer, Payments",
            "company": "Stripe",
            "location": "New York, NY",
            "jobUrl": "https://www.linkedin.com/jobs/view/4012345999/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("northbell/job-dedupe-merge").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "jobs": [
        {
            "title": "Sr. Software Engineer, Payments",
            "company": "Stripe",
            "location": "New York, NY",
            "jobUrl": "https://www.linkedin.com/jobs/view/4012345678/",
            "postedOn": "2026-10-03",
        },
        {
            "title": "Senior Software Engineer, Payments",
            "companyName": "Stripe, Inc.",
            "location": "NYC",
            "absolute_url": "https://stripe.com/jobs/listing/senior-software-engineer-payments/7001",
            "first_published": "2026-09-28",
        },
        {
            "positionName": "Senior Software Engineer, Payments",
            "company": "Stripe",
            "location": "New York, NY 10001",
            "url": "https://www.indeed.com/viewjob?jk=a1b2c3",
            "postedAt": "3 days ago",
        },
        {
            "title": "Software Engineer, Payments",
            "company": "Stripe",
            "location": "New York, NY",
            "jobUrl": "https://www.linkedin.com/jobs/view/4012345999/",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("northbell/job-dedupe-merge").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "jobs": [
    {
      "title": "Sr. Software Engineer, Payments",
      "company": "Stripe",
      "location": "New York, NY",
      "jobUrl": "https://www.linkedin.com/jobs/view/4012345678/",
      "postedOn": "2026-10-03"
    },
    {
      "title": "Senior Software Engineer, Payments",
      "companyName": "Stripe, Inc.",
      "location": "NYC",
      "absolute_url": "https://stripe.com/jobs/listing/senior-software-engineer-payments/7001",
      "first_published": "2026-09-28"
    },
    {
      "positionName": "Senior Software Engineer, Payments",
      "company": "Stripe",
      "location": "New York, NY 10001",
      "url": "https://www.indeed.com/viewjob?jk=a1b2c3",
      "postedAt": "3 days ago"
    },
    {
      "title": "Software Engineer, Payments",
      "company": "Stripe",
      "location": "New York, NY",
      "jobUrl": "https://www.linkedin.com/jobs/view/4012345999/"
    }
  ]
}' |
apify call northbell/job-dedupe-merge --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,northbell/job-dedupe-merge"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zGnNWT2hOTVN9L20r/builds/ZIkapXpyc5f24zzwX/openapi.json