Job Dedupe & Merge

You run more than one job scraper — LinkedIn, Indeed, a company's own Greenhouse or Lever page — and the same opening shows up two or three times, spelled differently each time. Sr. Software Engineer, Payments at Stripe in New York, NY on LinkedIn is Senior Software Engineer, Payments at Stripe, Inc. in NYC on the career page. Give this Actor all the rows — pasted in, or the IDs of your runs' datasets — and get one row per job, with every URL and source it was found on. Turn on memory and a daily run returns only the jobs that are new — or every job with a newSinceLastRun flag to filter on. Nothing is scraped: the Actor only reads what you send.

$0.50 per 1,000 jobs returned, plus $0.01 per run.

Try it now — the input is already filled in. Press Start: the four example rows (one Stripe opening found on LinkedIn, on Stripe's career page and on Indeed, plus a different job) come back as two rows in about 3 seconds, for $0.011 (measured on 28 Oct 2026):

title	duplicateCount	allSources
Senior Software Engineer, Payments	2	linkedin.com, stripe.com, indeed.com
Software Engineer, Payments	0	linkedin.com

Each row keeps allUrls (every link it was found at) and earliestPostedAt. Replace the example with your own rows, or put your runs' dataset IDs in Datasets to merge.

Measured on real data

On 11 October 2026, Stripe's postings from two of our scrapers: 510 rows from LinkedIn and 728 from Stripe's Greenhouse board became 1,079 unique jobs. 143 of them were on both.

LinkedIn wrote	The career page wrote
Product Operations Manager · New York, United States	Product Operations Manager · NYC, SF, Seattle, US
Operations Insights, Tax · United States	Operations Insights, Tax · US Remote
Product Designer - Dashboard · United States	Product Designer - Dashboard · US / Canada

Each pair is one row in the output, with both URLs in allUrls and allSources: ["linkedin.com", "greenhouse"].

Run it again and again

• On a schedule — save your input as a task and add a Schedule: right after your job scrapers' scheduled runs. Send each run to Google Sheets, Slack or email with an integration on the task.
• Watch for changes — Turn on Remember jobs between runs (with Return only new jobs on): each run returns only jobs not seen before, counted once across all your boards.
• In bulk — Pick every scraper run's dataset in Datasets to merge: one run reads and merges them all.

What counts as the same job

• Same company: Stripe, Stripe, Inc. and Stripe Inc are one company (legal endings, &/and, accents and capitals are ignored).
• Same title, written differently: Sr./Senior, Mgr/Manager, (Hunter)/- Hunter, extra spaces, Remote and a trailing job number are ignored. No other word is dropped: Engineering Manager, ML - Fraud Detection and … - Credit Risk stay two jobs, and so do Software Engineer and Senior Software Engineer.
• Same place: NYC = New York, NY 10001, Remote, US = United States (Remote). When the same title appears once on each source, it is one job even if the places are written differently (LinkedIn says United States, the career page lists three cities). When one source lists it in several cities, each city stays its own job.
• Same link: the same LinkedIn job ID or Indeed jk with different tracking parameters is always one job.
• A row without a place joins a job only when that company and title have exactly one place. A row without a company or title is never merged except by link.

We tested the rules on that Stripe data before publishing: an earlier version that dropped the words after a dash merged 42 pairs of different jobs; this one merges none.

What you get

One row per job: the fullest of its rows, with all its original fields, plus:

Field
duplicateCount	how many other rows were the same job
allSources, allUrls	every source and link it was found on
earliestPostedAt	the earliest posting date among them
dedupeKey	company · title · place, as matched (stays the same between runs)
newSinceLastRun, firstSeenOn	with memory on
title, company, location, url, postedAt	added only when your rows name them differently (companyName, positionName, absolute_url…), so every row has the same columns

Every row, marked (outputMode: "annotated") returns all your rows instead, each with dedupeKey, isDuplicate and duplicateOfUrl — to check the merge, or to keep your own choice of row.

A daily feed of new jobs only

Turn on Remember jobs between runs and schedule the Actor after your scrapers. Each run returns only jobs not seen before (or all of them, flagged, with Return only new jobs off). The memory is kept in your key-value store job-dedupe-history for 120 days; give each feed its own Memory name.

Input

• jobs: rows from any scraper, as they are. Field names read: title/jobTitle/positionName, company/companyName/hiringOrganization, location/jobLocation, url/jobUrl/applyUrl/absolute_url, postedAt/datePosted/postedOn/first_published (also 3 days ago). Summary and error rows are skipped.
• datasetIds: datasets from your other runs, read in full — no copying.
• Up to 100,000 rows per run.

Pricing

$0.50 per 1,000 jobs returned, plus $0.01 per run. Duplicates removed and jobs skipped as already seen are free. The Stripe run above (1,238 rows in, 1,079 jobs out) costs $0.55.

For AI agents

Call with {"jobs": [...rows from any job scraper...]} or {"datasetIds": ["<id>", "<id>"]} — seconds for a few thousand rows. Each row has type: "job"; duplicateCount and allSources tell what was merged.

Related

The rows to merge: LinkedIn Company Jobs Scraper (every open role at a company), Greenhouse Jobs Scraper, Jobs API (461 tech companies' own job boards). Pay, skills and remote from the merged postings: Job Posting Parser.