Job Listings Deduplicate: Match Roles Across ATS Boards

For recruiters, job-board operators and analysts who have overlapping job exports and need one useful record for each opening. A retained job costs $0.0005: keeping 100 jobs costs $0.05, plus Apify's small start fee. Removed duplicates, invalid rows and the run summary are free. No login is needed for the supported public ATS boards.

Deduplicate job listings from Greenhouse, Lever, Ashby, Workday, Indeed or another export. The Actor compares employer identity, role, seniority, location and job-description text, then returns duplicate clusters with the best source and the evidence behind each match. Workday and Indeed are supported as imported data; direct URL fetching supports Greenhouse, Lever and Ashby.

What you get

Each cluster begins with a readable verdict. For example: 2 postings of Senior Data Engineer at Example Labs (Greenhouse, Indeed): keep the Greenhouse one. Unique roles also receive their own row, so the result can become your replacement feed.

Field	Meaning
verdict	Plain-English result, including the source to keep
recordType	CLUSTER for a retained job, SUMMARY for the free final row
clusterId	Deterministic identifier for this exact set of normalized members
postingCount	Number of supplied postings represented by the cluster
duplicateCount	Number of redundant copies removed
employer	Employer label from the chosen representative
title	Original title of the chosen representative
location	Original location text, including remote scope where supplied
bestSource	Selected ATS, company source or export label
bestUrl	Selected posting URL with common tracking parameters removed
matchScore	Lowest pairwise match score in a multi-member cluster, 0 to 100; null for a singleton
matchReasons	Evidence such as matching employer, location, source ID or description similarity
bestListing	Representative job, including its normalized fields and description
members	Every member's input index, source, ID, requisition ID, URL and match evidence

The Duplicate clusters dataset view shows the main decision fields. The Deduplicated jobs view exposes representative fields for spreadsheet work. The DEDUPLICATED key-value record contains only the retained job objects, without the summary row. The SUMMARY record explains counts, input problems, truncation and spending-limit handling. The default dataset also ends with this free summary.

Member records omit repeated description bodies to keep exports compact. The chosen representative retains its description, and each member retains the identifying evidence needed to trace it back to your input. Descriptions longer than 50,000 characters are shortened in the output and marked with descriptionTruncated; differing shortened descriptions cannot establish a fuzzy match.

Get started in three steps

1. Paste job objects into listings, supply an accessible Apify datasetId, or add public ATS board URLs in urls. The form includes a fictional three-posting example with a duplicate senior role and a separate junior role.
2. Leave strictness on balanced for the first comparison. Set maxListings to the maximum number of input rows you want examined, and set Apify's maximum run charge to your budget.
3. Start the Actor. Open the cluster view to inspect match reasons, or download DEDUPLICATED for the clean job list. Read SUMMARY before treating an incomplete run as a complete feed.

Inputs

Input	Default	How to use it
listings	Empty outside the form example	Array of exported job objects; at least title and employer are needed
datasetId	None	ID or owner~name of one Apify dataset you can read
urls	Empty	Up to 10 Greenhouse, Lever or Ashby board or individual job URLs
strictness	balanced	strict, balanced or relaxed
maxListings	1000	Combined input-row limit, from 1 to 2,000

Inputs are imported in this order: listings, dataset, URLs. maxListings counts input postings, including copies. Split larger exports into batches, keeping likely duplicates together. The default 256 MB memory covers 2,000 descriptions. Inline JSON must also fit Apify's input-size limit; use a dataset for larger payloads. There is no cross-run matching database.

Common mappings include title, jobTitle, job_title, Lever's text, companyName, company.name, hiringOrganization.name, descriptionPlain, descriptionHtml and Workday's jobPostingInfo. Company website/domain fields and JSON-LD addresses are recognized. Your selected dataset receives read-only access through the input's storage picker.

{
  "listings": [
    {
      "title": "Senior Data Engineer",
      "companyName": "Example Labs",
      "location": "Remote US",
      "source": "Greenhouse",
      "id": "example-101",
      "url": "https://job-boards.greenhouse.io/example/jobs/example-101"
    },
    {
      "jobTitle": "Sr. Data Engineer",
      "company": "Example Labs Inc.",
      "location": "Remote US",
      "source": "Indeed",
      "id": "mirror-101",
      "originalUrl": "https://job-boards.greenhouse.io/example/jobs/example-101"
    }
  ],
  "strictness": "balanced"
}

These two example objects can be matched by their shared original posting URL. When no shared identity exists, include the full job descriptions and locations so there is enough evidence for a text-based match.

For raw Lever or Ashby API exports without an employer, add companyName or companyDomain. Fetched boards use the API company name or a board-identifier fallback reported in SUMMARY. Custom field mappings may need an upstream transformation.

How matching works

Employer names are normalized for punctuation, case and trailing legal suffixes such as Inc., LLC or Ltd. Matching supplied company domains are stronger evidence. Conflicting supplied domains keep records apart. This is a deterministic comparison; it does not call a language model to guess who owns a company.

Titles preserve seniority and specialization. A small explicit synonym list handles Sr. versus Senior, Jr. versus Junior, SWE versus Software Engineer, and Software Developer versus Software Engineer. It also normalizes common frontend, backend and full-stack spellings. A generic Engineer, a Senior Engineer and a Security Engineer are separate roles.

Locations are compared conservatively after a limited set of aliases, including NYC and New York City, SF and San Francisco, and common US/UK country spellings. Remote US and Remote UK remain separate. Missing location does not establish a text-only match. Different known workplace or employment types also keep jobs apart.

Shared original posting URLs or source-specific posting IDs can confirm copies when employer, title and available location fields do not conflict. IDs from different ATS platforms are not treated as a shared identity. Different IDs from the same source, or conflicting requisition IDs, require identical substantial full descriptions and matching locations. This deliberately avoids merging jobs simply because an employer reused a title.

For other candidates, the Actor compares overlapping three-word sequences in normalized descriptions. Strict mode requires 97% similarity, balanced 88%, and relaxed 80%. Descriptions need at least 30 words; the different-ID exception needs at least 50. Scores measure text evidence, not probability. Every member must match every other cluster member. Supported ATS sources are preferred, followed by employer-domain URLs, then other exports; description completeness breaks ties.

Price and spending limits

Retained jobs	Standard listing charge
2	$0.001
100	$0.05
500	$0.25
1,000	$0.50
2,000	$1.00
10,000 across five runs	$5.00

Prices count retained jobs, regardless of how many duplicates each cluster contains. If 1,000 postings become 600 jobs, the listing charge is $0.30. Apify's start fee is $0.00005 at the default memory allocation. Bronze receives 5% off the listing price, Silver 10%, and Gold and higher tiers 20%. The current Store pricing panel is authoritative for your plan.

Invalid input rows, empty boards, unsupported URLs and summary records have no job-listing charge. The platform start fee still applies to a run that produces only notes. There is no synthetic per-dataset-item charge. Free-plan users can use the same input limit because the Actor does not buy outside data or AI services.

The Actor checks the maximum charge before fetching and before writing paid results. When the budget fits only some clusters, it writes that affordable subset and reports the remainder in unwrittenClusters. It does not charge for an output cluster it did not write. Matching may inspect the supplied batch to determine which postings belong together before the affordable output subset is selected.

API examples

Store your Apify token in an environment variable. The following Python example imports an existing dataset and reads the result dataset:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("moggedhq/job-listings-deduplicate").call(
    run_input={"datasetId": "YOUR_DATASET_ID", "strictness": "balanced"},
    max_total_charge_usd=0.50,
)
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
clusters = [r for r in rows if r.get("recordType") == "CLUSTER"]

JavaScript uses the same input and output fields:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('moggedhq/job-listings-deduplicate').call({
  urls: ['https://jobs.lever.co/spotify'], maxListings: 20
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

For a small JSON export saved as input.json:

curl -X POST \
  'https://api.apify.com/v2/acts/moggedhq~job-listings-deduplicate/run-sync-get-dataset-items?maxTotalChargeUsd=0.50' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  --data-binary @input.json

Use with AI agents (MCP)

Connect your agent to https://mcp.apify.com and ask it to use moggedhq/job-listings-deduplicate. Ask it to deduplicate your job dataset and show match reasons. Supply the dataset ID, strictness and maximum charge.

Validation and limits

On 15 October 2026, 30 cloud test inputs constructed from real Greenhouse, Lever and Ashby exports all produced their expected clusters. Each case included a controlled duplicate copy, a separate real posting and a changed-attribute negative control. These are controlled precision checks, not a measured recall rate across naturally occurring third-party mirror feeds. Additional tests cover malformed rows, spending limits and direct ATS fetching. The initial four-posting cloud example returned three retained jobs in 5.8 seconds.

Public boards change, remove jobs and sometimes rate-limit requests. Requests are sequential per run, paced about three seconds apart for one host, and retried with a fresh session. Datacenter proxy is used first, with residential retry only for relevant refusals. Long Retry-After instructions end that source attempt with a note instead of hammering the endpoint. A run has a short URL-fetch time budget and reports sources it could not read.

The Actor reads only public supported ATS endpoints. It does not log into job sites, solve CAPTCHAs, submit applications or fetch arbitrary websites. Workday, Indeed and custom sites need an existing export. It does not translate titles, infer parent-company ownership, find hidden agency clients, verify live status of imported exports or guarantee that every spelling variation will merge. Ambiguous cases are generally left separate. Use source IDs and URLs to inspect consequential decisions.

FAQ

Why did two similar jobs stay separate? Their employer, title, seniority, location, workplace type, requisition or description evidence did not satisfy the selected rules. Missing fields reduce what can be confirmed. Inspect their normalized members before relaxing strictness.

Can it combine different ATS platforms? Yes, when supplied employer, role, location and text provide sufficient evidence, or an original posting URL links the copies. A numeric ID that happens to match across platforms is insufficient.

Does it scrape Workday or Indeed directly? No. It accepts their job exports and maps common fields. Direct URL fetching is limited to Greenhouse, Lever and Ashby.

Are the input datasets modified? No. The Actor reads them and writes its own output. It does not remove records from the original scraper results.

Will the cluster ID remain the same next week? Only when the normalized membership remains the same. A changed description or an added mirror can change the cluster ID. Use the representative source ID or URL for downstream continuity.

What happens with an empty or invalid input? The run writes a free summary explaining the missing data or unsupported source. There is no job-listing charge; Apify's start fee remains.

Does relaxed mode remove the safeguards? No. It lowers only the description-similarity threshold. Employer, role/seniority and location checks remain in place.

Is this affiliated with the ATS providers? No. This is an independent job-listings deduplication tool using supplied exports and the providers' public posting endpoints. Use input data you are entitled to process and respect the source's applicable terms.