Job Feed Deduplicator

Merge duplicate job records from multiple feeds without collapsing conflicting requisitions. The output keeps stable canonical jobs, clean apply URLs, source provenance, and match confidence.

Start in 30 seconds

1. Select Try for free and run with no input for a labeled demo.
2. Supply job records from authorized job boards, aggregators, or ATS exports.
3. Export the canonical dataset for search, analytics, or downstream feeds.

Price: $0.0005 per supplied job record, plus a $0.00005 start event. Demo runs are free, and billing is never based on duplicates removed.

The Actor keeps records separate when requisition identities conflict or the evidence is insufficient.

Features

• Tracking Stripping: Normalizes marketing & tracking parameters (utm_*, ref, gh_src, from, etc.) from apply URLs while preserving functional requisition routing.
• Entity Normalization: Standardizes company names (removes corporate suffixes like Inc, LLC, Ltd) and job titles (standardizes seniority like Sr/Jr, levels, removes bracketed req numbers).
• Safe Merging:
  • Never merges on an absent key alone: Missing companies, titles, or locations are not treated as wildcards.
  • Multi-city protection: Distinct physical locations for the same company and title are preserved as distinct job requisitions.
• Full Provenance & Confidence: Each canonical job links back to all contributing source postings with an explicit match confidence rating (exact, high, medium, low).
• Zero-Network Execution: Runs completely offline and deterministically without external API dependencies.

Input

Input is provided via the standard Apify Actor input or by specifying a local fixture file:

{
  "jobs": [
    {
      "id": "gh-1",
      "source": "greenhouse",
      "title": "Senior Backend Engineer",
      "company": "Stripe, Inc.",
      "location": "San Francisco, CA",
      "applyUrl": "https://boards.greenhouse.io/stripe/jobs/5829101?gh_src=indeed"
    },
    {
      "id": "ind-1",
      "source": "indeed",
      "title": "Sr. Backend Engineer",
      "company": "Stripe",
      "location": "San Francisco, CA",
      "applyUrl": "https://boards.greenhouse.io/stripe/jobs/5829101?utm_source=indeed"
    }
  ]
}

Or reference a local fixture:

{
  "fixturePath": "fixtures/seeded-jobs.json"
}

Output

Canonical job records are pushed to the default dataset:

{
  "canonicalId": "can_f8a9e7b2104c95d3",
  "title": "Senior Backend Engineer",
  "normalizedTitle": "senior backend engineer",
  "company": "Stripe, Inc.",
  "normalizedCompany": "stripe",
  "locations": ["San Francisco, CA"],
  "isRemote": false,
  "canonicalApplyUrl": "https://boards.greenhouse.io/stripe/jobs/5829101",
  "matchConfidence": "exact",
  "matchReasons": ["exact_apply_url_match"],
  "sourceCount": 2,
  "sources": [ ... ]
}

Deduplication summary statistics are written to the default Key-Value Store under the OUTPUT record.

Local Development & Testing

Run the test suite:

$npm test

Run against local fixture:

$npm start

Customer input can contain nonpublic requisitions. Minimize fields, preserve source references, and delete run datasets under your chosen retention policy. Support owner: Mehdi Badawi through the Apify Store support channel, with an initial-response target of two business days.