Unified Jobs Aggregator — one deduplicated feed across LinkedIn, Indeed, Hacker News & 10 ATS platforms

Stop running four scrapers and reconciling the duplicates yourself. This actor queries LinkedIn, Indeed, Hacker News "Who is hiring?", and 8 applicant-tracking systems (Greenhouse, Lever, Ashby, SmartRecruiters, Workday, Recruitee, Workable, BambooHR) in a single run and returns one deduplicated, cross-source-merged feed.

The same role posted on LinkedIn and Indeed and the company's own careers page collapses into one canonical record — with a sources[] array listing every place it appeared and the union of the richest fields (best salary, longest description, earliest posting date). You pay once per canonical job, not three times for the same opening.

Why this exists (what no single-source scraper gives you)

Every other jobs scraper on the store is single-source. If you want coverage you run several, then you own the hard part yourself: matching "Senior Backend Engineer" on LinkedIn to "Backend Software Engineer - Infrastructure" on the company's Lever board to the Indeed listing — across three schemas that each drift independently. This actor is that matching layer, maintained for you.

Before (three raw feeds, duplicated)

LinkedIn : Backend Software Engineer - Infrastructure | Palantir Technologies | New York, NY
Lever    : Backend Software Engineer - Infrastructure | palantir               | New York, NY
Indeed   : Backend Software Engineer                   | Palantir               | New York

After (one canonical record — real output, Palantir, verified live)

{
  "id": "79ef84322edd4d78",
  "title": "Backend Software Engineer - Infrastructure",
  "company": "palantir",
  "location": "New York, NY",
  "remote": true,
  "employment_type": "Full-time",
  "posted_at": "2025-08-06T19:28:07.190Z",   // earliest across sources
  "merged": true,
  "primary_url": "https://jobs.lever.co/palantir/6fe5515f-...",  // company ATS = canonical
  "sources": [
    { "source_platform": "lever",    "url": "https://jobs.lever.co/palantir/6fe5515f-...", "posted_at": "2025-08-06T..." },
    { "source_platform": "linkedin", "url": "https://www.linkedin.com/jobs/view/...4281111545", "posted_at": "2025-08-13T..." }
  ]
}

One record instead of two. merged: true and a sources[] of length ≥ 2 tell you it was found in more than one place — and nothing is ever silently dropped (see below).

Output schema (unified across every source)

Field	Notes
id	Stable hash of normalized company + title + city — deterministic across runs
title, company, company_domain, location, remote	Canonical values
employment_type, department, team, seniority_level, industries	Union across sources
salary, salary_min, salary_max, salary_currency, salary_period, salary_raw	Taken whole from the source with the most explicit figures
posted_at	Earliest post date seen across sources (ISO 8601)
description_text, description_html	Longest description across sources
sources[]	{ source_platform, source_job_key, url, posted_at, company, location } per place it appeared
primary_url	URL of the most canonical source (company ATS > LinkedIn > Indeed > Hacker News)
merged	true when the record combined ≥ 2 source hits
scraped_at	Run timestamp

Input (quick start)

{
  "keywords": "data engineer",
  "location": "United States",
  "companies": ["stripe.com", "lever:palantir", "ashby:ramp"],
  "sources": ["ats", "linkedin", "indeed", "hn"],
  "dedupe": true,
  "maxResultsPerSource": 50,
  "maxResults": 200
}

• keywords + location drive the LinkedIn and Indeed sources.
• companies drives the ATS source — give domains (stripe.com), slugs (stripe), or explicit platform:token (lever:palantir). The ATS is auto-detected; no board slugs required.
• hn needs no input at all. The Hacker News source returns the current month's "Who is hiring?" thread (≈200+ posts) with zero config, then filters by keywords/remoteOnly if you set them — so even an empty input yields a feed.
• sources lets you pick a subset. dedupe: false returns every raw source hit 1:1.

The Hacker News "Who is hiring?" source (why it's maintained, not a wrapper)

Every month the @whoishiring bot posts a new thread and hundreds of companies reply with a free-text job post — no schema, delimiters and field order that drift post to post, and a thread ID that changes on the 1st of every month. This source:

• auto-discovers the current thread each run (the monthly rotation never breaks the feed, and you never re-point an ID);
• parses the loose header convention (Company | Role | Location | REMOTE | $comp and the em-dash variant) into the same unified schema as every other source, with a salary plausibility clamp so a stray "12,000 people" never becomes a bogus salary;
• degrades open — a malformed or meta ("please follow the format") comment is skipped, never crashes the batch;
• merges cross-source — an HN post for Stripe — Backend Engineer collapses into the same canonical record as Stripe's own ATS posting, contributing the apply URL and salary the structured sources often lack.

That recurring upkeep (new thread monthly + free-text drift) is exactly why a buyer rents the maintained feed instead of re-writing the parser every month.

How dedup works (and why it won't lie to you)

• Canonical key = normalized company | normalized title | city token. Company legal suffixes (Inc, LLC, GmbH…) are stripped so Stripe and Stripe, Inc. match.
• Conservative by design. Seniority words and role qualifiers (- Infrastructure vs - Defense) are kept — we would rather emit two records for one role than merge two genuinely different openings and lose one.
• Degrade open, never drop. If a record's company or title can't be normalized confidently, it is emitted on its own with merged: false instead of being dropped or force-merged.
• The run's RUN_STATS key reports raw_source_hits, canonical_jobs, cross_source_merges, and collapsed so you can see exactly how much it deduped.

Pricing

Pay per event — one canonical job after dedup is one billable event. Because dedup collapses N source hits into one record, you pay less than running three single-source scrapers over the same roles. Billing is event-based, so autonomous agents can call and pay for this actor per request.

Maintenance (why you rent instead of build)

Each source rots on its own schedule — LinkedIn reshuffles its guest markup, Indeed rotates anti-bot, ATS vendors change JSON shapes, and companies migrate between ATS vendors. Keeping adapters and a fuzzy cross-source matcher working across all of them is ongoing work. That upkeep is the product.

A built-in cross-source freshness monitor makes that upkeep self-detecting: it probes each source family (ATS / LinkedIn / Indeed) against known-good live canaries and grades three failures that a plain "it ran" check misses — endpoint down, canary returns zero rows, or records missing key fields (silent schema drift). Run it any time with npm run freshness; it exits non-zero and writes a machine-readable status file when a gating source drifts, which triggers a maintenance pass. Indeed is graded advisory (keyless-blocked by design, served via proxy on real runs) and never raises a false alarm.

Scope & limits

• Public postings only. Company ATS public JSON, and LinkedIn/Indeed public guest pages. No login, no member-area data, no PII beyond publicly-posted job fields. See SECURITY.md.
• Indeed is blocked for keyless requests and uses Apify Proxy (datacenter → residential). If it is unreachable on a given run, the feed is still built from the other sources (the run does not fail).
• High-volume LinkedIn/Indeed pulls rotate through Apify Proxy with a per-source circuit breaker that aborts a run whose block-rate is too high, to cap proxy spend.

Related actors (the suite)

Need just one source? Use the single-source actors this flagship builds on: multi-ats-jobs-scraper, linkedin-jobs-scraper, indeed-jobs-scraper. They share this exact schema, so their datasets merge cleanly with this one.