# Unified Jobs — one deduplicated feed across all sources (`greyridge-signals/unified-jobs-aggregator`) Actor

Query LinkedIn, Indeed and 10 ATS platforms in ONE run and get a single deduplicated feed: the same role posted in three places collapses to one canonical record with a sources\[] array and the richest merged fields (salary, full description, earliest date). Pay per event, per canonical job.

- **URL**: https://apify.com/greyridge-signals/unified-jobs-aggregator.md
- **Developed by:** [Rez Archer](https://apify.com/greyridge-signals) (community)
- **Categories:** Jobs
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.50 / 1,000 canonical jobs

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Unified Jobs Aggregator — one deduplicated feed across LinkedIn, Indeed, Hacker News & 10 ATS platforms

**Stop running four scrapers and reconciling the duplicates yourself.** This actor
queries **LinkedIn**, **Indeed**, **Hacker News "Who is hiring?"**, and **8
applicant-tracking systems** (Greenhouse, Lever, Ashby, SmartRecruiters, Workday,
Recruitee, Workable, BambooHR) in a single run and returns **one deduplicated,
cross-source-merged feed**.

The same role posted on LinkedIn *and* Indeed *and* the company's own careers page
collapses into **one canonical record** — with a `sources[]` array listing every
place it appeared and the **union of the richest fields** (best salary, longest
description, earliest posting date). You pay once per canonical job, not three times
for the same opening.

### Why this exists (what no single-source scraper gives you)

Every other jobs scraper on the store is **single-source**. If you want coverage you
run several, then you own the hard part yourself: matching "Senior Backend Engineer"
on LinkedIn to "Backend Software Engineer - Infrastructure" on the company's Lever
board to the Indeed listing — across three schemas that each drift independently.
This actor **is** that matching layer, maintained for you.

#### Before (three raw feeds, duplicated)

```
LinkedIn : Backend Software Engineer - Infrastructure | Palantir Technologies | New York, NY
Lever    : Backend Software Engineer - Infrastructure | palantir               | New York, NY
Indeed   : Backend Software Engineer                   | Palantir               | New York
```

#### After (one canonical record — real output, Palantir, verified live)

```jsonc
{
  "id": "79ef84322edd4d78",
  "title": "Backend Software Engineer - Infrastructure",
  "company": "palantir",
  "location": "New York, NY",
  "remote": true,
  "employment_type": "Full-time",
  "posted_at": "2025-08-06T19:28:07.190Z",   // earliest across sources
  "merged": true,
  "primary_url": "https://jobs.lever.co/palantir/6fe5515f-...",  // company ATS = canonical
  "sources": [
    { "source_platform": "lever",    "url": "https://jobs.lever.co/palantir/6fe5515f-...", "posted_at": "2025-08-06T..." },
    { "source_platform": "linkedin", "url": "https://www.linkedin.com/jobs/view/...4281111545", "posted_at": "2025-08-13T..." }
  ]
}
```

One record instead of two. `merged: true` and a `sources[]` of length ≥ 2 tell you it
was found in more than one place — and nothing is ever silently dropped (see below).

### Output schema (unified across every source)

| Field | Notes |
|---|---|
| `id` | Stable hash of normalized company + title + city — deterministic across runs |
| `title`, `company`, `company_domain`, `location`, `remote` | Canonical values |
| `employment_type`, `department`, `team`, `seniority_level`, `industries` | Union across sources |
| `salary`, `salary_min`, `salary_max`, `salary_currency`, `salary_period`, `salary_raw` | Taken whole from the source with the most explicit figures |
| `posted_at` | **Earliest** post date seen across sources (ISO 8601) |
| `description_text`, `description_html` | **Longest** description across sources |
| `sources[]` | `{ source_platform, source_job_key, url, posted_at, company, location }` per place it appeared |
| `primary_url` | URL of the most canonical source (company ATS > LinkedIn > Indeed > Hacker News) |
| `merged` | `true` when the record combined ≥ 2 source hits |
| `scraped_at` | Run timestamp |

### Input (quick start)

```json
{
  "keywords": "data engineer",
  "location": "United States",
  "companies": ["stripe.com", "lever:palantir", "ashby:ramp"],
  "sources": ["ats", "linkedin", "indeed", "hn"],
  "dedupe": true,
  "maxResultsPerSource": 50,
  "maxResults": 200
}
```

- `keywords` + `location` drive the **LinkedIn** and **Indeed** sources.
- `companies` drives the **ATS** source — give domains (`stripe.com`), slugs
  (`stripe`), or explicit `platform:token` (`lever:palantir`). The ATS is
  auto-detected; no board slugs required.
- **`hn` needs no input at all.** The Hacker News source returns the *current*
  month's "Who is hiring?" thread (≈200+ posts) with zero config, then filters by
  `keywords`/`remoteOnly` if you set them — so even an empty input yields a feed.
- `sources` lets you pick a subset. `dedupe: false` returns every raw source hit 1:1.

#### The Hacker News "Who is hiring?" source (why it's maintained, not a wrapper)

Every month the `@whoishiring` bot posts a new thread and hundreds of companies reply
with a free-text job post — no schema, delimiters and field order that drift post to
post, and a **thread ID that changes on the 1st of every month**. This source:

- **auto-discovers the current thread** each run (the monthly rotation never breaks
  the feed, and you never re-point an ID);
- **parses the loose header convention** (`Company | Role | Location | REMOTE | $comp`
  and the em-dash variant) into the same unified schema as every other source, with a
  salary plausibility clamp so a stray "12,000 people" never becomes a bogus salary;
- **degrades open** — a malformed or meta ("please follow the format") comment is
  skipped, never crashes the batch;
- **merges cross-source** — an HN post for *Stripe — Backend Engineer* collapses into
  the same canonical record as Stripe's own ATS posting, contributing the apply URL
  and salary the structured sources often lack.

That recurring upkeep (new thread monthly + free-text drift) is exactly why a buyer
rents the maintained feed instead of re-writing the parser every month.

### How dedup works (and why it won't lie to you)

- **Canonical key** = normalized company **|** normalized title **|** city token.
  Company legal suffixes (`Inc`, `LLC`, `GmbH`…) are stripped so `Stripe` and
  `Stripe, Inc.` match.
- **Conservative by design.** Seniority words and role qualifiers (`- Infrastructure`
  vs `- Defense`) are **kept** — we would rather emit two records for one role than
  merge two genuinely different openings and lose one.
- **Degrade open, never drop.** If a record's company or title can't be normalized
  confidently, it is emitted on its own with `merged: false` instead of being dropped
  or force-merged.
- The run's `RUN_STATS` key reports `raw_source_hits`, `canonical_jobs`,
  `cross_source_merges`, and `collapsed` so you can see exactly how much it deduped.

### Pricing

**Pay per event** — one canonical job after dedup is one billable event. Because
dedup collapses N source hits into one record, you pay **less** than running three
single-source scrapers over the same roles. Billing is event-based, so autonomous
agents can call and pay for this actor per request.

### Maintenance (why you rent instead of build)

Each source rots on its own schedule — LinkedIn reshuffles its guest markup, Indeed
rotates anti-bot, ATS vendors change JSON shapes, and companies migrate between ATS
vendors. Keeping adapters **and** a fuzzy cross-source matcher working across all of
them is ongoing work. That upkeep is the product.

A built-in **cross-source freshness monitor** makes that upkeep self-detecting: it
probes each source family (ATS / LinkedIn / Indeed) against known-good live canaries
and grades three failures that a plain "it ran" check misses — endpoint down, canary
returns zero rows, or records missing key fields (silent schema drift). Run it any
time with `npm run freshness`; it exits non-zero and writes a machine-readable status
file when a gating source drifts, which triggers a maintenance pass. Indeed is graded
advisory (keyless-blocked by design, served via proxy on real runs) and never raises a
false alarm.

### Scope & limits

- **Public postings only.** Company ATS public JSON, and LinkedIn/Indeed public guest
  pages. No login, no member-area data, no PII beyond publicly-posted job fields. See
  `SECURITY.md`.
- **Indeed** is blocked for keyless requests and uses Apify Proxy (datacenter →
  residential). If it is unreachable on a given run, the feed is still built from the
  other sources (the run does not fail).
- High-volume LinkedIn/Indeed pulls rotate through Apify Proxy with a per-source
  circuit breaker that aborts a run whose block-rate is too high, to cap proxy spend.

### Related actors (the suite)

Need just one source? Use the single-source actors this flagship builds on:
`multi-ats-jobs-scraper`, `linkedin-jobs-scraper`, `indeed-jobs-scraper`. They share
this exact schema, so their datasets merge cleanly with this one.

# Actor input Schema

## `keywords` (type: `string`):

Search terms for the LinkedIn and Indeed sources, e.g. "data engineer", "product manager python". Pair with location for best results.

## `location` (type: `string`):

City, region or country for LinkedIn/Indeed, e.g. "United States", "London", "Remote".

## `companies` (type: `array`):

Company domains, slugs, or "platform:token" entries for the ATS source — e.g. \["stripe.com", "lever:palantir", "ashby:ramp"]. The actor auto-detects which ATS (Greenhouse, Lever, Ashby, SmartRecruiters, Workday, Recruitee, Workable) each company uses. Leave empty to query only LinkedIn/Indeed.

## `sources` (type: `array`):

Subset of job sources to include. Default = all. HN Who-is-hiring needs no companies/keywords (it returns the current monthly thread, optionally keyword-filtered). Indeed requires Apify Proxy (it is blocked keyless); if it is unreachable the run degrades gracefully to the other sources.

## `dedupe` (type: `boolean`):

Collapse the same role found on multiple sources into ONE canonical record with a sources\[] array and the union of the richest fields. Turn off to get every raw source hit 1:1.

## `remoteOnly` (type: `boolean`):

Restrict LinkedIn/Indeed to remote postings.

## `datePosted` (type: `string`):

Only postings newer than this window (LinkedIn/Indeed).

## `country` (type: `string`):

Indeed country subdomain, e.g. "www" (US), "de", "uk". Only affects the Indeed source.

## `fullDescription` (type: `boolean`):

Fetch each job's full description + criteria where available (one extra request per job; higher COGS, richer records).

## `maxResultsPerSource` (type: `integer`):

Cap on raw results pulled from EACH source before dedup. Keeps COGS bounded.

## `maxResults` (type: `integer`):

Cap on the final deduplicated records (the billable unit).

## Actor input object example

```json
{
  "keywords": "data engineer",
  "location": "United States",
  "companies": [
    "lever:palantir"
  ],
  "sources": [
    "ats",
    "linkedin",
    "indeed",
    "hn"
  ],
  "dedupe": true,
  "remoteOnly": false,
  "datePosted": "",
  "country": "www",
  "fullDescription": false,
  "maxResultsPerSource": 50,
  "maxResults": 200
}
```

# Actor output Schema

## `results` (type: `string`):

The normalized rows this run produced, in the default dataset.

## `overview` (type: `string`):

Open the run results in the Apify Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keywords": "data engineer",
    "location": "United States",
    "companies": [
        "lever:palantir"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("greyridge-signals/unified-jobs-aggregator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "keywords": "data engineer",
    "location": "United States",
    "companies": ["lever:palantir"],
}

# Run the Actor and wait for it to finish
run = client.actor("greyridge-signals/unified-jobs-aggregator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keywords": "data engineer",
  "location": "United States",
  "companies": [
    "lever:palantir"
  ]
}' |
apify call greyridge-signals/unified-jobs-aggregator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,greyridge-signals/unified-jobs-aggregator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/AmixKxbgY5LdTvBtO/builds/O7jLv8U1n7wo3JzmE/openapi.json