Job Scraper — ten boards, one deduplicated list

This job scraper runs one search across Indeed, LinkedIn, Glassdoor, Dice, StepStone, Seek, Xing, Google Jobs, Wellfound and ATS, and returns a single deduplicated list of open roles. The same vacancy cross-posted to four boards becomes one row that names every board it appeared on and carries every apply link, plus salary text and range, remote flag, posted date and employer rating where the board publishes one. It is built for recruiters who source candidates, job boards and job-seeker products that need a feed, and teams doing hiring-intelligence or labour-market research.

How this job scraper works

Two modes cover the usual asks. Search takes a job title plus an optional city and collects matching postings on every board you allowed. Company takes one employer name and pulls every open role that company has posted across the same boards — a compact hiring signal for competitive research or outbound lists.

Both modes run through the same pipeline: collect per board, normalize each posting into one row shape, merge duplicates on a normalized company + title + city key, then emit one summary row followed by the unique roles. Nothing is filtered on how good the posting looks: every row the boards returned is either delivered or accounted for in errors.

Input

Field	Type	Default	Meaning
mode	search | company	search	Which collection style to run.
query	string	—	Job title / keyword. Required by the input schema.
location	string	""	City or region for search mode. Empty means the whole market.
company	string	""	Employer name for company mode.
boards	string[]	mode default	Subset of board names. Unknown names and names from the other mode are dropped.
max_per_board	integer 1–200	50	How many rows to take from each board. A board that returns more is cut off at this number.
concurrency	integer 1–5	4	Accepted for compatibility; it does not change the output.
actor_ids	object	{}	Accepted for compatibility only; the standard board endpoints are used. It no longer overrides sub-actor ids.

Search mode needs a non-empty query. Company mode needs a non-empty company. Invalid input fails the run before any charge, with a fixed status message in English — for example company mode needs \company`.andsearch mode needs `query`.`

Board names: indeed, linkedin, glassdoor, dice, stepstone, seek, xing, google in search mode; indeed, linkedin, glassdoor, dice, google, wellfound, ats in company mode. Give boards in the order you want them visited; names outside the mode are dropped, and a list that filters down to nothing still returns the summary row.

Output

The dataset always starts with exactly one job_summary row, then zero or more unique job rows.

Summary row (_type: "job_summary"): mode, query (the search term, or the company name when that is what you asked for), location, boards_queried, raw_postings, unique_jobs, cross_posted and errors. raw_postings is the pre-dedup count; unique_jobs is what you pay for; cross_posted counts roles that appeared on more than one board.

Job row: board (first board that had it), title, company, location, url, apply_url, salary_text, salary_min, salary_max, remote, posted, job_id, company_rating, source_boards[] and apply_urls[]. Every field is always present; missing values are null. salary_min / salary_max keep the board's own numbers, including sentinel values like -1. A board's company_rating: 0 or remote: false survives normalization. When the same role is merged across boards, those zeros and falses count as gaps and a later filled value overwrites them (the old Python gap-fill rule); a later empty or false value never wipes the first hit. Every job row also carries fetched_at, the ISO timestamp of the collection, and a job_id — the board's own posting id when it published one (string, number, or 0/false exactly as the board sent it), otherwise null. There is no fallback to the posting URL on the field you receive. The summary row has neither.

Duplicates collapse on a normalized company + title + city key: legal suffixes (Inc., GmbH, B.V.) and seniority words (Senior, Sr., Junior) are ignored, and trailing state or country tokens are stripped from the city. Rows without a company name are never collapsed together.

Per-board failures are isolated. A board that errors contributes an errors entry {board, error} and the other boards still deliver; the run succeeds with whatever was collected. A board with no hits is an empty contribution, not an error. Soft failures such as nothing found still give a successful run with an empty job list.

Example from a captured run

This input was run against the live boards and returned 135 raw postings that merged into 134 unique jobs:

{
  "mode": "search",
  "query": "software engineer",
  "location": "New York",
  "boards": ["indeed", "linkedin", "glassdoor", "google"],
  "max_per_board": 50,
  "concurrency": 4
}

The dataset opens with this summary row:

{
  "_type": "job_summary",
  "mode": "search",
  "query": "software engineer",
  "location": "New York",
  "boards_queried": ["indeed", "linkedin", "glassdoor", "google"],
  "raw_postings": 135,
  "unique_jobs": 134,
  "cross_posted": 0,
  "errors": []
}

and continues with rows like this one, captured in the same run:

{
  "board": "indeed",
  "title": "Campus Undergraduate Full-Time Engineer - 2027 Software Engineer I, Enterprise Technology Services- New York, NY",
  "company": "American Express",
  "location": "New York, NY",
  "url": "https://www.indeed.com/viewjob?jk=5cf1c70f82ef3f7f",
  "apply_url": "https://www.indeed.com/viewjob?jk=5cf1c70f82ef3f7f",
  "salary_text": "$78,000 - $124,750 a year",
  "salary_min": 78000,
  "salary_max": 124750,
  "remote": false,
  "posted": "30+ days ago",
  "job_id": "5cf1c70f82ef3f7f",
  "company_rating": 4.1,
  "source_boards": ["indeed"],
  "apply_urls": ["https://www.indeed.com/viewjob?jk=5cf1c70f82ef3f7f"],
  "fetched_at": "2026-10-07T22:36:11.881Z"
}

A role that two boards both had comes back once, with both boards and both links. This row from a company-mode run appeared on google and ats:

{
  "board": "google",
  "title": "Account Executive, AI Startups - Grower",
  "company": "Stripe",
  "location": "New York, NY",
  "url": null,
  "apply_url": null,
  "salary_text": "$210K–$314K a year",
  "salary_min": 210000,
  "salary_max": 314000,
  "remote": null,
  "posted": "22 days ago",
  "job_id": "NraCMLm9TGGhcRQ6AAAAAA==",
  "company_rating": null,
  "source_boards": ["google", "ats"],
  "apply_urls": ["https://stripe.com/jobs/search?gh_jid=8204645"],
  "fetched_at": "2026-10-07T22:38:49.336Z"
}

Large collections

The actor fetches big result sets board by board behind the scenes and returns one complete dataset: the summary row plus the unique jobs. There is no cursor to resend and nothing to resume — a wide search is one run, one dataset, one charge per delivered job.

Use cases for a job scraper

• Sourcing: find every place a role is posted, with all apply links in one row, instead of opening four boards per candidate.
• Hiring intelligence: run company mode on a competitor to see its whole open-roles footprint, and diff successive runs to see which teams keep hiring.
• Salary benchmarking: collect salary_text together with salary_min / salary_max for one job title across cities, keeping the period with the amount.
• Job boards and aggregators: refresh your own index of public vacancies with a link back to the original posting and the boards that carried it.
• Labour-market analysis: count postings per title, board or remote flag over time; raw_postings versus unique_jobs tells you how much cross-posting a market has.
• Lead lists: company mode plus a location is a compact list of employers actively hiring for a role, with the board and posting date as evidence.

The actor's own deduplication covers one run. For a monitor that runs daily, deduplicate across runs using the board together with a non-null job_id only where that board's id stays stable across scrapes. For other rows, build your own key from normalized company, title and city; the posting URL can be a secondary candidate. job_id may be null, and stability across runs is not guaranteed for every board.

Job scraper compared with the alternatives

Open-source scraper libraries. Projects such as JobSpy collect several boards from your own machine for free. They need a working environment, they break whenever a board changes its markup, and merging duplicates and parsing salary text is your code. This actor runs on a schedule through the Apify API, keeps the row shape stable, and returns salary already split into minimum, maximum and text.

Generic scraping APIs and no-code builders. Tools from Bright Data, Zyte, ScraperAPI, ScrapingBee or Octoparse fetch pages you point them at. You still decide which element is the salary, which is the posting date, and you still merge the same role that appears on three boards. Here those decisions are already made per board, for ten boards, and the merge is part of the product.

One board at a time. Single-board scrapers are the right tool when you only care about one market — one country, one niche. The moment you want "everywhere this role is posted", three boards mean three runs, three row shapes and no link between them.

What this job scraper adds. Ten boards in one run, one normalized row shape, a source_boards list and an apply_urls list per role, a summary row that states exactly how many raw postings merged into how many unique jobs, per-board errors that never sink the run, and a price per delivered job rather than per request.

Pricing

You pay per unique job returned (job, $0.006) plus a run start fee (run_start, $0.02). The summary row is free. Empty results cost only the start fee. Runs from a free Apify plan deliver at most 10 dataset rows — the summary plus up to 9 jobs — and only those jobs count; paid plans are untouched.

FAQ

How does the job scraper avoid duplicate rows across boards? Every posting is normalized to a company + title + city key and merged. The surviving row keeps the first board's fields, lists every board in source_boards, and gathers every apply link in apply_urls. Gaps in salary or rating are filled from later hits.

What happens when one job board fails or returns nothing? Nothing found is an empty contribution. A real failure becomes one errors entry for that board while the rest of the run continues, and cross_posted and the counts only reflect what was collected. The run still succeeds.

Which mode should I use for hiring intelligence? Company mode. Give one employer name and you get every open role that company posted across the boards, with board coverage in source_boards — useful to compare a competitor's footprint or build an outbound list of teams that are actively hiring.

Is the summary row billed as a job? No. Only unique job rows count toward the job event. The job_summary row is a rollup and is excluded from the charge, on free-tier runs as well.

Can I limit the run to fewer boards? Yes. Pass boards with the names you want. Unknown names and names from the other mode are dropped; an empty list after filtering still returns the summary row.

Why does company mode still need query? The input schema requires query on every run, company mode included — sending company mode without it is rejected before the run starts. In company mode it is only echoed back in the summary row; the boards are selected by company.

Why is salary_min sometimes -1 or company_rating 0? Those are the values the board published. A -1 minimum means the board showed a salary text but no parseable number, and a rating of 0 is a real rating rather than a missing one. Nothing is invented where a board showed nothing: the field stays null. If the same role was also posted on another board, a later filled rating can still overwrite an earlier 0 when the rows merge — see the job-row notes above.