LinkedIn Jobs Scraper

A lightweight, anti-bot-resilient Apify Actor that scrapes public LinkedIn job postings at scale — without a browser and without login.

Apify Actor · TypeScript · Crawlee · CheerioCrawler · HTTP-first · Residential Proxy

The LinkedIn Jobs Scraper collects job listings from LinkedIn's public guest endpoints using a fast, HTTP-only crawler. It runs in guest mode by default (no credentials required), optionally accepts an li_at cookie for richer data, and uses query segmentation to break past LinkedIn's ~1,000-results-per-query limit. Results are streamed directly to the dataset, keeping runs light on memory and compute.

---

Table of Contents

• Features
• How It Works
• Input Reference
• Example Input
• Example Output
• Guest Mode vs. li_at Cookie Mode
• Anti-Bot & Rate-Limit Strategy
• Performance & Cost
• Limitations
• Troubleshooting
• Optional Playwright Fallback
• Running Locally & Deploying
• Legal & Compliance

---

Features

• Lightweight, HTTP-first crawling. Built on Crawlee's CheerioCrawler, which fetches and parses raw HTML without launching a headless browser. This makes runs dramatically faster and cheaper than browser-based scrapers. CheerioCrawler uses got-scraping internally, so realistic browser-like headers and TLS fingerprints are applied automatically.
• No-login guest mode. By default the Actor targets LinkedIn's public guest endpoints and requires no account or credentials.
• Optional li_at cookie. Provide a session cookie to unlock richer, more complete data. The cookie is treated as a secret and is never written to logs or the dataset.
• Anti-bot resilience. Combines Apify Residential Proxy with an automatic session pool that rotates IPs on rate-limit, soft-ban, or CAPTCHA signals — plus retries with exponential backoff and configurable request throttling.
• Query segmentation to exceed the ~1,000-result cap. Automatically splits a single search into multiple filtered sub-queries (by posted time, experience level, and work type) so you can collect far more than the ~1,000 results LinkedIn returns for any single query. Global de-duplication by jobId prevents duplicates across segments.
• Full job fields. Extracts a comprehensive set of fields including title, company, location, salary, seniority, employment type, job function, industries, applicant count, and the full job description (HTML and plain text).
• Streaming output. Each job is pushed to the dataset as soon as it is parsed (Actor.pushData), so memory stays flat even on very large runs.

---

How It Works

The Actor uses a two-stage architecture driven by a Crawlee RequestQueue:

1. Search list stage (SEARCH_LIST). The Actor requests LinkedIn's guest job-search endpoint, parses each job card, extracts the jobId, and enqueues a detail request for it. It then enqueues the next page of results (start += 25) until it runs out of results, reaches maxResults, or hits the per-segment page limit.
2. Detail stage (JOB_DETAIL). For each queued job, the Actor fetches the guest job-posting endpoint and parses the full detail (description, criteria, seniority, etc.), then streams the finished record to the dataset. If includeJobDescription is false, records are pushed directly from the list stage and the detail stage is skipped for speed.

Segmentation Strategy (in plain terms)

LinkedIn only returns roughly the first 1,000 results for any single search, no matter how many jobs actually match. To get more, the Actor splits one broad search into many narrower ones that together cover the same ground:

1. First it segments by posted time (e.g. past 24 hours / past week / past month).
2. Then, within each of those, it segments by experience level (6 levels).
3. Then, within each of those, it segments by work type (on-site / remote / hybrid).

Each combination is its own search with its own pagination, so each one gets its own ~1,000-result budget. Because different segments can surface the same job, a global de-duplication set keyed on jobId ensures every job is emitted only once. Segmentation is enabled by default and can be turned off with enableSegmentation: false.

---

Input Reference

Every input field is defined in the Actor's input schema. All fields except keywords are optional.

Field	Type	Default	Description
keywords	string	(required)	Search terms for the job title/role, e.g. "software engineer".
location	string	"United States"	Geographic location to search within, e.g. a country, city, or region.
maxResults	integer	100	Maximum number of jobs to collect. Use 0 for no explicit cap (bounded by LinkedIn availability and segmentation).
timePosted	string (enum)	"any"	Filter by posting recency. One of any, 24h, week, month. Maps to LinkedIn's f_TPR.
experienceLevel	array of string (enum)	[]	Filter by seniority. Values: internship, entry-level, associate, mid-senior, director, executive. Maps to f_E.
jobType	array of string (enum)	[]	Filter by employment type. Values: full-time, part-time, contract, temporary, volunteer, internship. Maps to f_JT.
workSchedule	array of string (enum)	[]	Filter by work arrangement. Values: on-site, remote, hybrid. Maps to f_WT.
easyApplyOnly	boolean	false	If true, restrict results to Easy Apply jobs. Maps to f_AL.
includeJobDescription	boolean	true	If true, fetch each job's detail page for the full description and extra fields. Set to false for faster, lighter list-only runs.
enableSegmentation	boolean	true	If true, split the search into filtered sub-queries to exceed the ~1,000-result-per-query cap.
liAtCookie	string (secret)	""	Optional LinkedIn li_at session cookie for richer data. Treated as a secret and never logged or stored.
proxyConfiguration	object	{ "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }	Apify proxy configuration. Residential proxies are strongly recommended for reliability.
maxConcurrency	integer	50	Maximum number of requests processed in parallel.
maxRequestRetries	integer	3	Maximum retry attempts per request before it is marked failed.
maxRequestsPerMinute	integer	200	Global throttle on request rate to reduce the chance of rate limiting.

---

Example Input

{
  "keywords": "software engineer",
  "location": "United States",
  "maxResults": 200,
  "timePosted": "week",
  "experienceLevel": ["mid-senior", "associate"],
  "jobType": ["full-time"],
  "workSchedule": ["remote", "hybrid"],
  "easyApplyOnly": false,
  "includeJobDescription": true,
  "enableSegmentation": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  },
  "maxConcurrency": 50,
  "maxRequestRetries": 3,
  "maxRequestsPerMinute": 200
}

---

Example Output

Each dataset item represents one job posting:

{
  "jobId": "3801234567",
  "title": "Senior Software Engineer",
  "company": "Acme Corporation",
  "companyUrl": "https://www.linkedin.com/company/acme-corp",
  "location": "San Francisco, CA",
  "workplaceType": "Hybrid",
  "salary": "$150,000 - $190,000/yr",
  "postedAt": "2026-07-02T00:00:00.000Z",
  "jobUrl": "https://www.linkedin.com/jobs/view/3801234567",
  "description": "We are looking for a Senior Software Engineer to join our platform team. You will design, build, and operate scalable backend services...",
  "descriptionHtml": "<p>We are looking for a <strong>Senior Software Engineer</strong> to join our platform team. You will design, build, and operate scalable backend services...</p>",
  "seniorityLevel": "Mid-Senior level",
  "employmentType": "Full-time",
  "jobFunction": ["Engineering", "Information Technology"],
  "industries": ["Software Development", "Technology, Information and Internet"],
  "applicantsCount": 47,
  "easyApply": false,
  "criteria": [
    { "name": "Seniority level", "value": "Mid-Senior level" },
    { "name": "Employment type", "value": "Full-time" },
    { "name": "Job function", "value": "Engineering and Information Technology" },
    { "name": "Industries", "value": "Software Development" }
  ],
  "scrapedAt": "2026-07-09T12:34:56.000Z"
}

Output Fields

Field	Type	Description
jobId	string	LinkedIn's unique identifier for the posting.
title	string	Job title.
company	string	Hiring company name.
companyUrl	string | null	LinkedIn URL of the company page.
location	string	Job location as displayed by LinkedIn.
workplaceType	string | null	On-site, Remote, or Hybrid.
salary	string | null	Salary text if published.
postedAt	string (ISO 8601) | null	When the job was posted.
jobUrl	string	Public URL of the job posting.
description	string | null	Full job description as plain text.
descriptionHtml	string | null	Full job description as HTML.
seniorityLevel	string | null	Seniority level.
employmentType	string | null	Employment type (Full-time, Contract, etc.).
jobFunction	array of string	Job function categories.
industries	array of string	Relevant industries.
applicantsCount	integer | null	Number of applicants, when available.
easyApply	boolean	Whether the posting supports LinkedIn Easy Apply.
criteria	array of object	Name/value pairs of the job criteria block.
scrapedAt	string (ISO 8601)	Timestamp when the record was scraped.

The dataset ships with two views:

• overview — a compact table: title, company, location, postedAt, jobUrl.
• detailed — all fields above.

---

Guest Mode vs. li_at Cookie Mode

Guest Mode (default)

The Actor targets LinkedIn's public guest endpoints and needs no account. This is the recommended starting point:

• No credentials to manage and no risk to a personal account.
• Fully covers the core fields most users need.
• Lower friction and simpler to run reliably.

li_at Cookie Mode (optional)

Supplying a valid li_at session cookie can unlock richer and more complete data on some pages (for example, more consistent applicant counts or fields that guest pages omit).

Security warning. A li_at cookie is equivalent to being logged into that LinkedIn account. Handle it with care:

• Use a secondary / throwaway account, never your primary personal account. Automated access may trigger account restrictions or bans.
• The cookie field is marked as a secret; its value is never printed to logs and never stored in the dataset (it is redacted as [REDACTED] everywhere).
• Cookies expire. If richer fields stop appearing, refresh the cookie (see Troubleshooting).
• Never commit cookies to source control or share them.

---

Anti-Bot & Rate-Limit Strategy

LinkedIn actively defends against automated access. This Actor uses several layers to stay resilient:

• Residential proxy rotation. By default it routes traffic through Apify's RESIDENTIAL proxy group, which presents real consumer IPs that are far less likely to be blocked than datacenter IPs.
• Session pool with automatic retirement. A pool of sessions is maintained; when a session encounters a 429 (rate limit), 999 (soft ban), or a CAPTCHA / challenge response, that session is retired so the next request uses a fresh IP and identity.
• Retries with exponential backoff. Failed requests are retried up to maxRequestRetries times, with increasing delays to avoid hammering the target.
• Request throttling. maxRequestsPerMinute caps the global request rate, and maxConcurrency bounds parallelism — both help you stay under LinkedIn's radar.
• Realistic fingerprints. CheerioCrawler (via got-scraping) applies browser-like headers and TLS fingerprints automatically.
• Defensive parsing. Every field is extracted with null-safe helpers wrapped in per-card try/catch, so a markup change or a missing field yields null/defaults instead of failing an entire page.

If you still see frequent blocks, lower maxConcurrency and maxRequestsPerMinute, and keep Residential proxies enabled.

---

Performance & Cost

This Actor is designed to be lightweight ("ringan"): because it uses CheerioCrawler (raw HTTP + HTML parsing) instead of a headless browser, it consumes far fewer compute units (CUs) than browser-based alternatives.

Practical guidance for keeping usage light:

• Start small. Set a modest maxResults (e.g. 50–200) for your first runs to validate your inputs before scaling up.
• Skip detail pages when you can. Set includeJobDescription: false to collect list-level fields only — this roughly halves the number of requests and is significantly faster and cheaper.
• Disable segmentation for small jobs. If you only need a few hundred results, set enableSegmentation: false to avoid issuing many sub-queries.
• Tune concurrency to your needs. Lower maxConcurrency and maxRequestsPerMinute reduce peak resource usage and block risk; raise them (carefully) for throughput.
• Proxy costs. Residential proxy traffic is billed separately from compute. Fetching detail pages increases proxy usage, so includeJobDescription: false also reduces proxy spend.

Actual compute units depend on result volume, whether detail pages are fetched, and retry rates. Monitor your first few runs in the Apify console to calibrate.

---

Limitations

• ~1,000 results per query cap. LinkedIn returns at most ~1,000 results for any single search. The Actor mitigates this with query segmentation (enabled by default), but the total reachable set is still bounded by the filter combinations available.
• JS-rendered content. Some content is rendered client-side and may not appear in the raw HTML served to an HTTP client. Such fields may be null. For those cases, see the Optional Playwright Fallback.
• Markup changes. LinkedIn changes its markup periodically, which can break selectors. Parsing is defensive and selectors are centralized to ease updates, but occasional maintenance may be required.
• Terms of Service. Scraping LinkedIn may conflict with LinkedIn's Terms of Service. You are responsible for how you use this Actor — see ./LEGAL_NOTES.md.

---

Troubleshooting

Getting 429 / 999 / CAPTCHA responses or frequent retries

• Ensure proxyConfiguration uses Apify Residential proxies.
• Lower maxConcurrency (e.g. to 10–20) and maxRequestsPerMinute (e.g. to 60–100).
• Increase maxRequestRetries slightly to give sessions more chances to recover.

Empty or very few results

• Verify keywords and location are valid and not overly narrow.
• Relax filters (timePosted, experienceLevel, jobType, workSchedule, easyApplyOnly).
• Confirm the proxy is working (check run logs for proxy errors).
• Try again later — transient blocking can temporarily reduce results.

Missing detail fields (description, applicants, etc.)

• Ensure includeJobDescription: true.
• Some fields are only reliably available with a valid li_at cookie — consider cookie mode.

Cookie no longer works / richer fields disappeared

• li_at cookies expire. Log in with your secondary account, copy a fresh li_at value, and update the liAtCookie input.

Fewer results than expected on large searches

• Make sure enableSegmentation: true.
• Remember the ~1,000-per-query cap; add more specific filters to create additional segments.

---

Optional Playwright Fallback

For the rare pages whose content is only available after client-side JavaScript execution, a Playwright-based browser fallback could be added as an escape hatch. This is documented but not implemented by default, because a headless browser is heavier, slower, and more expensive in compute units — which conflicts with this Actor's lightweight goal.

If you need it, the intended approach is to swap CheerioCrawler for PlaywrightCrawler (or add a secondary crawler) for the specific requests that require rendering, keeping the same proxy, session, and parsing strategy. Enable this only when HTTP-first parsing genuinely cannot retrieve the data you need.

---

Running Locally & Deploying

Prerequisites

• Node.js 20+
• Apify CLI: npm install -g apify-cli
• An Apify account and API token for proxy access: apify login

Run locally

# Install dependencies
npm install

# Build (TypeScript -> JavaScript)
npm run build

# Provide your input in ./storage/key_value_stores/default/INPUT.json
# then run the Actor locally
apify run

Local run output is written to ./storage/datasets/default/.

Deploy to the Apify platform

# Push the Actor to your Apify account
apify push

Once deployed, you can run the Actor from the Apify console, via the API, or on a schedule.

---

Legal & Compliance

Please read ./LEGAL_NOTES.md before using this Actor. It covers LinkedIn's Terms of Service, robots.txt, GDPR and personal-data handling, ethical scraping practices, and an important disclaimer. You are solely responsible for ensuring your use complies with all applicable laws and terms.