LinkedIn Open Profile Status: Identify LinkedIn profiles with open profile status and collect available public details, including names, job titles, companies, locations, profile URLs, and professional information for B2B prospecting, lead generation, and sales research.
User asked for the internal proxy/status fields to be dropped from the output.
Removed proxy_mode, the debug object, the nested data wrapper, and the
internal message string (e.g. "Failed to fetch profile. Last=http:status=999
...") from every dataset row and from OUTPUT - those were implementation
details useful during this fix, not something an end user needs. Each row is now
just {"url": ..., "open_to_work": true|false|null}. Updated the dataset's
default table view in .actor/actor.json to match (dropped the now-gone
"Message"/"Proxy mode" columns). All internal logging (proxy tier, block
reasons, warm-up status, etc.) is unchanged - still fully visible in the run log
for debugging, just no longer duplicated into the data output.
[0.9] - 2026-08-30
Residential proxy now targets US, matching the config that was actually verified to work
Even after fixing session-id rotation (0.8), a full 10-attempt residential run
still got HTTP 999 on every single attempt. Comparing against this session's own
successful raw-proxy test (the one that captured a genuine, complete real profile
page) turned up a real difference: that test explicitly requested
country-US, while the actor's own
never specified a country at all, pulling from Apify's full
international residential pool. Added country_code="US" so the actor's proxy
config actually matches the one configuration verified live to work.
[0.8] - 2026-08-30
Fixed a real bug: residential "retries" were reusing the same sticky IP every time
Two 0.7 live verification calls both came back fully blocked (HTTP 999 on every
tier). Investigating why the residential tier - which direct raw-proxy testing had
already proven can get a clean IP - never succeeded in an actual run surfaced a
real bug: _process_one fetched one residential proxy session_idbefore the
retry loop and reused that same proxy_url for every attempt inside it. An Apify
Proxy session-<ID> is specifically designed to pin to the same outbound IP for
its whole lifetime (that's the point of a "session") - so every "retry" was
silently re-requesting through the exact same, already-blocked IP, never actually
trying a different one. With an observed ~1-in-9 clean-IP rate, retrying the same
IP 3 (or even 10) times could never have worked.
Fix (src/main.py, _process_one): fetch a fresh proxy session
(f"{session_id}_{attempt}") inside the residential retry loop itself, so each
attempt gets a genuinely different sticky IP from Apify's pool. Direct/datacenter
modes are unaffected (they only ever make one attempt, so this bug didn't apply to
them). Also raised RESIDENTIAL_ATTEMPTS from 3 to 10 (see the constant's comment)
now that each retry actually has a chance of landing a clean IP.
[0.7] - 2026-08-30
Fixed: a genuinely loaded profile with no badge was reported "unknown" instead of a real False
Live verification of 0.6 (DIRECT mode, no proxy) got a 200, not caught by any
block signal, but _extract_open_to_work found no badge and returned None -
the actor reported "unknown" even though nothing was actually wrong. The
function never distinguished "this is the real, fully-loaded profile page and it
genuinely has no badge" (a confirmed False) from "this might be LinkedIn's
stripped SEO-only preview page, which returns HTTP 200 too but is not the full
profile and could never show the badge either way" (a real None/unknown).
Fix (src/main.py, _extract_open_to_work): added a length check
(_REAL_PROFILE_MIN_LEN = 50_000) before falling back to False. Genuine full
profile pages captured live this session (Fernando Mora, Gregory F. Liverpool,
Anthony J James) were all 768 KB-1.1 MB; the hard-block stub is ~1.5 KB - a
50 KB floor sits safely between the two without needing to reverse-engineer
every possible stub/teaser shape LinkedIn might serve. Below that size, "no
badge found" still correctly falls back to None (honest uncertainty), matching
this actor's standing policy of never fabricating a confident answer.
[0.6] - 2026-08-30
Fixed a false-positive that was discarding genuinely successful fetches
After 0.5 removed Playwright, live verification showed 0.5 could still get through
LinkedIn's per-IP block on a lucky residential session - direct testing against the
raw residential proxy (9 sessions, bypassing the actor) captured one real, complete
profile page (768 KB, real title
Anthony J James - Trinity Consulting Services | LinkedIn
). But _is_blocked_html's bare "captcha" substring check matched a
normal LinkedIn A/B-test feature-flag attribute present on that same real, working
page - data-recaptcha-v3-integration-lix-value="control" - and the actor
discarded its own successful result as "blocked". Removed the bare "captcha"
check; the other five phrases (/checkpoint/, "verify you are a human", "unusual
activity", "access denied", "too many requests") are specific enough to real
challenge pages that they don't share this problem.
Known limitation (unchanged)
This account's shared proxy pools (direct/datacenter egress and the residential
pool) are still frequently pre-blocked (HTTP 999) by LinkedIn's per-IP
reputation system - only a fraction of residential sessions get an IP clean enough
to reach the real page at all (observed roughly 1 in 9 in direct testing). This fix
means the actor no longer wastes a lucky clean session's real result by
misclassifying it as blocked; it does not change how often a clean session comes up
in the first place.
[0.5] - 2026-08-30
Removed Playwright entirely - it was never the right tool for this site
User reported this was still failing and, separately, far too slow, with a live
failure showing Playwright's own proxy tunnel breaking outright:
Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED. Re-investigated from scratch using
a real Chrome browser's own DevTools-level network requests (not assumption, not a
repeat of the 0.4 Playwright test) against two real profiles with the genuine
"Open to work" feature enabled:
A request with zero LinkedIn cookies (credentials: 'omit') got an instant
HTTP 999 block - confirming the anti-bot gate is real.
The exact same URL, same client, with only the ordinary anonymous cookies
LinkedIn sets on any normal page visit (bcookie, bscookie, JSESSIONID,
lidc, lang, timezone, li_theme, visit, liap, li_sugr, fid,
sdui_ver) - no login, no li_at cookie anywhere - returned a genuine
HTTP 200 with the full, real, ~1 MB server-rendered profile HTML.
This is the same class of fix already proven on this account's Reddit actors: the
site doesn't hard-block anonymous automated HTTP requests, it blocks requests that
skip the ordinary cookie warm-up a real browser always does first. It has nothing
to do with authentication (li_at) and nothing to do with needing a JS engine -
the profile page itself is plain server-rendered HTML once the right (still
anonymous) cookies exist. Playwright was solving a problem that didn't require a
browser at all, while adding real cost: slow container/browser startup on every
request, and its own separate proxy-tunnel failure mode in production.
Fix (src/main.py):
Removed the Playwright strategy (_try_playwright, _playwright_proxy_from_url)
and the playwright dependency entirely.
Added _fetch_with_warmup: one aiohttp.ClientSession (shared cookie jar) does a
GET https://www.linkedin.com/ warm-up request first, then the real profile GET
reusing the cookies the warm-up acquired - mirroring exactly what was verified
live above. If a li_at cookie is supplied it's still sent (may help on IPs
already flagged for anonymous traffic), but it is not required.
Removed the old blind fallback
if re.search(r"\bopen to work\b", html):return True
in _extract_open_to_work. It produced a real false positive: a profile
that simply typed "... @ Open To Work | ..." into their own headline sentence
(verified live) does NOT have the actual platform feature enabled and would have
been wrongly reported as open_to_work=True. Replaced with a check for the
actual markup LinkedIn renders for the real badge - a standalone
<strong>Open to work</strong> tag - confirmed present on real "Open to work"
profiles and absent on the headline-text false-positive case.
pageTimeoutSeconds input renamed in spirit to requestTimeoutSeconds (old key
still read for backward compatibility) since there's no browser page anymore,
just an HTTP request timeout.
Dockerfile / requirements.txt: switched back from
apify/actor-python-playwright to the plain apify/actor-python:3.13 base image -
smaller image, faster cold start, no browser/Xvfb machinery at all.
Known limitation (unchanged from 0.4)
LinkedIn's per-IP reputation/rate-limiting on this account's shared proxy pools
remains a separate, real gate independent of the above - a request can still get
blocked purely because the outbound IP is already flagged, regardless of cookies.
The plain-HTTP rewrite removes the Playwright-specific failure modes and the
false "needs li_at" framing; it does not by itself guarantee every IP this
account's proxy ladder lands on is unflagged.
[0.4] - 2026-08-30
Correction to the 0.3 conclusion about li_at, based on new live evidence
0.3 shipped a fix for a real false-negative bug, but its accompanying message
(profile_not_loaded:no_li_at_cookie_provided) overclaimed the cause. A user
pushed back that li_at should not be required, which prompted a fresh,
from-scratch live re-investigation (not a repeat of the same single test):
A real headless Playwright/Chromium browser (no li_at, realistic UA and
viewport) was launched against 3 real public LinkedIn profiles, including
two independently verified via search-engine indexing to have "Open to
Work" literally in their profile headline
(linkedin.com/in/fernando-mora-29a487262,
linkedin.com/in/gregoryfliverpool-putthehbackinhr). All three came back
with HTTP status 999 and a ~1.5 KB "authwall" stub (og:url: /authwall,
title Sign Up | LinkedIn) - an instant hard block, not a graceful
degraded page.
Critically, a plain, non-browser, unauthenticated HTTP GET (no cookie, no
proxy, only a User-Agent header - i.e. exactly what this actor's HTTP
strategy does) to https://www.linkedin.com/in/ajjames from a clean
residential IP did succeed earlier in this investigation with a genuine
HTTP 200 and a full, real 768 KB profile page. The exact same URL from the
exact same machine, after roughly half an hour and about seven more
automated requests, then returned the same HTTP 999 block. This proves
li_at is not what gates access - LinkedIn's public profile view is
reachable by unauthenticated requests, at least from a clean, low-volume
IP; the block is a per-IP reputation/rate-limit response, not a hard
authentication wall.
Every request routed through this Apify account's actual proxy pools
(BUYPROXIES94952 datacenter group, StaticUS3 static IPs), including one
with a full realistic browser header set (Accept, Accept-Language,
sec-ch-ua*, Sec-Fetch-*, etc.) added specifically to test whether bare
headers were the problem, got an instant HTTP 999 on the very first
request to each pool - even against profile URLs never touched before in
this investigation. This points at LinkedIn having already blacklisted
these specific shared/commercial proxy IP ranges wholesale, independent of
headers, cookies, or the specific profile.
This Apify account's RESIDENTIAL proxy group reports availableCount: 0
(checked via GET /v2/users/me) - meaning the "residential" tier this
actor's proxy ladder falls back to has no real inventory on this account,
even though the 0.3 live-verification run did show real
PROXY_RESIDENTIAL_TRANSFER_GBYTES usage (requests did go out through it,
they just weren't meaningfully different/cleaner IPs).
Fixed
Removed the misleading no_li_at_cookie_provided causal claim.src/main.py, _try_playwright: when the real profile view fails to
render, the result now reports
profile_not_loaded_possibly_blocked_or_rate_limited:<li_at_cookie_present|no_li_at_cookie>
describing the observed fact (whether a cookie was supplied) without
asserting that its absence caused the failure, since the evidence above
shows it frequently does not.
Added a real browser header set to the HTTP strategy
(src/main.py, new _browser_like_headers(), used by _http_get):
Accept, Accept-Language, Accept-Encoding, Upgrade-Insecure-Requests,
Sec-Fetch-*, and sec-ch-ua*. The previous code sent only a bare
User-Agent (plus the cookie, if any), which is itself a strong,
independent bot fingerprint regardless of IP or cookie. This did not
overcome the already-blacklisted proxy pools tested above, but is a
correct, low-risk improvement that matters most on IPs that are not
already flagged (e.g. an actor run's own DIRECT egress IP).
Reduced Playwright's automation fingerprint (src/main.py,
_try_playwright): launch Chromium with
--disable-blink-features=AutomationControlled and patch
navigator.webdriver to undefined via an init script. Bare headless
Chromium is one of the most heavily fingerprinted automation signatures
LinkedIn's bot detection checks for.
Fixed a real correctness gap in the Playwright block-detection
(src/main.py, _try_playwright): the code checked page.url for
/checkpoint/, /login, challenge substrings to detect a block, but
never inspected the actual HTTP status of the navigation response. Since
LinkedIn's HTTP 999 block is served at the same URL with no redirect,
a 999-blocked page previously fell straight through to the badge-detection
logic without ever being flagged as blocked=True, which could silently
skip proxy-ladder escalation for that specific request. The navigation
response status is now checked against the same BLOCK_STATUS_CODES set
the HTTP strategy already uses.
Updated .actor/actor.json (liAtCookie input description) and
README.md to stop implying the cookie is the primary fix, and to state
plainly that the dominant real-world failure mode is LinkedIn's per-IP
reputation/rate-limiting, not the absence of a cookie. liAtCookie was
already optional (not in the schema's required list) - it remains
optional; no functional flag needed to be "removed".
Known limitation (unchanged, but now correctly explained)
This actor's success rate is fundamentally gated by the reputation of the
outbound IP making the request at the time of the request, which this actor
cannot control beyond the DIRECT -> DATACENTER -> RESIDENTIAL proxy ladder it
already implements. Shared commercial proxy pools (datacenter and often
"residential" pools sold at scale) are frequently pre-blocked by LinkedIn
wholesale. A genuinely fresh, low-volume residential IP can succeed for a
period before also being rate-limited. Supplying a valid, still-active
li_at session cookie remains a legitimate way to improve reliability
(an authenticated session is not subject to the anonymous/public-view gate),
but it is optional, not required, and its absence should not be read as an
explanation on its own.
[0.3] - 2026-08-30
Root cause (the actual, verified reason every run hung and timed out)
After fixing the missing requirements.txt (see 0.2 below) the image built
and the actor could finally be started, but every live run still failed:
it ran to its full configured timeout (tested at both 180s and 90s) and
produced zero application log lines and an essentially flat, near-zero
CPU/memory profile (memMaxBytes ~20 MB) for the entire run.
Two live runs were made to isolate this:
Run etdaY72KZSJokFpoX: real input, startUrls: ["https://www.linkedin.com/in/ajjames"].
Run zQO1QjJDDJFIgMag9: diagnostic input, startUrls: ["https://example.com"]
(a trivially reachable, unrelated URL - chosen specifically so this test
does not touch LinkedIn at all).
Both hung identically to their timeout with no output whatsoever. Since the
second run never touches LinkedIn or anything remotely related to this
actor's scraping logic, this ruled out the HTTP/Playwright scraping code,
the proxy ladder, and LinkedIn's anti-bot behavior as the cause - Python
never even executed its very first log line (
Actor.log.info("Starting actor...")
, which fires immediately after Actor.init()) in either run.
The base image (apify/actor-python-playwright) sets
ENTRYPOINT ["./xvfb-entrypoint.sh"], i.e. every command is wrapped in
xvfb-run to provide a virtual X display for headful browser use. This
actor shipped its own copy of xvfb-entrypoint.sh and relied on that
inherited ENTRYPOINT. On this platform/image, the xvfb-run display
bring-up itself hung indefinitely at container start - before Python ever
ran - which explains the total silence and the flat, idle-looking resource
usage (a stuck shell/X11 bring-up process uses almost nothing).
Critically, this actor's own Playwright usage
(_try_playwright in src/main.py) launches Chromium with
headless=True exclusively and never uses a real or virtual display for
anything. Xvfb was pure, unnecessary overhead for this actor, and on this
image/platform combination it was actively fatal.
Fixed
Removed the Xvfb entrypoint wrapper entirely.Dockerfile now sets
ENTRYPOINT [] and runs CMD ["python3", "-m", "src"] directly, so the
container starts Python immediately with no xvfb-run bring-up step in
the way. xvfb-entrypoint.sh was deleted from the actor source since it
is no longer referenced anywhere.
Verified live after this fix, in two runs on build 0.3.1:
Run asajdznvuugrY9f9F (startUrls: ["https://example.com"],
diagnostic, no LinkedIn contact): container booted immediately, all
actor log lines appeared within ~2.5s of container start, and the run
SUCCEEDED end-to-end in ~13s.
Run Z4MIc8cbgtsyN7SVT (startUrls: ["https://www.linkedin.com/in/ajjames"],
the actor's own example profile, no li_at cookie supplied): the
container booted, and the actor genuinely ran its full DIRECT ->
DATACENTER -> RESIDENTIAL (3 retries) proxy ladder against LinkedIn,
finishing SUCCEEDED in ~96s. Every tier was blocked or returned no
signal (expected without a valid session cookie). The dataset row
correctly recorded open_to_work: null with
message: "Failed to fetch profile. Last=playwright:profile_not_loaded:no_li_at_cookie_provided"
proving both that the container/network/proxy pipeline now works
end-to-end, and that the pay-per-event / false-negative fix above is
live: the actor honestly reports "could not determine" rather than a
fabricated false.
[0.2] - 2026-08-30
Root cause
The actor could not run at all. The Dockerfile contained:
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
but no requirements.txt file existed anywhere in the actor source (only
.actor/actor.json, Dockerfile, README.md, src/__main__.py,
src/main.py, and xvfb-entrypoint.sh were present). This made the Docker
COPY instruction fail on every build with a "file not found in build
context" error, so the image could never be built and no run could ever
start. This was the actual, confirmed root cause of the actor being broken
not a LinkedIn anti-bot change.
While investigating, a second, independent bug was found and fixed in the
Playwright fallback strategy (see below): it produced confidently wrong
results (silent false negatives) on every unauthenticated run, which would
have made the actor look "working" but return garbage data even once the
build was fixed.
Fixed
Added the missing requirements.txt (apify>=4.0.0,<5.0.0,
playwright>=1.62.0,<1.63.0, aiohttp>=3.9.0,<4.0.0), pinned to match the
Playwright version already bundled in the base image so pip does not
install a Python playwright package whose expected browser binaries
differ from what the image ships.
Pinned the Docker base image from the floating apify/actor-python-playwright:latest
tag to the exact, verified tag apify/actor-python-playwright:3.12-1.62.0,
so the build is reproducible and does not silently pick up a future,
untested base image.
Fixed a silent false-negative bug in the Playwright "open to work" badge
check (src/main.py, _try_playwright). As of 2026, LinkedIn serves an
unauthenticated visitor a stripped-down public "preview" of a profile
(title/headline/OG meta tags for SEO only - confirmed live: no
window.__INITIAL_DATA__, no profile section markup, no Open-to-work
badge markup at all) instead of the real profile page, regardless of
whether that profile actually has the "Open to work" badge enabled. The
section.pv-top-card selector only ever renders on the real, authenticated
profile view. Previously, when that selector timed out (which happens on
every run without a valid li_at session cookie), the code fell straight
into the badge-detection selectors with open_to_work defaulted to
False, then reported the result as a confirmed "ok" /
open_to_work=False. In practice this meant every profile checked without
a real, live li_at cookie was reported as "not open to work" - a
confident-looking but false answer. The fix detects when the real profile
never loaded and reports the result as open_to_work=None with a
profile_not_loaded:no_li_at_cookie_provided (or
..._full_profile_did_not_load when a cookie was supplied but still
didn't work) detail, which the existing proxy-ladder logic already surfaces
honestly as "Failed to fetch profile" rather than a wrong "false" verdict.
Verified live (root cause confirmation)
A single, non-repeated, unauthenticated curl request to
https://www.linkedin.com/in/ajjames (the actor's own example/prefill
profile) was made to confirm the anti-bot theory before writing any code
(per the mandated methodology - not guessed). It returned HTTP 200 with a
768 KB HTML page containing only a <title> tag, one application/ld+json
block, and "Sign in" / "Join now" prompts - no window.__INITIAL_DATA__, no
openToWork field, and no "open to work" text anywhere in the page. This
confirms that, without a valid li_at session cookie, LinkedIn's public
profile response cannot yield a genuine Open-to-work verdict either via the
HTTP+HTML strategy or via the Playwright badge-detection strategy.
Known limitation (by design, not a bug)
This actor's liAtCookie input is optional but, per the above, is
functionally required for a genuine result. Without a valid li_at
cookie the actor now honestly reports "could not determine" (
open_to_work:null
, message: "Failed to fetch profile. Last=...") instead of a
fabricated false. This matches the existing input schema description
("Recommended to reduce blocks and improve reliability") but should be
understood as effectively mandatory for real data.
