LinkedIn Posts Scraper
Pricing
from $1.75 / 1,000 post scrapeds
Searches public LinkedIn posts by keyword and returns post text, author, exact publication time, engagement counts, media and comments. No login, no cookies, no session tokens.
Pricing
from $1.75 / 1,000 post scrapeds
Rating
5.0
(1)
Developer
coding laporte
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
2 days ago
Last modified
Categories
Share
All notable changes to this Actor are documented here. The format follows Keep a Changelog , and this project adheres to Semantic Versioning .
postedLimit
was applied afterwards, from the activity id. For a topic with a long tail
that ranking is the posts that have gathered the most links — the oldest —
so a month-limited run for hopital americain de paris found ten links,
dropped all ten as out of window, and reported that discovery had
succeeded. The window is now passed to each engine's own date filter as
well (DuckDuckGo df, Bing's epoch-day range, Mojeek since, Google
tbs), so the result page holds posts the run can use. The engine
restriction is never narrower than the window — presets are used only when
they span it whole, exact ranges are padded by a day at each end — so the
activity-id check remains the one that decides.s=0, which
no browser ever sends; on the platform DuckDuckGo's challenge met that page
on every run while page two got through on most. The offset is now omitted
on the first page, and search requests no longer overlay hand-written
Chrome-style Accept headers on the HTTP client's Firefox impersonation —
the mismatch is what a bot check keys on. With both, DuckDuckGo's first page
came back with results on the platform.clean to be of
type boolean". The output schema linked the dataset as
…/items?clean=true, and Console hands a template's query string to its
dataset viewer as strings, where clean must be a boolean. The link is now
the documented bare …/items; the viewer applies its own options.author, postedAt,
engagement) while also flattening them, and the platform filters columns
after flattening, when those names no longer exist. The lists now name the
flattened columns the views display.site: honoured, a working past-day/week/month filter and
no challenge page — everything Bing itself now withholds from a client that
is not a full browser, which answers site:linkedin.com/posts with general
web results. Seven results a page, so scrapePages buys less of it.google remains selectable; a run that selects
it is told once, in the log, when that wall is served.hopital americain de paris did not match a post body that says
"l’Hôpital Américain de Paris". Discovery was finding those posts and the
run was paying to fetch them, then discarding every one and reporting an
empty result set — the worst shape a failure can take, since it costs full
price and looks like "there are no such posts". Matching now folds accents,
ligatures and typographic punctuation on both sides, so either spelling
finds the other and a curly apostrophe or quote reads as its ASCII form.
Whole-word anchoring and phrase order are unchanged: hopital still does
not match hospital. This affected every non-English query — French,
Spanish, German and Portuguese most of all.Monetization readiness. Every change here is on the money path: what gets charged, what does not, and what a run is allowed to spend.
maxComments: 0 returned one comment. The per-post limit was applied
after the comment was appended rather than before, so a run asking for no
comments got one anyway — and because the charged event is selected by
whether the record carries comments at all, that single comment billed
every post of the run at the with-comments rate.maxPosts
budget and a warning naming the likely cause.searchEngines without naming authors or post URLs left nothing to crawl; the
run charged actor-start, crawled nothing and exited SUCCEEDED. It is now
rejected during input validation, before the start charge. targetUrls is
resolved rather than counted, so a list holding only unusable entries no
longer passes the check..comment node — a throttled response, or markup that never
hydrated — produced an entry whose every field was null, which was enough to
select the with-comments event. Contentless entries are now dropped before
they are counted, so a real comment further down the list keeps its slot.maxPosts: 10 still
enqueued 5,000 proxied fetches. Both routes now share one budget, spent on
author pages first because they do not depend on a search engine.crawler.run() purges the request queue by default, and Apify's queue client
cannot purge — it drops and recreates the queue, changing its id under the
running crawler, so the next round died with Request Queue was not found.
A second round happens whenever the keyword filter rejects enough posts for
the budget to top itself up, which is the normal case. Found by a real
platform run; it cannot reproduce locally, because the memory queue client
purges cleanly.401, 403 and 429 as a session block, checks for that before
it checks the status code, and exempts session errors from
max_request_retries entirely — so a single candidate that LinkedIn was
refusing burned up to ten proxied fetches, the default max_session_rotations,
and then reached the failed-request handler, which released the slot and
charged nothing. Observed in a real run: 19 rotations across 9 discovery
requests, two of them failing after exactly nine. Rotations are now bounded,
on the reasoning that a host which has refused several fresh sessions in a row
will refuse the next one too.post-filtered, once,
through the same accounting as every other terminal outcome.actor-start; a
caller capping maxTotalChargeUsd at the $0.01 platform minimum therefore got
a full discovery crawl for $0.005, on a healthy run rather than a pathological
one. Nothing read maxTotalChargeUsd at all. The cap is now priced against the
discovery the run has actually planned, before the start charge, and a run that
cannot be attempted for the money it is allowed to charge fails immediately
with the figure it would need — about $0.0115 for a default single-query run,
$0.0805 at the 200-page discovery ceiling. Refusing to start is cheaper for
everyone than a scrape delivered at a loss.maxPosts
alone credited a blocked, candidate-less discovery with a hundred fetches'
worth of future earnings, so the guard never fired on the exact run it was
written for. What discovery has actually found is now a ceiling on what
extraction can be projected to earn.maxPosts, a
larger cap became a larger licence to keep losing. Measured: at
maxPosts: 10,000 the guard missed tripping by four cents on a run that
ended $1.247 down. Attempts still in flight are subtracted first, so a
healthy run at full concurrency is not mistaken for a blocked one.sortBy: date counts as earned. A matched post is
held until the round ends, so for the length of a round the guard saw
attempts climbing against nothing — and killed runs that were $0.036 in the
black at the moment it killed them.if is_pay_per_event branch, so
the one case it could not catch was the platform reporting no pay-per-event
pricing at all — the run then did the entire job and charged nothing, looking
healthy throughout. That case now fails loudly, on the same reasoning as the
unpriced-event check: failing the first run is strictly cheaper than
delivering every run free.pytest could not collect the suite. Nothing put the repo root on
sys.path, so the bare pytest that CI runs failed at import while
python -m pytest succeeded — meaning the new CI would have been red on its
first run and every gate protecting the money path skipped.Startup verification of pay-per-event pricing. Charging an event the Actor has no configured price for is silently free: the SDK falls back to a price of zero, logs one line, and still reports the charge as successful — so a typo or an incomplete monetization tab would deliver every paid record for nothing while looking like a healthy run. All four event names are now checked against the platform's pricing before the first charge — a post event priced at zero counts as missing, since giving the product away cannot be deliberate — and a run fails loudly rather than scraping for free.
A ceiling on discovery spend. Search discovery fans out as queries ×
authors × engines × pages, and maxPosts capped only the results, never that
product — 100 queries with 10 authors at 20 pages each meant 80,000 proxied
search fetches for a run that could return at most 100 posts. The number of
search-result pages is now capped in proportion to maxPosts, truncating
round-robin so the budget is spread across engines and queries instead of
being spent entirely on the first of each. Proportional turned out not to be
the same as affordable, so an absolute ceiling of 200 discovery pages now
binds on top of it: discovery earns nothing at all, and tying the cap to
maxPosts alone let a large maxPosts authorise a crawl no plausible
extraction could repay — precisely the run the margin guard then has to kill
halfway through. searchQueries is capped at 10 entries in the input schema,
author lists at 50 and targetUrls at 200.
Input caps that bound a retry storm. maxConcurrency's schema maximum
drops from 20 to 10: max_tasks_per_minute is maxConcurrency * 20, so the
old ceiling let a caller quadruple the rate at which blocked requests were
retried against a host that was already refusing them — more 429s, not more
throughput. maxPosts: 0 still means "no limit" so existing callers keep
working, but it no longer resolves to the 10,000-post maximum: it resolves to
1,000. Nobody typing 0 has costed the run out, and the old mapping handed
that one keystroke a 5,000-page discovery crawl — $2.50 of unrecoverable proxy
spend against the $0.004 net that actor-start pays. A caller who genuinely
wants ten thousand can still type ten thousand, and having typed it has
accepted the bill.
Continuous integration (.github/workflows/ci.yml): lint, formatting, the
offline suite and a real image build on every push and pull request. Nothing
pinned the resolved apify/crawlee version inside the allowed range before.
A populated fields schema in .actor/dataset_schema.json, describing every
column the dataset views reference.
Search-engine result page fixtures for all four engines, covering each one's click-tracking wrapper end to end — including Bing's base64 payload and the visible-citation fallback used when that payload will not decode. Nothing exercised a whole result page before. They are hand-built to each engine's documented format, not captured live; a real capture would be worth more.
A fourth chargeable event, post-filtered ($0.001). Charged for a post
page that was fetched and examined but not delivered — rejected by the keyword
filter, or returned by LinkedIn without a post in it. The proxied fetch is the
expensive part of a run and costs the same whether the post turns out to
match, and a rejected post frees its slot to pull another candidate in behind
it, so a query matching nothing was previously the cheapest run to ask for and
the most expensive to serve: unbounded fetches against a single $0.005 start
charge. Measured, a hundred-post run matching nothing cost $0.92 and earned
$0.004. It now earns $1.20 against the same cost.
One fetch is billed, and freed, at most once. ResultBudget.release is
now keyed by activity ID and idempotent. A handler that raises is retried by
Crawlee and, on exhausting its retries, also reaches the failed-request
handler — so a single transient error while charging could release one post's
slot several times, and every freed-but-never-reserved slot let the run claim
more work than maxPosts allowed. Reproduced before the fix: four charges and
four proxied fetches for one post, and a run delivering four posts on a
maxPosts of three.
A delivered post's slot can no longer be freed. release consulted only
the rejected set, so a post that was written and charged and whose request
then failed — a handler that delivers and is cancelled or times out still
reaches the failed-request handler — had its slot handed back. The freed slot
pulled another candidate in, and the run delivered and billed past maxPosts.
Reproduced against a real crawler: maxPosts: 2 producing six rows and six
post-scraped charges, while the summary read "Scraped 0 post(s)". The same
bug counted one post in two mutually exclusive tallies.
A refused delivery returns its slot. Two discovered permalinks can
canonicalise to one post — a repost resolves to the original — so a duplicate
is reachable without any retry. It previously consumed a slot and was never
delivered, so the run stopped short of maxPosts with candidates queued. It
is not charged: a duplicate is this Actor's own discovery counted twice.
A post is written to the dataset at most once. The same retry hazard
existed on the delivery side and was worse there: push_data writes the item
and only then charges, so a failure at the charging step left the row in place,
escaped the handler, and had Crawlee re-fetch, re-write and re-charge — at
post-scraped, the dearest event. Reproduced against a real crawler: four
fetches, four duplicate rows and four charges for one post. Delivery is now
claimed before the write and refused on a repeat, and neither the handler path
nor the date-sorted flush can raise out of a push.
A charging failure can no longer fail the fetch. Charging goes over the network and can fail for reasons that have nothing to do with the post. Left to propagate it made Crawlee retry the whole request, and each retry re-ran the charging path — turning one transient error into several paid fetches. Losing a single charge is the cheaper failure, and it is logged.
A ceiling on post fetches, at five times maxPosts. The counterpart to the
event above: because rejected posts free their slots, extraction would
otherwise drain the whole candidate pool, and a narrow keyword could turn a run
sized at a hundred posts into a bill for a thousand fetches. Five leaves a
query matching one post in five room to fill the run, and caps what a caller
can be surprised by. The run logs why it stopped and how to search more
narrowly.
post-scraped is $0.00175 ($1.75 per 1,000 posts) and
post-scraped-with-comments is $0.003, post-filtered $0.001.
actor-start stays at $0.005 and is documented as covering the discovery
phase, which runs before any post is fetched. The README's comparison figures
were wrong and have been corrected.maxPosts,
not just 0. Documentation that overstates a spending ceiling is worse than
none, because it is the number a customer sizes a run against. Both files are
now derived from the constants, the four worked pricing rows recomputed against
the live event prices, and the two unfilled placeholders — the session-rotation
cap and the minimum viable maxTotalChargeUsd — filled in. The margin guard is
described as what it is: a stop, not a guarantee, with the run's normal
successful terminal state on one side and an outright pre-flight failure on the
other, which the previous copy ran together.main() against a local server, which is what
found the margin guard's projection and warm-up gaps.First public release.
AND, OR, NOT, quoted phrases, parentheses) evaluated against every
post's text — so a returned post really does mention what you asked for./in/…) or company page
(/company/…) of any author you name. LinkedIn-native and independent of
any third party.linkedin.com/posts. This is what makes an open keyword search possible;
LinkedIn's own post search is behind the login wall.targetUrls.scrapeComments) with text, author identity, exact
timestamp and per-comment like count, merged from the JSON-LD island and the
rendered markup because neither carries all of it.postedLimit and postedLimitDate are applied before a post is fetched —
an out-of-window post costs nothing — and sortBy: date orders the run rather
than just the output.maxPosts cap enforced through a reservation-based budget that releases a
slot when a post fails the keyword filter, so filtered-out posts do not
consume the cap. Extraction runs in rounds and tops up from the remaining
candidates, so a run whose first maxPosts candidates all fail the filter
still returns results instead of an empty dataset.actor-start, post-scraped,
post-scraped-with-comments. The run stops cleanly at a charge limit.RESIDENTIAL group.respectRobotsTxt input, off by default, documenting the conflict between
LinkedIn's robots.txt and this Actor's function.null, so CSV and Excel exports keep one shape.shares is
always null rather than guessed.