π·οΈ Facebook Hashtag Scraper pulls relevant hashtag data from Facebook for faster trend research and content planning. π Boost reach, discover audience interests, and optimize campaignsβperfect for marketers, creators, and analysts.
0.3 β The actual root cause, confirmed with real cookies
Live-tested 0.2 with real, user-provided c_user/xs cookies: got exactly
the previously-flagged-as-a-risk error, now surfaced loudly instead of
silently (proof the 0.2 error-visibility fixes work) β
missing_required_variable_value on every request.
Root cause, confirmed by direct evidence: captured the real request
Facebook's own browser JS sends for facebook.com/hashtag/nyc via Playwright
network interception, using the same real cookies. It's the same
SearchCometResultsPaginatedResultsQuery doc_id our code discovers β but
its variables payload carries 28 __relay_internal__pv__* Relay
"provided variables" (feature-gate flags the query component declares as
required) that the old build_variables() never sent at all. Relay's
server-side resolver rejects the request outright when any declared
provided-variable is missing β exactly the generic, unhelpful
missing_required_variable_value error observed, and the reason this
"looked like 0 results" is that this same error class is indistinguishable
from other failures until specifically surfaced (which 0.2 already did).
Fix (src/main.py): added _RELAY_PROVIDED_VARIABLES, the full set of
28 flags with the exact values captured from a real session, spread into
build_variables()'s output. Also corrected fetch_filters from True to
False to match the real client (the old value was never verified against
anything live).
0.2 β Fix "0 results on every run" (static analysis, no live cookies used)
This pass diagnosed why the actor was returning zero posts on every run. No
Facebook cookies were used for this pass β everything below was verified by
static analysis, offline unit checks with synthetic response data, and
python -m py_compile, per the constraint of not burning real c_user/xs
cookies on test requests. A live cookie-authenticated run is still required
to confirm the fix actually restores results end to end.
Fixed with confidence (concrete bugs, verified without live cookies)
Silently swallowed GraphQL errors reported as "no more posts."src/main.py (graphql_page) and src/parser.py (new
extract_graphql_errors). Facebook's GraphQL endpoint can return HTTP 200
with valid JSON that still failed the query β a stale doc_id, a revoked
session, a malformed variable β via the standard
{"data": null, "errors": [...]} shape. The old code only ever read
data.serpResponse... and treated the resulting KeyError the same as an
empty page, so a hard GraphQL error and "you've reached the end of the
results" were indistinguishable and both logged as
β No more posts available. Now graphql_page checks for a top-level
errors array and raises a descriptive RuntimeError when no posts were
extracted, so the real Facebook error message reaches the log instead of
being reported as success.
Bare except Exception: blocked = True; break swallowed the actual error
with zero logging.src/main.py, pagination loop in scrape_keyword. Any
failure while fetching a page (network error, JSON decode failure, the new
GraphQL-error RuntimeError above) was caught and discarded without even a
log line, so a run could return 0 posts with literally nothing in the logs
explaining why. Now the exception message is logged via Actor.log.warning
before the retry/escalation logic runs.
Unhandled JSONDecodeError if Facebook's response body isn't valid
JSON.src/main.py (graphql_page). resp.text.splitlines() was fed
straight into json.loads with no error handling; if Facebook ever answers
with an HTML error/interstitial page instead of the expected NDJSON body,
this raised an opaque JSONDecodeError with no context. It's now caught and
re-raised as a RuntimeError that includes the HTTP status code and the
first 300 characters of the actual response body, so the operator can see
what Facebook actually sent back.
doc_id extraction regex was pinned to a single hardcoded minified
variable name.src/helper.py (find_doc_id). The old pattern required
the literal text a.exports="<digits>" β but the letter a minifier assigns
to the exports parameter is not stable across Facebook builds/deploys.
Generalized to accept any valid JS identifier before .exports= and added
re.DOTALL so a module body split across lines still matches. Verified
offline against synthetic minified snippets using both a.exports= and a
different variable letter β the old pattern only matched the former, the
new one matches both.
get_edges / get_story / page_info extraction were pinned to a single
exact JSON path with no fallback, per the classic "parse tolerantly"
failure mode: if Facebook's response wrapper is ever renamed/renested (a
cosmetic change, not a breaking one from Facebook's point of view), the
pinned chunk["data"]["serpResponse"]["results"]["edges"] lookup throws
KeyError, is caught, and silently returns [] β indistinguishable from a
hashtag genuinely having no posts. Added shape-based fallbacks
(_find_edges_list, _find_parent_with(edge, "comet_sections"), and
parser.find_page_info) that locate the same data anywhere in the response
by structural signature when the pinned path doesn't match. The primary
pinned path is tried first and unchanged; the fallback only activates when
it fails, so this cannot regress currently-working extraction. Verified
offline with synthetic reshaped responses.
Missing fb_api_caller_class field in the GraphQL POST body.src/main.py (graphql_page). Every real-browser capture of a Facebook
doc_id-based /api/graphql/ request includes
fb_api_caller_class: "RelayModern", which is what tells the endpoint to
route the request through the persisted-query/Relay resolver rather than
the generic AJAX handler. Its absence is a plausible root cause of the
"always returns 0 posts" symptom: a request missing this field can be
rejected/mishandled before it ever reaches the search resolver, and (prior
to the error-handling fixes above) that failure was invisible. Added, along
with the standard server_timestamps: "true" field and a computed
jazoest checksum ("2" + sum(ord(c) for c in fb_dtsg), Facebook's
well-documented anti-bot checksum derived deterministically from fb_dtsg
β not a guessed/unrelated value).
Still needs a live cookie-authenticated test to confirm
Whether the fixes above actually restore non-zero results. All of the
above were reasoned and verified without live Facebook traffic; only a real
run with valid c_user/xs can confirm the request now succeeds end to
end.
The exact JSON shape of a live SearchCometResultsPaginatedResultsQuery
response β i.e. whether data.serpResponse.results.edges /
rendering_strategy.view_model.click_model.story is still the correct path
today, or whether the new shape-based fallback is the one actually doing
the work. The fallback was added specifically to survive this being wrong
without needing to know the real answer up front.
Whether https://www.facebook.com/search/top?q=<term> (kept as-is, with
the # left in the query text) actually returns hashtag-specific results
equivalent to browsing facebook.com/hashtag/<tag>, versus generic keyword
search results. This is a design/endpoint-choice question, not a code bug,
and needs a real logged-in search to compare.
Whether fb_api_caller_class / jazoest / server_timestamps are
strictly required by this endpoint, versus merely present in typical
browser traffic. They are low-risk, standard additions, but only a live
request confirms they were the actual missing piece.
The literal _facebookRelayOperation module-name marker and the
"<digits>" assignment shape used by find_doc_id β this matches the
pattern used by other public Facebook-scraping reverse-engineering
write-ups, but Facebook's actual current bundle format has not been
fetched/inspected in this pass (no requests were made).
Whether the response body ever carries a for (;;); JSON-hijacking
prefix on this specific endpoint (used historically by some Facebook AJAX
endpoints). If it does, json.loads on the first line will fail via the
now-handled JSONDecodeError path β meaning it will at least fail loudly
and visibly instead of silently, but would need a small strip-prefix fix to
actually work.
Whether extra required headers (e.g. x-asbd-id) exist for this specific
query β seen in some other Facebook internal-API traffic, but not added
here because there was no live capture to confirm it applies to this
endpoint/query.
0.1 β Initial release
π·οΈ Scrape public Facebook posts by one or more hashtags / search terms (bulk input).
π― Configurable max posts per hashtag with automatic pagination.
π Auto-escalating connection ladder: direct β datacenter β residential (sticky), residential retried up to 3Γ.
πΎ Live, per-post saving to the dataset.
π Two dataset table views: Posts Overview and Engagement Breakdown.
