# Changelog of Facebook Posts Scraper — Page Timeline Export (`devilscrapes/facebook-posts-scraper`) Actor - **URL**: https://apify.com/devilscrapes/facebook-posts-scraper/changelog.md - **Full Actor documentation**: https://apify.com/devilscrapes/facebook-posts-scraper.md ## Changelog ### Deep-run verification of 0.1.3 — 2026-09-17 Same shape as the deep run that shelved 0.1.2 (`page_identifiers: [nasa, nike, microsoft]`, `max_posts_per_page: 40`, RESIDENTIAL), against build 0.1.3 (run `C2lLtRWYndyN8lSpG`, `apify api` POST, not `apify call`): ``` statusMessage: "Done — 120 post(s) from 3/3 Page(s)." per-identifier: nasa=40, nike=40, microsoft=40 (was 3/3/3=9 on 0.1.2) unique post_id count: 120/120 (no duplicate pages) usageTotalUsd: $0.0356 for 120 rows -> ~$0.30/1,000 rows ``` Field fill rates on the same 120 rows: | Field | Fill rate | Note | |---|---|---| | `permalink`, `creation_time`, `author_name`, `author_id`, `author_url`, `comment_count`, `reaction_count` | 100.0% | | | `share_count` | 98.3% (118/120) | 2 posts genuinely show 0 shares | | `text` | 92.5% (111/120) | 9 posts are photo/video-only with no caption — real, not a bug | | `is_cross_post` | 0/120 `True` | still no cross-post in-sample; the actor-id-vs-page-id comparison itself was verified correct against real data in the 0.1.3 fix, not re-derived here | Both blockers are cleared: the ceiling is gone (40/40/40 against a request of 40, not 3/3/3) and the four previously-always-empty fields are now filled at 92-100%. Cost is healthy against the $3.00/1,000-post price (~$0.30/1,000 measured, and 120 rows still amortizes the bootstrap fetch more than a full production run would, so real cost/1k is likely close to or below this figure — `reference-actor-cost-measurement`). **Still not published.** Per the CEO's explicit hold: this fix earns a verified, working product — not a publish decision. Stays on the SHELVED list; `isPublic` untouched. ### 0.1.3 — 2026-09-17 Fixes both blockers the 2026-09-17 deep run found. Neither the doc\_id nor Blocker B (Camoufox) were involved — both bugs were the parser reading fields that don't exist on the real payload, same class as `wwwURL`, one layer deeper. Confirmed against a fresh live capture from `https://www.facebook.com/nasa` (`.venv/bin/python3` + `curl_cffi`, no proxy, no cloud run — see `scripts/recon/live_probe.py`). - **Fix: the cursor ceiling.** `extract_next_cursor` checked `line.get("path")` and expected a string; a real refetch response's `path` is a JSON **array** (e.g. `["node", "timeline_list_feed_units"]`), so `isinstance(path, str)` was always `False` and the function always returned `(None, False)` — capping every scrape at exactly one refetch page regardless of `max_posts_per_page`, identically across every Page (the "exactly 3 per page" ceiling the deep run reported). The page-info chunk is actually identified by its Relay **`label`** (`..._timeline_list_feed_units$page_info`, a string), and the cursor itself is one level deeper than the old code read too: `data["page_info"]["end_cursor"]`, not `data["end_cursor"]`. Fixed both. A local live walk with the fix now advances cleanly for 8 consecutive pages on `/nasa` — 3 new posts every page, `has_next_page` still `true` at the end, matching `research/RECON-facebook-posts-doc-id-2026-09-15.md`'s 25-page depth proof exactly. - **Fix: `text`, `reaction_count`, `share_count`, `comment_count` were always empty/zero.** `_node_to_row` read these from top-level node keys (`message`, `reaction_count`, `share_count`, `comments_count_summary_renderer`) that **do not exist** on a real post node — the actual values live several `comet_sections` layers down: `comet_sections.content.story.message.text` for the post body, and `comet_sections.feedback.story.story_ufi_container.story.feedback_context .feedback_target_with_context.comet_ufi_summary_and_actions_renderer .feedback.{reaction_count,share_count,comment_rendering_instance.comments .total_count}` for the engagement counts. Added `_dig`/`_dig_dict`/ `_dig_str` helpers that walk a key chain and return `None`/`{}` on a miss instead of raising — a missing intermediate key means the field genuinely wasn't rendered for that post, not a parse failure. - **`is_cross_post` was not a bug.** Checked directly against real captured data: the actor-id-vs-page-id comparison is correct as coded. It read `False` for every post in both the deep run's sample and this fix's fresh capture simply because none of the sampled posts happened to be cross-posts (NASA rarely reposts) — grouping it with the other four "always empty" fields in the deep-run note was a sampling artifact, not a fourth bug. Left unchanged. - **Replaced the fabricated NDJSON test fixtures with real, live-captured ones.** `tests/fixtures/ndjson_page_ok.txt` and `ndjson_page_ok_with_benign_leaf_error.txt` were hand-synthesized to the shape the (buggy) code expected — `path` as a dotted string, cursor fields flat under `data` — which is exactly why the unit suite stayed green through both bugs above. They are now two distinct real pages captured from `/nasa` (trimmed of irrelevant video-defer chunks and Relay bookkeeping via `scripts/recon/trim_fixture.py`, never fabricated in shape) with real `post_id`s, real cursors, and the real benign `save_flow` leaf error every live response carries. - **New regression tests**, both of which fail on the pre-fix code: `test_pagination_walks_past_the_first_batch` (asserts a multi-page walk returns rows from **two** distinct real pages, not one fixture served twice) and `test_field_fill_rates_on_a_real_captured_page` (asserts 100% fill rate for `text`/`reaction_count`/`share_count`/`comment_count`/ `permalink`/author fields against a real fixture). - **Widened the QA-exercised depth.** The prior "6-row/2-page QA PASS" was `tests/fixtures/input.qa.json` (`page_identifiers: [nasa, nike]`, `max_posts_per_page: 10`) run against the pre-fix code: 2 identifiers x the 3-row ceiling = 6 rows, comfortably inside a lenient "SUCCEEDED with some rows" pass bar even though it fell far short of the 20 rows asked for. Changed the fixture to a single identifier at `max_posts_per_page: 12` — 4x the natural ~3-row-per-refetch-page batch size, so a reintroduced ceiling now delivers only 25% of what was asked (an unmissable shortfall) instead of a plausible-looking undercount, at roughly the same egress cost (one Page, ~4 refetch pages instead of 2 Pages x 1). Note `scripts/os/qa_sweep.py`'s own `SMOKE_CAP = 3` pins any `max*` input field to 3 for its automated sweep regardless of this fixture or the input schema's own `max_posts_per_page` prefill (already 40\) — that is deliberate cost control on its part, not a gap. The ceiling can only be re-proven broken by a **deliberate** deep run (`apify api` POST with an explicit `max_posts_per_page` well above 3), never by the automated smoke sweep alone. ### 0.1.2 — 2026-09-17 - Fix: the real cloud QA failure after 0.1.1 (build `RkyH07eEQZsfFoGXy`, run `ILtH01v0dsbHhRXAr`) was **not** a doc\_id problem at all — it was the default proxy. `apify api` reproduction through the actual `BUYPROXIES94952` datacenter group returned Facebook's application-level `{"message": "Rate limit exceeded", "severity": "CRITICAL", "code": 1675004}` on the very first refetch POST; the identical request through `RESIDENTIAL` cleared cleanly and returned real posts. Changed the default/prefill `proxy_configuration` (input schema, `ActorInput` model default, QA fixture) from `BUYPROXIES94952` to `RESIDENTIAL`. **Cost note for whoever prices this next:** residential proxy egress is materially more expensive per GB than datacenter — the $3.00/1,000-post price in this README/actor.json was set against a datacenter-cost assumption (`research/RECON-facebook-posts-doc-id-2026-09-15.md`'s \~$0.055/1,000-post estimate) and has NOT been re-checked against residential egress cost. Re-measure with `run_usage.py` per `reference-run-cost-settles-late` before this ships public. - Also cleared the named KVS store `facebook-posts-scraper-doc-id` (`timeline_refetch_doc_id` key), which the pre-0.1.1 failing runs had poisoned with a self-healed doc\_id from the old, over-broad error classifier — a stale cached value would otherwise keep being read first on every future run regardless of code fixes. ### 0.1.1 — 2026-09-17 - Fix: cloud QA (run `siorjFJWZ3B83iN1q`, build `QEbA60EMwdlmc5P01`) failed every Page with `refetch still failing after doc_id self-heal: persisted_query_miss`. Root cause was **not** the doc\_id — the named `29183747561227112` still works — it was `classify_top_level_errors` treating *any* error as a hard failure. Every real refetch response carries one always-present, leaf-scoped, non-fatal GraphQL error for the logged-out "Save post" flow (path ending `story_ufi_container/story/save_info/save_flow`); the old classifier had no way to tell it apart from a real miss, so it self-healed on every single request and then failed the retry for the same reason. Now classified by matching the error's path *suffix* against that always-benign shape; a real miss (confirmed live against a deliberately-invalid doc\_id) carries no `path` at all and is still caught. - Fix: `_node_to_row` read the post permalink from a guessed field name (`wwwURL`) that doesn't exist on a real response — every row was silently dropped as "missing a required field" even after the classifier fix, a `SUCCEEDED`-with-zero-rows result (`reference-success-is-not-delivery`). The real field, confirmed live 2026-09-17, is `permalink_url`. - Fix: bumped the shared HTTP timeout from 30s to 45s (`graphql_client. HTTP_TIMEOUT_S`) — the same cloud QA run also timed out 5/5 on the bootstrap GET for `/nike` at exactly 30000ms with a partial/zero-byte body (curl 28), consistent with a slow first-byte stall rather than a block. Per-`page_identifiers` fault isolation (see 0.1.0) already kept this from taking down the rest of the batch; the longer timeout gives a stalling connection room to complete instead of retrying into the same wall five times. - Verified live against `https://www.facebook.com/nasa` and `https://www.facebook.com/nike` (the exact cloud QA fixture) with `curl_cffi` + the actual `src/` modules: both Pages now return real `post_id`/`creation_time`/`author_name` rows end to end. ### 0.1.0 — 2026-09-17 - Initial build. Exports public Facebook Page timeline posts (text, permalink, timestamp, author, reaction/share/comment counts, and photo/video attachments) via the internal `ProfileCometTimelineFeedRefetchQuery` GraphQL endpoint — bootstrap GET for session/cursor plumbing, NDJSON refetch pagination, `curl-cffi` only, no login and no browser automation. - Self-heals the persisted-query `doc_id` Meta rotates on deploy by re-grepping the Page's own `rsrc.php` JS bundles and caching the result in a named key-value store, so the fix survives across runs. - Per-`page_identifiers` fault isolation: a Page that fails to bootstrap (private/deleted/login wall) is reported on its own and never stops the rest of the batch; a Page that parses but genuinely has no posts succeeds with zero rows instead of raising. - PPE pricing: `actor-start` $0.02 + `post-scraped` $0.003 (~$3.02/1,000 posts). ### Cost check against RESIDENTIAL egress — 2026-09-17 The 0.1.2 note flagged that the **$3.00 / 1,000 posts** price was set against a DATACENTER egress assumption, and that the fix switched the default to RESIDENTIAL. Measured on the QA run (`run_usage.py Wlkmht9vOcgxgDyca`): ``` total: $0.005509 for 6 rows -> ~$0.92 / 1,000 rows PROXY_RESIDENTIAL_TRANSFER_GBYTES $0.004744 (86% of the bill) ACTOR_COMPUTE_UNITS $0.000642 ``` RESIDENTIAL CONFIRMED from billing, not from what we requested — the tier is real, not a silent fallback to direct. **Read this as an upper bound, not the number.** A 6-row run amortises the bootstrap fetch over 6 rows, and our own record is that a tiny QA run inflates cost/1k by up to 50x. So true cost/1k is very likely *below* $0.92 and the $3.00 price has margin — but the honest gate before monetizing is **one ~100-row scale run**, not this one. Also note `reference-run-cost-settles-late`: residential transfer can land on the bill after the run finishes. This figure already shows residential, so it is at least partly settled, but re-read it before pricing off it. **Status: stays PRIVATE until the scale run.** Not blocked on a slot — blocked on a number. ### 2026-09-17 — DEEP RUN OVERRIDES THE "FIXED" VERDICT. Do not publish. The 0.1.2 fix made cloud QA green and that reading was **too generous**. QA passed on 6 rows / 2 pages, which is inside the defect. A deliberate deep run (`eBfE2hUuAEmzh0vr4`, 3 pages, `max_posts_per_page: 40`) exposes two blockers: **1. Depth is capped at 3 posts per page, and the input is ignored.** ``` requested : max_posts_per_page = 40, pages = nasa, nike, microsoft delivered : 9 rows total — EXACTLY 3 per page ``` Asking for 40 and receiving 3, identically on three unrelated pages, is not a thin feed — it is a ceiling. The pagination cursor is not being walked past the first batch. The earlier QA PASS (6 rows / 2 pages) is the *same* 3-per-page ceiling; at n=2 it simply looked like a small fixture. **2. Five fields are ALWAYS EMPTY, including the one the product is named for.** ``` always-empty: text, is_cross_post, reaction_count, share_count, comment_count ``` `text` is the post body. **A "Facebook Posts Scraper" that returns no post text is not a product**, however green its run status is. `post_id`, `permalink`, `creation_time`, `author_name` and `author_id` do populate correctly. This is the same shape as the bug 0.1.2 fixed (`wwwURL` — a guessed field name that does not exist), one layer further in: the row shape was validated against what the *code* expects, never against what the payload actually carries. **Cost, for the record** (`run_usage.py eBfE2hUuAEmzh0vr4`, RESIDENTIAL confirmed from billing): the residential transfer line dominates, and at 9 rows the per-row figure is meaningless. **The scale-run cost question is still open** — it cannot be answered until the depth bug is fixed, because a run that stops at 3 rows/page never moves representative bytes. **Status: PRIVATE. Two hard blockers, not one.** Depth first, then field fill, then price. Not slot-blocked — blocked on being a working product.