# Changelog of lemon8-scraper (`teodor_banea/lemon8-scraper`) Actor - **URL**: https://apify.com/teodor\_banea/lemon8-scraper/changelog.md - **Full Actor documentation**: https://apify.com/teodor\_banea/lemon8-scraper.md ## Changelog ### \[1.4.0] - 2026-08-31 #### Added - **Pagination (feed + profile):** the Actor now pages through category feeds (`method=stream-loadmore`) and full creator post histories (`method=user-post-loadmore`) using Lemon8's own cursor mechanism (`max_behot_time`), up to `maxResults`. Previously every source returned a single page (~10-30 posts). Page ceilings (`MAX_FEED_PAGES = 50`, `MAX_PROFILE_PAGES = 100`) and an empty-page streak guard bound each chain. Posts are deduplicated by `groupId` (the profile loadmore cursor is inclusive — each page repeats the previous page's last post; the For You feed recycles items). - **Automatic proxy escalation:** proxies are now fully managed via Crawlee tiered proxies — datacenter first, automatic escalation to RESIDENTIAL (pinned to the `region` input's country) after blocked attempts, with periodic de-escalation probes. The user-facing `proxyConfiguration` input was removed. - **Block detection:** HTTP 403/429 and soft blocks (HTTP 200 with a data-less body) now throw `SessionError` — retiring the session, retrying on a fresh IP, and feeding the proxy-tier tracker. Previously a soft block was recorded as a successful empty run. Session rotation budget: 5 (`maxSessionRotations`); transport retry budget stays 3 (`maxRequestRetries`, hardcoded, never user input). - **Rich profile records:** the `profiles` output now includes follower count, following count, total likes, post count, and structured external links, extracted from the inline `$UserDetailV2` payload (previously always null). - **Zero-result signaling:** a run that produced 0 posts because of blocks/failures now fails with a clear status message instead of exiting green with an empty dataset. - Cost estimate log line for runs asking for more than 1,000 results; price sourced from one `PRICE_PER_RESULT` constant. - `output_schema.json` and pinned memory (1024 MB) / timeout (3600 s) in the Actor manifest. #### Changed - **Compression enabled** (`decompress: true` instead of `Accept-Encoding: identity`) — cuts proxy bandwidth ~5-10×. - `tt_webid` cookie is now cached per session (rotates with the session) instead of regenerating per request. - **Profile records moved to the run's key-value store** (`PROFILES` key, deduped by username). The `profiles` named dataset never worked on the platform (LIMITED\_PERMISSIONS returns HTTP 403 for named datasets), and platform QA correlated the failed named-dataset open with stalled default-dataset pushes afterwards — so there is now one sink for local and platform runs alike. - **Lemon8's `text/plain` 429 responses are now accepted MIME types** on both crawlers. Platform QA showed the MIME allowlist aborting 429 bodies *before* `retryOnBlocked` could classify them as blocks, which silently bypassed session rotation and proxy escalation entirely. - **Dedup marking is transactional:** posts are marked as seen only after the dataset push succeeds, so a handler that times out mid-push delivers the page on retry instead of discarding it as duplicates. Discover and post modes now dedup by `groupId` too. - The final status message notes permanently failed requests, so partial deliveries (e.g. 90 of 100) are visible at a glance. - **Every early stop is disclosed:** a feed category that runs out of posts logs "category exhausted" with pages fetched and delivered/requested counts (previously a `maxResults: 0` run could stop at 149/500 silently); a fully-scraped profile logs that its history is complete. - **Sparse discover pages retry once, then disclose:** when Lemon8 serves a discover page without embedded data, the Actor retries once on a fresh session (which can escalate the proxy tier); if still sparse it keeps the basic results (title/likes/url) and logs exactly which fields will be null, instead of degrading silently. - Run logs no longer print internal API URLs (query strings are redacted from all request logging); zero-result failures distinguish "URLs not found — check your input" from genuine blocking, and only mention proxy escalation when it happened. - Crawlee's periodic status messages are suppressed in favor of the Actor's own "Scraped N posts" messages. - Post-mode prefill switched to a live post (`@herrealsoul`'s profile listing began returning 404). - **Price raised from $0.002 to $0.0025 per result** alongside the capability release (full-history pagination, enriched profiles, managed proxies). Billing model switches to pay-per-event: actor start + per default-dataset item, charged automatically by the platform. - `maxResults` schema maximum lowered from 100,000 to a realistic 10,000; `region` moved to the top of the input form (it now also selects the proxy country). - ESLint 8 → ESLint 9 flat config; `cheerio` and `@vitest/coverage-v8` declared as devDependencies; versions synced (actor 1.4 / package 1.4.0). - User-Agent refreshed to Chrome 140. #### Removed - `includeComments` input — comment text is only available in Lemon8's mobile app (authenticated API); the web payloads never contain it. `topComments` stays in the output (always `[]`) for shape stability. ### \[1.0.5] - 2026-03-24 #### Added - **Proxy warning logs:** All route handlers now log a clear, actionable warning when a request fails or returns no results, advising users to enable residential proxies (`apifyProxyGroups: ["RESIDENTIAL"]`) and set `apifyProxyCountry` to match the target region (e.g. `"JP"`, `"GB"`). This guidance appears directly in the Apify Console log so users know exactly what setting to change. ### \[1.0.4] - 2026-03-24 #### Fixed - **Post mode `likes`/`saves` null:** Post detail suspense data uses `diggCount`/`favoriteCount` (same as discover), not `articleLikes`/`collectCount`. Both field names now checked as fallbacks. - **Post mode raw HTML in `content`:** Added `stripHtml()` to clean rich-text HTML from the Lemon8 post editor into plain text. - **Post mode `topComments` empty:** Now checks both `topComments` and `comments` keys. Comment text also HTML-stripped. - **Post mode 35+ junk images:** DOM fallback now only selects `wap_large_logo_image` URLs (post's actual slide images). - **Post detail suspense wrapper:** Handles `{article: {...}, author: {...}}` wrapper structure correctly. ### \[1.0.3] - 2026-03-24 #### Fixed - **Discover mode (critical):** Posts were always `null`/empty because Lemon8 now delivers data via Remix suspense-replace scripts (`