RedNote Xiaohongshu Scraper — Posts, Comments, Profiles, Video
Pricing
from $60.00 / 1,000 post scrapeds
Scrape RedNote (Xiaohongshu) posts, comments, profiles and video — six modes in one Actor. Chinese social media and user generated content at the source: Chinese reviews, China consumer opinion, china market research and the creator data behind influencer marketing. No login for posts and profiles.
All notable changes to the RedNote (Xiaohongshu) Scraper Actor.
cookieString and came back with 0 items no longer says "add a cookieString". Measured 3-9 Oct 2026: seven daily searches whose cookieString was read as web_session alone returned 0 rows, and every status told the buyer to add the cookie they had pasted. In search, user_posts, comments and profile the zero-row status of a cookie run now gives as a likely cause that RedNote treated the cookie as logged out, names the cookies the run read (names only, never a value; a paste of more than 4 cookies, or one with no known RedNote name, is given as a count plus which of a1, webId and web_session it held), and asks for the whole Cookie: request header from a fresh login. A cookieString nothing could be read from is reported as such. A cookie search that RedNote answered with its recommendation feed names the cookies too, unless a long searchQuery would push the status past Apify's 500-character cut. Runs without a cookie keep their text byte for byte.cookieString" now copies the whole Cookie: request header. It used to copy web_session alone and say "That's it. One cookie.", which was never verified; the Pro tip that described the whole header is folded into Step 3. The troubleshooting table has a row for a cookie search that returns 0 items.deltaMode and cookieString descriptions and the README said a creator feed is "typically 16-61 posts". A cookieless user_posts run stops at the first ~30 posts the anonymous profile page serves (since 2026-09-21); the texts now say so. The cookieString text's "(the prefill caps it at 40)" went with that sentence: the prefill takes up to 20 posts from each of its two profiles, so 40 caps the prefilled run, not one creator's feed.python -m pytest tests/test_cookie_status_text.py.cookieString now reads the formats buyers actually paste. Only the a=b; c=d header value worked. The README's own DevTools steps copy the Value column of web_session, and that value pasted on its own gave 0 cookies; so did a JSON export from a cookie extension. A leading Cookie: corrupted the first cookie, and one cookie per line became one cookie with a newline in it, which made the browser fail to start. The parser now also reads a leading Cookie:/cookie:, one name=value per line, a JSON array of {name, value} objects (Cookie-Editor, EditThisCookie), and a bare value with no =, taken as web_session. The a=b; c=d format parses exactly as before (offline test against the previous function).web_session, a1, webId...) and counts the others; it never prints a value. It says so when a cookieString gave no cookie at all.= that is not a valid cookie name (a Netscape cookies.txt row, a pasted request header such as x-s: ...) is dropped instead of becoming a cookie: it could carry a value into the log, and a name with a tab or : can stop the browser from starting. A pasted block of request headers now applies the real a1 from its cookie: line. A bare value ending in = padding is read as web_session; name= with nothing after it still gives no cookie.domain is xiaohongshu.com (or that name no domain), so another site's session cookie is never sent to RedNote.python -m pytest tests/test_cookie_parser.py.user_posts): adding or losing a cookieString re-billed every post. Without a cookie RedNote hides the note ID, so a post was remembered by its cover file; with one it was remembered by its note ID. The first run after the switch (cookie added, cookie expired) found every post "new" and billed it again (offline: 10 of 10 posts). A post with a note ID now also remembers its cover file, and either key counts as seen. A memory saved by an earlier build learns the cover files on its first run with a cookie on this build. A cover shared by two different posts in one run (a placeholder image) is never used as a key.deltaStateKey: about 156,000 note IDs. Memories saved by earlier builds load as they are; nothing is reset.deltaMode are unchanged (checked offline against the previous commit). Offline tests: python -m pytest tests/test_delta_recency.py.cookieString, user_posts had one browser attempt per profile, so a navigation timeout (7 of 66 measured launches) delivered 0 rows. An attempt that raised after the browser was up now gets up to 2 more launches on a fresh connection with the same cookie. A logged-in feed that comes back empty or filtered to nothing still costs exactly one launch, and so does a browser that fails to start (the same cookie would fail it again). Cookie launches wait up to 45 s for navigation instead of 25 s. Cookieless runs are unchanged (25 s, same rotation).create_browser_context left the launched Chromium running when creating the context, the stealth script or the cookies raised (measured with a real Chromium: a cookieString pasted one cookie per line makes add_cookies fail). It now closes it and re-raises the same error.run_user_posts now stops at that same instant and returns the posts it has; they are pushed, then billed, like any other rows, and the status says the profile hit the per-profile time limit, or names the run's own time limit when that is what ended it. A pull that already had its maxResults when the budget ran out gets no such note. The budget itself is unchanged.python -m pytest tests/ (64 tests, virtual clock, no network).search runs ("Want only NEW posts next time? Turn on deltaMode + a daily Schedule…") and the README banners/lines that sold deltaMode as the cheap "only new" way. The README's scheduling and delta-mode sections now describe both neutrally, with the same facts (plus one correction: each scheduled run writes to its own dataset — the README said runs append to the same one). The partial-failure status note on a non-delta provider search no longer ends with a deltaMode suggestion, and the README now says a cookieless scheduled search still pays the Actor start fee. Nothing else changed: no default, prefill, request, row, charge or delta behaviour was touched, and deltaMode works and is documented as before.Speed-focused release. Cuts in-actor scrape time for multi-URL modes by 50-80%; cuts single-URL search wall-clock by ~46%. No schema changes; one new optional input field added (networkCapture).
networkCapture input (default true). Listens to xiaohongshu.com API responses during page load and pulls JSON data out of the wire directly, instead of waiting for Vue to hydrate and parsing state. Falls back to v2.0 state/DOM extraction automatically when capture finds nothing. Disable only for debugging.src/response_capture.py. ResponseCapture class — a per-page response listener that buckets JSON payloads by shape (search feed / user notes / note detail / comments / user profile) without hard-coding endpoint paths in user-facing code.post_details, comments, video. Instead of one page per URL, N workers (N = concurrency) each hold one page for an entire chunk of URLs and navigate between them via page.goto(). Saves the ~500-800ms page-create overhead per URL after the first one in each chunk.BrowserPool.page() now yields (page, capture) instead of page. Single-page modes receive both; multi-URL modes get the pool and acquire their own pages internally.wait_for_selector reduced from 15000ms → 8000ms in navigate_and_wait and per-mode waiters. random_delay magnitudes halved across most call sites. wait_for_function for state hydration reduced from 8000ms → 5000ms in post_details, from 10000ms → 6000ms in user_posts and profile.search, user_posts, post_details, comments, video, profile) now consult ResponseCapture first and fall back to Vue state / DOM only when capture finds nothing. The dual-path design preserves v2.0 reliability while making the common case faster.fast_path=N, slow_path=M line so we can audit which path is winning in production.RedNote search uses a single endless-scroll feed; pagination is JS-driven via signed API calls. Constructing parallel &page=N URLs against /search_result returns page 1 every time — the param is ignored. Replicating the JS pagination would require the same signed-request reverse engineering that v2.x forbids. The practical concurrency win for high-volume search workloads is multiple distinct queries in separate runs. Note documented in src/modes/search.py docstring.
The in-page JS is what (a) populates window.__INITIAL_STATE__, (b) triggers the data-bearing API requests our v2.1 capture listens for, and (c) responds to anti-bot fingerprint probes. Blocking resource_type == "script" leaves a static HTML shell with no extractable data through any path. Classifying "essential vs non-essential" scripts ahead of time requires the forbidden RE. liteMode already saves on the post-extraction formatting work and skips the comment scroll loop; that's the realistic ceiling without breaking extraction. Note documented in src/resource_blocker.py.
| Mode | v1.0.28 | v2.0 rc1 | v2.1 rc2 | Δ vs v2.0 | Target | Hit? |
|---|---|---|---|---|---|---|
| search 10 items duration | 39.1s | 58.9s | 31.9s | −46% | ≤ 20s | ❌ (closer, but Apify container startup is ~12-15s floor) |
| post_details per URL @ conc=3 | 60s | 28.7s | 5.7s | −80% | ≤ 18s | ✅ beat by 3× |
| search 10 items cost | $0.0132 | $0.0115 | $0.0108 | −6% | ≤ $0.008 | ❌ near-flat (the 30-50% target is gated by container time, not in-actor work) |
| post_details cost per URL @ conc=3 | $0.020 | $0.0097 | $0.0019 | −80% | ≥ 30% cheaper | ✅ beat by 2.7× |
| Search fast-path firing? | n/a | n/a | ✅ fast_path=1, slow_path=0, items=10 | new | n/a | ✅ |
The single-URL targets (search ≤20s, search cost ≤$0.008) were not hit because ~12-15s of the 31.9s wall-clock is Apify's container pull + Python+Playwright init, which the Actor code cannot influence. The realistic ceiling for in-actor-only search time is now ~17s (31.9 − 15 container) vs v2.0's ~44s, a ~60% reduction in actual scraping work. The multi-URL targets (post_details per URL) blew past the goal because chunking + same-page navigation + fast-path stack their wins.
| Mode | Run | Status | Note |
|---|---|---|---|
| search (full, 10 items) | UWM2U5P1TSGzlegM5 | ✅ 10/10, 19-field schema | fast_path=1 |
| search (lite, 5 items) | 6wbIfJsuW6K2Q8irS | ✅ 5/5, exact 8-field set | fast_path=1 |
| search (lite, 5 items, retry) | d0jxYRSkYcCc1crWG | ⚠ 0 items | IP-rotation variance — same input on parallel runs got 5 items elsewhere; unchanged from v2.0 known behavior |
| post_details 3 URLs conc=3 | iVtvQL5zFftKRwD3a | ✅ 3/3 real records, titles populated | Phase B chunking |
| comments | m9rwajgaugRe8ZCgk | ✅ Full-schema record, errorCode=None | — |
| video | H1tzaPVvh6R8sjw0i | ✅ videoUrl extracted, hasVideo=true | — |
| profile | GI14TJS0hkMxRCCBA | ✅ 18-field diagnostic, errorCode=login_required | Schema parity with v2.0; underlying gating unchanged |
| user_posts | kVz88U5mywmw1otoq | ⚠ 0 items | Parity with v2.0 known limitation for gated users |
5 of 6 modes returning useful data on the rc2 baseline. 1 mode (user_posts) consistent with v1.0.28/v2.0 limitation. 0 regressions detected.
latest tag — 1.0.28 (production buyers, untouched)rc1 tag — 2.0.1 (v2.0 baseline, available as rollback target if v2.1 ever flips to latest)rc2 tag — 2.1.2 (this release)Promotion to latest requires explicit greenlight after rc2 bake period.
Performance-focused release. Cuts per-item runtime and bandwidth at no schema cost. Output schema unchanged for existing fields; three new optional input fields added (concurrency, blockResources, liteMode).
concurrency input (default 4, max 10). post_details, comments, and video modes now process input URLs in parallel via a semaphore-bounded asyncio task group. Single-page modes (search, user_posts, profile) ignore the setting since they're inherently single-page.blockResources input (default true). Network-layer filter that aborts requests for images, fonts, media, and known tracker/analytics domains. Extracted data is unchanged — media URLs are still emitted in the output, only the bytes are skipped on the wire. Roughly halves bandwidth per page.liteMode input (default false). Returns a minimal field set per item: for posts {mode, postId, postUrl, xsecToken, type, title, likes, scrapedAt}; for profiles a similarly trimmed subset. The xsec_token in postUrl is preserved so search → post_details pipelines still work on lite output. Skips the scroll-load-more loop in comments mode and the DOM-walk fallbacks in video mode.BrowserPool (src/browser_pool.py). One browser + one shared context per run; pages acquired through an async with pool.page() context manager bounded by the concurrency semaphore. Pages are closed automatically after each unit of work.src/resource_blocker.py). Mode-aware: video mode opts into allow_media=True since some player implementations require the media network request to expose <video>.src.main.py now drives all modes through the BrowserPool rather than allocating a single shared page.post_details.py, comments.py, video.py rewritten around asyncio.gather over per-URL tasks. Each task acquires its own page from the pool, processes one URL, releases. No shared state between tasks.format_post_data / format_profile_data / format_comment_data accept a lite_mode kwarg.cookieString input behavior. Premium-mode extraction still routes through context.add_cookies.RESIDENTIAL). RedNote blocks datacenter IPs.xsec_token propagation through search/user_posts → post_details/comments/video (the v1.0.28 fix).Original v2.0 spec included a stretch goal to use httpx for paginated requests after Playwright bootstrapped the session, targeting 70-90% cost reduction on search and user_posts. This phase was not pursued in v2.0 for two reasons:
x-s, x-t, signed payload) generated by client-side JS. Producing those signatures from httpx requires either (a) running the signing JS inside a Playwright context per request — no cost savings, just relocates the work — or (b) replicating the signature algorithm in Python.Net: Phase 4 is structurally infeasible under the v2.0 constraints. Reconsider if RedNote ever offers an unsigned public API endpoint.
Search "护肤教程" — 10 items (same input both runs)
| Metric | v1.0.28 (V3QBqmrWFRCnK0adv) | v2.0 (kArYMFGWMjA1OfkFL) | Δ |
|---|---|---|---|
| Memory peak | 1215 MB | 562 MB | −54% |
| Memory avg | 891 MB | 281 MB | −68% |
| Network RX | 6.51 MB | 4.42 MB | −32% |
| Run cost | $0.0132 | $0.0115 | −13% |
| Duration | 39.1s | 58.9s | +51% (within single-sample noise) |
| Items returned | 10 | 10 | — |
post_details — 3 URLs (v2.0 in parallel with concurrency=3 vs v1.0.28 baseline of 1 URL)
| Metric | v1.0.28 (PmvIfvCopAMp2fdKp, 1 URL) | v2.0 (h2lhMwzsHtSLg5xKP, 3 URLs) | Δ per URL |
|---|---|---|---|
| Duration | 60s / 1 URL = 60s/URL | 86s / 3 URLs = 28.7s/URL | −52% time/URL |
| Cost | $0.020 / URL | $0.029 / 3 = $0.0097 / URL | −52% cost/URL |
| Items with full schema | 1 | 3 | same |
The concurrency win is the big delta — multi-URL modes scale near-linearly with concurrency up to the proxy/rate-limit ceiling. Memory ceiling is also dramatically lower, leaving headroom for higher concurrency on the same memory bucket.
| Mode | Run | Status |
|---|---|---|
| search | kArYMFGWMjA1OfkFL | ✅ 10/10 real records, full schema |
| user_posts | zvIDhr46DM8f0BFdB | ⚠ 0 records — known limitation for gated users (unchanged from v1.0.28) |
| post_details | h2lhMwzsHtSLg5xKP | ✅ 3/3 real records, full schema |
| comments | eJzf2pti7p0QMnzUR | ✅ Real comments extracted |
| video | AkQdF9SqjpM3D8hwl | ✅ videoUrl extracted, hasVideo=true |
| profile | aj7eGyqD8l4srUJuM | ✅ Diagnostic record with errorCode (anonymous gated, unchanged) |
| search liteMode | u3JLgJG29kl9rLfa0 | ✅ 5 records, exactly 8 fields: {mode, postId, postUrl, xsecToken, type, title, likes, scrapedAt} |
5 of 6 modes returning useful data, 1 returning diagnostic record consistent with v1.0.28. No regression.
v1.0.28 remains tagged latest until v2.0 (currently on rc1) is promoted. Rollback from v2.0 = re-tag 1.0.28 as latest via Apify console build selector. No dual code paths to maintain.
Honest error categorization + app-share URL expectation-setting.
errorCode: "login_required" (RedNote's anti-bot serves a TCP-level abort instead of redirecting to a login page; the buyer-actionable fix is the same).Critical bug fixes for post_details / comments / profile modes.
xsec_token now propagated through search and user_posts output (postUrl + dedicated xsecToken field). Previously stripped despite being captured in Vue state. Unblocks the canonical search → post_details pipeline.userPageData, userInfo, profile, otherInfo, currentUser in order.errorCode instead of empty rows; PPE never charged on failures."Scraped X items, Y failed" instead of misleading totals.See Apify console build history for earlier changes.