Juejin Scraper

Scrapes 稀土掘金 (Juejin, ByteDance's developer community) — without login — via the public api.juejin.cn endpoints.

Data source

Articles and users come from Juejin's public web APIs (juejin.cn) — no login, no API key. The search mode is geo-restricted to China residential IPs (documented below); use recommended, byCategory or byUser. See the Notes section for further caveats.

What it does

Mode	Endpoint (public, no auth)	Output
recommended (default)	POST recommend_api/v1/article/recommend_all_feed	Recommended feed
search	GET search_api/v1/search?query=&id_type=2&cursor=&limit=	Paginated article search
byCategory	POST recommend_api/v1/article/recommend_cate_feed	Per-category articles (8 categories)
byUser	POST content_api/v1/article/query_list	All public articles by user id

All endpoints verified live (HTTP 200 err_no: 0 with real content) from a datacenter network, no auth headers needed.

Search geo-restriction: Juejin's search_api/v1/search returns empty result pages (HTTP 200, data: []) from non-China datacenter IPs — verified against 8 request variants from Apify infrastructure. It works from China-based residential IPs (verified from the authoring network). The actor fail-softs with a clear status message (and a typed EMPTY_ARTICLES error record if the API returns items that cannot be parsed as articles). Use recommended / byCategory / byUser modes instead — they are not geo-restricted and are the actor's default.

Tag feed is also geo-gated: recommend_api/v1/article/recommend_tag_feed returns err_no: 0 with an empty data from non-CN datacenter IPs (verified with real tag ids), and the tag-list / article-detail routes (tag/recommend_tag_list, content_api/v1/article/detail) are unreachable from non-CN IPs — so byTag and byArticleUrl modes are documented as out of scope rather than shipped as zero-record modes.

Output fields

article records (recordType: "article") — articleId, title, briefContent, viewCount, diggCount, collectCount, commentCount, hotIndex, rankIndex, authorId, authorName, authorAvatarUrl, authorJobTitle, authorCompany, authorLevel, tags[], tagIds[], categoryId, category, coverImage, isOriginal, isHot, isEnglish, ctime (unix sec), mtime, rtime, readTime, contentCount, originalType, linkUrl, sourceUrl, scrapedAt.

error records (recordType: "error") — a typed EMPTY_ARTICLES record when the search API returns items that cannot be parsed as articles. Invalid user ids and unknown categories fail soft with a clear status message and 0 records — nothing is silently dropped.

Every data record carries sourceUrl, scrapedAt and recordType; error records carry inputValue in place of sourceUrl (the offending input, not a page URL). Null/empty fields are stripped before push.

Rehosted media

Juejin article cover images (*.byteimg.com with x-expires/x-signature) are signed and expire after ~8 days — a customer would hit 403 on a stored URL. Covers are therefore rehosted into the actor's key-value store at scrape time: coverImage becomes a permanent Apify KV URL and coverImageOriginal keeps the upstream signed URL for traceability. Author avatars (passport.byteacctimg.com) are unsigned and passed through unchanged.

The rehosted KV URL is signed by the actor itself using the platform's base62(hmac_sha256(secret, key)[:30hex]) scheme (some SDK builds emit truncated signatures that the platform rejects with 403 from a clean shell) and is verified to resolve 200 image/* anonymously.

Filters

• minViewCount, minDiggCount, minCommentCount
• containsKeyword (title substring)

Notes

• All feeds paginate via the API's opaque cursor token; has_more drives the stop condition.
• Category list was refreshed in this cycle: Juejin migrated its category id space — the previous 12-entry enum had 10 stale ids that returned empty feeds. The current enum has 8 categories verified to return real data (前端, 后端, 人工智能, Android, iOS, 运维, 代码人生, 阅读); 大数据/测试/云计算/开发工具 no longer resolve to a working recommend_cate_feed id and were dropped.
• recommend_cate_feed requires both cate_id and category_id plus sort_type: 3 to return data; sort types 1 (hot) and 2 (latest) return empty pages from non-CN IPs, so the sortBy input only exposes the working 3 (recommended) option.
• Auto-escalation: on 403/429 the actor lazily engages the Apify AUTO (datacenter) proxy with exponential backoff.
• Memory: 1024 MB (Tier 1, plain HTTP).