Douyin Profile & Keyword Scraper

Scrape Douyin video data from:

• Creator profile pages (/user/...)
• Search keywords (/search/...)

This Actor uses a browser interception strategy:

1. Parse first-screen data from RENDER_DATA
2. Intercept Douyin XHR responses while scrolling to capture more videos

Implementation stack:

• Crawlee PlaywrightCrawler (official Apify/Crawlee browser component)
• SessionPool + retries
• Optional proxyConfiguration (Apify Proxy or your own proxy)
• Session warmup + cookie pool binding

Input

{
    "profileUrls": [
        "https://www.douyin.com/user/MS4wLjABAAAAa5_5w7d9T5XUaB_qY9yW1A8y2a-R1hC_4xK8W8X2qGg"
    ],
    "searchKeywords": [
        "AI工具"
    ],
    "maxItemsPerSource": 30,
    "maxScrollRounds": 8,
    "scrollDelayMs": 1800,
    "navigationTimeoutMs": 60000,
    "maxConcurrency": 1,
    "maxRequestRetries": 3,
    "sessionMaxUsageCount": 10,
    "enableWarmup": true,
    "warmupUrls": [
        "https://www.douyin.com/",
        "https://www.douyin.com/hot"
    ],
    "warmupDelayMs": 1200,
    "warmupNavigationTimeoutMs": 45000,
    "headless": true,
    "includeRawAweme": false,
    "forceChinaProxy": true,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ],
        "apifyProxyCountry": "CN"
    },
    "cookie": "msToken=...; ttwid=...; passport_csrf_token=...",
    "cookiePool": [
        "msToken=...; ttwid=...;",
        "msToken=...; ttwid=...;"
    ]
}

Output fields

Each dataset item contains normalized fields, including:

• awemeId, desc, createTimestamp, createTimeIso
• durationMs, playUrl, playUrlNoWatermark
• diggCount, commentCount, collectCount, shareCount
• author object
• music object
• hashtags
• sources (one or multiple source hits for the same video)

Pricing recommendation (PPE)

Code already emits the billing event:

• Event name: item-scraped
• Event count: number of unique awemeId in one run

You can set final PPE unit price in Apify Console.

Notes

• Profile mode usually works with valid public profile links.
• Keyword mode may trigger Douyin verify_check / captcha depending on IP reputation.
• On Apify cloud, use residential CN proxy (proxyConfiguration) whenever possible.
• If captcha still appears, use cookie or cookiePool.
• cookiePool is session-bound (one cookie per crawler session), better than a single cookie for long runs.
• When verification is hit, the run summary will show keyword source failure explicitly.