Bilibili Video Transcript & Metadata Extractor

📹 Bilibili Video Transcript & Metadata Extractor

---

💡 Common Use Cases & Applications

This actor is designed to enable various data workflows. Some of the primary use cases include:

1. LLM Summarization & QA Platforms: Extract transcripts from Bilibili educational videos, tutorials, or lectures and feed them into LLMs (like GPT, Claude, or local LLMs) for bullet-point summaries, key takeaway extraction, or building Q&A systems.
2. Content Archiving & SEO Indexing: Save video scripts, uploader details, metadata statistics, and direct CDN URLs to local databases for media archiving, search engine indexing, or historical tracking of engagement metrics.
3. Language Learning & Translation: Leverage dual-track subtitles and speech-to-text outputs to build bilingual corpora, generate transcripts for regional accents, or validate machine translation models against spoken audio.
4. Competitor & Content Trend Analytics: Scrape statistics such as views, likes, danmaku counts, coins, and replies over time to track audience engagement, analyze creators' content strategies, and discover trending topics.

---

1. Introduction and Overview 🌟

Welcome to the official developer documentation for the Bilibili Video Transcript & Metadata Extractor! 📹

Bilibili is one of the most prominent video-sharing platforms in China, famous for its interactive video playback experience, bullet-chat (Danmaku) culture, and an extensive repository of user-generated content ranging from tech tutorials, anime, gaming guides, to lifestyle vlogs. As the platform grows, the demand for extracting textual content from Bilibili videos has increased exponentially. Whether you are building an LLM-based summarization application, training translation models, archiving educational video content, or performing deep visual-textual analytics, accessing high-quality subtitles and metadata is a crucial prerequisite.

The Bilibili Video Transcript & Metadata Extractor is a high-performance, containerized scraper built as an Apify Actor. It provides a robust, dual-strategy extraction pipeline designed to work under production loads:

• Primary Strategy (API Scraping): It queries Bilibili's internal endpoints to retrieve official developer-uploaded or auto-generated subtitle tracks (e.g., in Simplified Chinese zh-Hans, Traditional Chinese zh-Hant, etc.). This process is incredibly fast, returning results in milliseconds.
• Secondary Strategy (Whisper ASR Fallback): If no subtitles are available on the video, the actor triggers a local speech-to-text pipeline. It downloads the highest-quality audio track using yt-dlp, dynamically locates the ffmpeg binary via imageio-ffmpeg to process the audio stream, loads OpenAI’s Whisper tiny neural network model, and transcribes the audio locally within the container.

In addition to transcripts, the actor gathers comprehensive metadata about the video, including view count, likes, shares, danmaku count, coins, replies, favorites, publish times, high-resolution thumbnail URLs, video author (uploader) profile information (avatars and names), and direct CDN URLs for streaming audio and video.

By combining direct HTTP requests, proxy rotation (via Apify Proxy), and on-device Whisper machine learning, this actor delivers a highly resilient scraping solution that handles rate-limiting and missing data natively.

---

2. Key Features and Workflows 🛠️

The Bilibili Video Transcript & Metadata Extractor is designed to handle complex edge cases and provide a smooth, reliable extraction pipeline. Below is a detailed breakdown of its core features and internal workflows.

graph TD
    A[Start Actor] --> B[Parse Input URL & proxyConfiguration]
    B --> C[Extract BVID from URL]
    C --> D[Fetch Video Metadata from Bilibili API]
    D --> E{Metadata Fetch Success?}
    E -- No --> F[Log Error & Terminate]
    E -- Yes --> G[Extract Stream URLs using yt-dlp]
    G --> H{Are Official Subtitles Available?}
    H -- Yes --> I[Download Subtitle JSON & Parse Segments]
    H -- No --> J[Download Best Audio Stream with yt-dlp]
    J --> K[Locate & Configure ffmpeg PATH]
    K --> L[Load Whisper tiny Model]
    L --> M[Transcribe Audio locally & generate Segments]
    I --> N[Combine Metadata, Stream URLs, and Subtitles/Segments]
    M --> N
    N --> O[Push Result JSON to Apify Dataset]
    O --> P[End Actor]

2.1 Comprehensive Metadata Extraction 📊

The actor scrapes a wide array of metadata fields directly from the Bilibili platform. Unlike simple scrapers that only extract the title, our tool accesses deep statistic arrays:

• BVID, AID, CID Resolution: It resolves the modern alphanumeric Bilibili Video ID (BVID), the legacy integer Archive ID (AID), and the specific Part/Page ID (CID). This makes the data fully compatible with legacy APIs and third-party tools.
• Interaction Statistics: It captures the complete popularity metrics—views, danmaku (bullet chat count), comments (replies), favorites, virtual coins (inserted by fans), shares, and likes. These metrics are critical for analytics and virality modeling.
• Uploader (Owner) Profiles: Retrieves the uploader's user name (nickname) and their avatar URI (avatarUri), allowing you to map content back to specific creators.
• Video Details: Extracts the high-res cover image (img), publication/creation timestamp (createTime), raw description text (desc), title, and duration in seconds.

2.2 Official Subtitle Fetching API 📝

Bilibili supports bilingual and creator-provided subtitles.

1. The actor inspects the subtitle array returned by the Bilibili view API.
2. It filters for preferred languages, prioritizing Chinese Simplified (zh-Hans / zh-CN) or falling back to the first available language track.
3. Once the subtitle URL is retrieved, the actor issues an authenticated GET request to download Bilibili's native JSON subtitle format.
4. It loops through the subtitle timeline body, converting the time formats into seconds and building a clean array of objects containing start, end, and text segments.

2.3 Local Whisper Automatic Speech Recognition (ASR) Fallback 🤖

If the uploader did not submit subtitles and Bilibili's automatic translation isn't available, the actor falls back to local machine learning.

1. yt-dlp Extraction: The actor isolates the audio track from the Bilibili page and downloads it directly to a temporary directory.
2. FFmpeg Pipeline: Whisper requires ffmpeg to load and decode audio files. The actor uses imageio-ffmpeg to locate the platform-specific ffmpeg executable. If the host machine doesn't have ffmpeg installed globally, it copies the executable to a temporary binary folder and dynamically inserts it into the system's PATH environment variable.
3. Whisper Model Loader: The actor loads OpenAI’s whisper model. By default, the tiny model is selected. It provides a fantastic balance of speed, low memory footprint (perfect for serverless containers with 1-2GB RAM), and acceptable transcription quality.
4. Local Inference: The audio file is transcribed. Whisper automatically detects the spoken language, segments the audio based on silences, and translates/transcribes the voice into structured segments containing timestamps.

2.4 Streaming URL Extraction 🎬

For offline archiving, media player integrations, or secondary audio analyses, the actor uses a background yt-dlp context to fetch the direct streaming URLs of the raw video and audio files from Bilibili’s content delivery network (CDN). Because these links are direct CDN URLs, they are highly time-sensitive and are meant to be consumed immediately upon execution.

2.5 Proxy Support and Anti-Blocking Measures 🔐

Bilibili employs strict rate-limiting on its public APIs, returning status code 412 (Precondition Failed) or IP bans if multiple requests originate from the same IP address. The actor is fully integrated with Apify Proxy, allowing you to pass residential proxy parameters. This rotates the request IP for every execution, ensuring high reliability under heavy load.

---

3. Input and Output Specification 📥📤

[!IMPORTANT] This section outlines the structural formats of inputs accepted by the actor and the JSON schema returned upon successful execution.

3.1 Input Parameters Configuration

The actor accepts JSON inputs containing the Bilibili URL to process and proxy configuration.

Field Name	Type	Required	Default Value	Description
videoUrl	String	Yes	https://www.bilibili.com/video/BV1ojn8z1EUq	The full HTTP URL of the Bilibili video to scrape. Supports standard formats (e.g. https://www.bilibili.com/video/BV...).
proxyConfiguration	Object	No	{"useApifyProxy": true}	Proxy configuration object. Residential proxies are highly recommended to prevent rate limits and Bilibili 412 errors.

Input JSON Example

{
  "videoUrl": "https://www.bilibili.com/video/BV1ojn8z1EUq",
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}

---

3.2 Output Data Schema

The Bilibili Video Transcript & Metadata Extractor outputs results to the Apify dataset as a JSON array. Below is the list of fields returned in the output object:

Field Name	Type	Description
url	String	The original target URL processed by the scraper.
bvid	String	The unique Bilibili Video ID alphanumeric identifier (starts with BV).
aid	Integer	The legacy Archive ID (AV number numerical representation).
cid	Integer	The internal Part/Page ID of the specific video file.
title	String	The official title of the video.
desc	String	The uploader's video description text.
createTime	Integer	Unix epoch timestamp indicating when the video was published.
img	String	URL pointing to the video's cover image / thumbnail.
view	Integer	Total view count at the time of scraping.
danmaku	Integer	Total number of bullet comments (danmaku) sent by viewers.
reply	Integer	The number of root-level and nested replies in the comment section.
favorite	Integer	The count of users who saved the video to their personal favorites folders.
coin	Integer	The total number of virtual coins donated to the video creator.
share	Integer	The number of times the video has been shared across social media.
like	Integer	Total count of positive feedback (likes) received.
duration	Integer	The total duration of the video in seconds.
nickname	String	The display name of the video creator / uploader.
avatarUri	String	The URL profile image of the video creator / uploader.
videoUrl	String	A temporary direct CDN video stream URL (no audio).
audioUrl	String	A temporary direct CDN audio stream URL (no video).
text	String	The complete, flattened transcription or subtitle text.
segments	Array	An array of time-aligned transcript segments containing timestamps.

Each element inside the segments array has the following structure:

• start (Float): The starting timestamp in seconds.
• end (Float): The ending timestamp in seconds.
• text (String): The text spoken or displayed during this time frame.

Output JSON Example

Below is an example of the structured JSON data returned by the scraper.

[
  {
    "url": "https://www.bilibili.com/video/BV1ojn8z1EUq",
    "bvid": "BV1ojn8z1EUq",
    "aid": 1130541764,
    "cid": 1658421045,
    "title": "Bilibili Web API Development Tutorial and Best Practices",
    "desc": "In this guide, we dive deep into scraping, processing, and analyzing video feeds from modern platform systems. Source code and guidelines are shared.",
    "createTime": 1716124800,
    "img": "https://i0.hdslb.com/bfs/archive/a1b2c3d4e5f6g7h8i9j0.jpg",
    "view": 128400,
    "danmaku": 1540,
    "reply": 842,
    "favorite": 5301,
    "coin": 2180,
    "share": 750,
    "like": 10240,
    "duration": 345,
    "nickname": "TechDeveloperCN",
    "avatarUri": "https://i0.hdslb.com/bfs/face/member_avatar.jpg",
    "videoUrl": "https://upos-sz-mirrorcos.bilivideo.com/upgcxcode/45/10/1658421045/1658421045-1-30080.m4s?e=ig8epXgD61F6hwdhiSgVhwd17WdHhwdV7W...",
    "audioUrl": "https://upos-sz-mirrorcos.bilivideo.com/upgcxcode/45/10/1658421045/1658421045-1-30280.m4s?e=ig8epXgD61F6hwdhiSgVhwd17WdHhwdV7W...",
    "text": "Hello everyone. Today we are going to talk about Web API structures. We will show you how to write robust client applications.",
    "segments": [
      {
        "start": 0.0,
        "end": 3.25,
        "text": "Hello everyone."
      },
      {
        "start": 3.25,
        "end": 6.8,
        "text": "Today we are going to talk about Web API structures."
      },
      {
        "start": 6.8,
        "end": 10.5,
        "text": "We will show you how to write robust client applications."
      }
    ]
  }
]

---

4. Deep-Dive Technical Architecture 🏗️

The codebase is built cleanly with modular Python scripts. The logic is separated into a local execution script (app.py) for off-platform developers and a production entry point (src/main.py) optimized for the Apify platform runtime.

4.1 Bilibili HTTP Metadata Resolution

The scraper makes an HTTP GET request to: https://api.bilibili.com/x/web-interface/view?bvid={BVID}

To avoid headers detection, the actor sets:

• User-Agent: A modern browser user agent string (Chrome/Windows).
• Referer: https://www.bilibili.com/ (Crucial for Bilibili, as they block requests lacking referrer headers).

The returned JSON payload is parsed. If the Bilibili server returns a response code other than 0 (e.g. -400 for bad parameters, -403 for geoblocks, -412 for rate limiting), the script exits with descriptive logs.

4.2 Official Subtitle Download Pipeline

When subtitle tracks are defined, Bilibili embeds their config in the subtitle array inside the data node of the JSON response:

"subtitle": {
  "allow_submit": true,
  "list": [
    {
      "id": 123456789,
      "lan": "zh-Hans",
      "lan_doc": "中文（简体）",
      "is_lock": false,
      "subtitle_url": "//subtitle.hdslb.com/bfs/subtitle/a1b2c3d4.json",
      "type": 1,
      "ai_type": 0
    }
  ]
}

The scraper iterates through this list, matching the lan tag. It formats the URL (appending https: if it begins with double slashes) and queries the JSON. Inside Bilibili’s subtitle files, content segments are modeled with from and to timestamps representing seconds. The scraper extracts these, rounds them to 2 decimal places, and maps them to a normalized output format.

4.3 Local Whisper ASR Offline Transcriber

When official transcripts are not present, the machine learning workflow is initialized.

Audio Downloader (yt-dlp)

The script configures yt-dlp in python code using direct options:

• format: bestaudio/best ensures we extract the lightweight audio stream rather than downloading the entire heavy video file. This conserves bandwidth, increases speed, and keeps the container disk space usage low.
• outtmpl: Audio is written to a secure temporary folder resolved using python's tempfile module.

FFmpeg Path Resolver

Whisper requires ffmpeg to process audio files. Since local environments and Docker containers differ, we employ a dynamic search mechanism. The actor imports imageio_ffmpeg and calls imageio_ffmpeg.get_ffmpeg_exe(). If the executable isn't globally available, it is copied to a temp directory and appended to os.environ["PATH"].

def get_ffmpeg_path(temp_dir):
    try:
        import imageio_ffmpeg
        imageio_ffmpeg_exe = imageio_ffmpeg.get_ffmpeg_exe()
        exe_name = "ffmpeg.exe" if os.name == "nt" else "ffmpeg"
        ffmpeg_exe_path = os.path.join(temp_dir, exe_name)
        if not os.path.exists(ffmpeg_exe_path):
            shutil.copy(imageio_ffmpeg_exe, ffmpeg_exe_path)
            if os.name != "nt":
                os.chmod(ffmpeg_exe_path, 0o755)
        return temp_dir
    except Exception as e:
        # Falls back to system ffmpeg if copy fails
        return None

Whisper Model Execution

We load the tiny model:

import whisper
model = whisper.load_model("tiny")
result = model.transcribe(audio_output_path)

The transcription result returns a list of dictionaries representing segments. These contain start, end, and text fields. Using tiny keeps memory footprint low (approx 150MB-250MB VRAM/RAM) while still giving high accuracy on Chinese and English speeches.

---

5. Local Development and Setup 💻

If you want to run the Bilibili Video Comments Extractor on your local machine for debugging, development, or testing, follow this step-by-step guide.

5.1 Prerequisites

• Python 3.11 or higher.
• PIP (Python Package Installer).
• FFmpeg: Although imageio-ffmpeg is configured, having ffmpeg installed globally on your operating system is highly recommended.
  • Windows: Install via Chocolatey (choco install ffmpeg) or download the binaries from the official site and add them to your system PATH.
  • macOS: Install via Homebrew (brew install ffmpeg).
  • Linux: Install via apt (sudo apt-get update && sudo apt-get install ffmpeg -y).

5.2 Step-by-Step Installation

1. Clone or Copy the Files: Extract all the repository files into your local directory.
2. Create a Virtual Environment: Navigate to the project root and create a virtual environment to avoid dependency conflicts:

   $python -m venv venv
   

3. Activate the Virtual Environment:
   • Windows (CMD/PowerShell):

     .\venv\Scripts\activate
     

   • macOS/Linux:

     $source venv/bin/activate
     

4. Install Dependencies: Run pip with the requirements file:

   $pip install -r requirements.txt
   

   This command downloads:
   • torch (CPU version specified via the extra index link to keep the image lightweight).
   • openai-whisper for machine learning transcription.
   • apify for platform integration.
   • requests for API scraping.
   • yt-dlp for media stream downloading.
   • imageio-ffmpeg for path configurations.

5.3 Running the Local Script

The Bilibili Video Transcript & Metadata Extractor project includes a standalone file app.py built for local testing. It does not require Apify SDK environments or credentials.

1. Open app.py in your text editor.
2. Look at the top of the file where the hardcoded input is defined:

   # 1. Direct hardcoded input at the top
   input_data = {
       "videoUrl": "https://www.bilibili.com/video/BV1ojn8z1EUq"
   }
   

3. Replace the URL with the Bilibili video link you wish to test.
4. Execute the script:

   $python app.py
   

5. Output Review: The script prints the JSON result directly to your terminal. Additionally, it writes the result to output.json in the current working directory. You can inspect output.json to verify the structure, transcription text, and segment timelines.

---

6. Deployment and Apify Hosting 🚀

[!TIP] Deploying to the Apify platform allows you to scale this crawler, run it periodically via schedulers, hook it to webhooks, or expose it as a custom API.

6.1 Project Configuration Files

Three files configure the environment for the Bilibili Video Transcript & Metadata Extractor on the Apify platform:

1. .actor/actor.json: This defines the metadata of the actor, the input schemas, user options, and proxy editors.
2. Dockerfile: Defines the virtual environment container settings. Let's look at how the Dockerfile is structured:

   FROM apify/actor-python:3.11
   
   # Install system dependencies (like ffmpeg for Whisper audio processing)
   USER root
   RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
   USER myuser
   
   COPY requirements.txt ./
   RUN pip install --no-cache-dir -r requirements.txt
   
   COPY . ./
   
   CMD ["python", "-m", "src.main"]
   

   Why do we run apt-get install -y ffmpeg? While the fallback helper can run binary copies, installing system-level ffmpeg in the Linux container guarantees smooth Whisper decoding without permission or path failures.
3. requirements.txt: Specifies PyTorch CPU libraries:

   --extra-index-url https://download.pytorch.org/whl/cpu
   torch
   openai-whisper
   apify>=1.6.0
   requests
   yt-dlp
   imageio-ffmpeg
   

   By explicitly requesting PyTorch CPU builds (--extra-index-url https://download.pytorch.org/whl/cpu), we prevent Docker from downloading the default CUDA GPU builds, reducing the final Docker image size from ~5GB down to ~1.2GB.

6.2 Deploying via CLI

To deploy this actor, use the official Apify CLI.

1. Install Apify CLI globally:

   $npm install -g apify-cli
   

2. Log in to your Apify Account:

   $apify login
   

   (Enter your API token when prompted).
3. Run the actor locally in an Apify simulated environment to verify integration:

   $apify run
   

4. Deploy to the Apify cloud platform:

   $apify push
   

   This command packages your directory, uploads it to Apify, builds the Docker image on their servers, and deploys it under your account. Once built, you can trigger the actor via API calls, scheduler templates, or the online Console.

---

7. Troubleshooting, Rate Limits, and Best Practices ⚠️

When using the Bilibili Video Transcript & Metadata Extractor to scrape data at scale, you might encounter issues. Here are common problems and recommended solutions.

7.1 Bilibili Code 412 Errors and Rate Limits

• Symptom: The API request returns "code": -412 or an HTTP status code 412 with a message like "请求被拦截" (Request intercepted) or "访问被拒绝" (Access denied).
• Cause: Bilibili has detected that your IP address is sending too many requests in a short period.
• Solutions:
  • Residential Proxies: Enable Apify Proxy in your input settings and specify residential IP ranges. Bilibili rarely blocks residential IPs.
  • Request Spacing: If you are running multiple actors or iterating over many links, add random delays (e.g., time.sleep(random.uniform(2, 5))) between requests.
  • Referer Headers: Ensure your request headers include "Referer": "https://www.bilibili.com/". Bilibili's CDN and API servers immediately drop requests that do not specify a valid referer.

7.2 Whisper Out-Of-Memory (OOM) Errors

• Symptom: The actor crashes during the transcription phase, showing logs such as Killed or Memory Limit Exceeded.
• Cause: Whisper models require significant memory. Even the tiny model can consume up to 1GB of RAM during audio decoding and inference.
• Solutions:
  • Increase Memory Allocation: Go to your actor settings in the Apify Console and set the memory allocation to at least 2048 MB (2 GB).
  • CPU Limitation: Running on CPU requires memory overhead. Do not run the actor with 512MB RAM if you expect it to transcribe videos using the Whisper fallback strategy.

7.3 Long Execution Timeout Issues

• Symptom: The actor takes a very long time to complete and is terminated by Apify's default timeout limits.
• Cause: Transcription of long videos (e.g., videos over 1 hour) on CPU via Whisper is slow.
• Solutions:
  • Check if the video has official subtitles. If it does, make sure the API requests are succeeding so the script doesn't fall back to Whisper.
  • Adjust the Apify Timeout limit on the run configuration page to allow up to 10-15 minutes of execution for long-form content.

---

8. FAQ & Reference Guide 💡

Here is a collection of frequently asked questions to help developers implement the scraper effectively.

Q1: Can this scraper extract subtitles in languages other than Chinese?

Yes. Bilibili creators can upload multiple subtitles. While the script defaults to searching for Simplified Chinese (zh-Hans / zh-CN), it will fall back to the first available language in the subtitle list if Chinese is missing. If you want to force English or Japanese tracks, you can modify the search sequence in src/main.py where it inspects subtitle_list.

Q2: What happens if a Bilibili video is restricted or geoblocked?

Bilibili geoblocks certain content (e.g., anime series, regional shows) to users inside mainland China. If the target video is geoblocked, requests made from US or European IP addresses will fail. To bypass this, configure your Apify Proxy settings to route requests through Chinese proxy servers.

Q3: Why does yt-dlp fail to extract video streams occasionally?

Bilibili frequently updates its video signature algorithms and video player code. When this happens, old versions of yt-dlp might fail to resolve the formats. The solution is to ensure your requirements.txt pulls the latest version of yt-dlp. You can also force a build update on your Apify console to rebuild the Docker cache with updated libraries.

Q4: Is there a way to use a larger Whisper model (e.g., base, small, medium)?

Yes, but you must be aware of the hardware constraints. You can modify the load parameter in src/main.py: model = whisper.load_model("base") Note that larger models take longer to download on launch, require significantly more RAM (e.g., 4GB+ for small / medium), and run much slower on CPUs. For serverless deployments, tiny is the most cost-effective and practical choice.

Q5: How accurate are the timestamps in Whisper output?

Whisper segment timestamps are highly accurate (usually within 50-100 milliseconds). However, Whisper segments audio based on natural speaking pauses. Sometimes, sentences might be split differently compared to manual subtitles, but the overall alignment remains fully synchronized.

Q6: Can I parse list formats like Bilibili Playlists (Bw/ep)?

Currently, this actor is designed to process individual video URLs. To parse lists, write a wrapper script that uses the Bilibili playlist APIs to retrieve all individual video URLs, then dispatch them to this actor in parallel or sequentially.

Q7: Are the direct CDN URLs (videoUrl and audioUrl) permanent?

No. Bilibili stream URLs returned in the metadata contain verification tokens in the query string (?e=ig8ep...). These signatures expire after a few hours (typically 2-6 hours). These URLs are designed for immediate consumption, processing, or downloading.

---

9. Conclusion & License 🏆

The Bilibili Video Transcript & Metadata Extractor represents a robust developer tool for parsing modern web media content. By packaging this tool into an Apify Actor, you gain immediate access to automated scalability, scheduling, and API distribution.

This project is open-source and free for developers. Please ensure you respect Bilibili's Terms of Service and content copyright guidelines when using extracted media files.

Happy Coding! 🚀📹🤖