# Webnovel Book Scraper (`research_master/webnovel-book-scraper`) Actor

Collect public book metadata and bounded chapter-index metadata from Webnovel (起点国际).

- **URL**: https://apify.com/research\_master/webnovel-book-scraper.md
- **Developed by:** [Research Master](https://apify.com/research_master) (community)
- **Categories:** Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.90 / 1,000 book results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Webnovel Book Scraper

Collect public book metadata from **Webnovel（起点国际）** for catalog research, market monitoring, and novel discovery. The Actor accepts Webnovel keyword searches and direct book-detail URLs, deduplicates books by canonical URL, and keeps search provenance on each successful book row.

> The default research unit is one public book-detail page. Chapter collection is opt-in and returns chapter metadata only (title and URL); the Actor does not download chapter text.

### What it collects

- Title, subtitle, author and public author URL
- Description, genres, tags, language, status, cover URL
- Rating, rating/review/view counts when exposed by the page
- Word count, chapter count, latest chapter, published/updated timestamps
- Matching search query, rank, and source URL
- An optional bounded chapter index

The default Dataset contains only successful `book` rows. Run, search-page, and failure audits are saved as named Key-Value Store records (`RUN_AUDIT` and `SEARCH_AUDITS`) so operational records do not become book result events.

### Input

Use `searchQueries`, `startUrls`, or both. A direct URL is the most reliable bounded smoke test.

```json
{
  "searchQueries": ["cultivation", "fantasy"],
  "startUrls": [
    { "url": "https://www.webnovel.com/book/lord-of-mysteries_11022733006234505" }
  ],
  "maxBooksPerQuery": 20,
  "maxSearchPages": 3,
  "includeChapterList": false,
  "maxChaptersPerBook": 100,
  "concurrency": 2,
  "requestDelayMs": 500,
  "maxRequestRetries": 2,
  "useApifyProxy": false,
  "proxyCountry": ""
}
```

| Field | Meaning |
| --- | --- |
| `searchQueries` | Keywords sent to Webnovel's public search pages. |
| `startUrls` | Public Webnovel book-detail URLs. Unsupported hosts are ignored. |
| `maxBooksPerQuery` | Per-query result cap before cross-query deduplication. |
| `maxSearchPages` | Maximum pages requested for each query. |
| `includeChapterList` | Add up to `maxChaptersPerBook` public chapter-index entries. No chapter body is fetched. |
| `concurrency` | Number of browser pages in parallel, 1-5. |
| `requestDelayMs` | Delay before every navigation. |
| `maxRequestRetries` | Total browser navigation attempts. |
| `useApifyProxy` | Route through Apify Proxy when Webnovel is unstable or region-specific. This does not bypass login or access controls. |
| `proxyCountry` | Optional ISO country code such as `US`. |

### Output and audit

Each successful Dataset row has `record_type: "book"` and `platform: "webnovel"`. Stable keys are `book_url` and, when available, `book_id`. `search_hits` contains every query/rank that matched a deduplicated book.

`RUN_AUDIT` contains discovered, collected, failed, and chapter-list counts plus sanitized failure messages. `SEARCH_AUDITS` contains one object per attempted keyword search page. Inspect those records before treating a run as complete.

### Access and responsible use

- Only public Webnovel pages are requested. The Actor does not log in, solve CAPTCHA, or access private/deleted content.
- Webnovel can return anti-bot or region-dependent pages. A successful build is not evidence of successful extraction; verify `book` Dataset rows and `RUN_AUDIT` after a bounded run.
- Site markup and search behavior can change. Parser fallbacks use JSON-LD, embedded page state, and rendered HTML; review failures when fields become unavailable.
- Respect Webnovel terms, robots guidance, copyright rules, rate limits, and applicable law. Do not use the optional chapter list to redistribute copyrighted chapter text.

### Current pricing

This Actor is published with Apify `PAY_PER_EVENT` pricing. One successful book row is the paid economic unit; audit and failure records stay out of the default Dataset.

| Event | FREE | BRONZE | SILVER | GOLD+ |
| --- | ---: | ---: | ---: | ---: |
| Book result (`apify-default-dataset-item`) | $0.0049 | $0.0039 | $0.0034 | $0.0029 |

Actor Start is `$0.00005` per memory-based start event. Platform usage is paid by the run owner, and the minimum accepted maximum-charge cap is `$0.10`. The source does not emit a custom `book-completed` event; only successful default-Dataset book rows are billed.

### 中文说明

该 Actor 仅采集起点国际 Webnovel 的公开书籍详情页，支持关键词搜索和直接书籍 URL。默认每本书输出一条 `book` 记录；章节功能只采集章节标题和 URL，不抓取正文。反爬、地区限制和页面改版可能导致失败，请结合 `RUN_AUDIT` 判断覆盖情况，并遵守网站条款、版权和速率限制。

# Actor input Schema

## `searchQueries` (type: `array`):

Keywords searched on Webnovel's public search pages. Results are deduplicated by canonical book URL.

## `startUrls` (type: `array`):

Public Webnovel book-detail URLs. Use this for a deterministic bounded smoke test.

## `maxBooksPerQuery` (type: `integer`):

Maximum accepted search results per keyword before cross-query deduplication.

## `maxSearchPages` (type: `integer`):

Hard cap on pages requested for each keyword.

## `includeChapterList` (type: `boolean`):

Collect public chapter titles and URLs when exposed on the book page; no chapter text is fetched.

## `maxChaptersPerBook` (type: `integer`):

Chapter-index cap when chapter collection is enabled.

## `concurrency` (type: `integer`):

Number of pages processed in parallel. Reduce this if a source returns challenges.

## `requestDelayMs` (type: `integer`):

Politeness delay before each navigation.

## `maxRequestRetries` (type: `integer`):

Total attempts for transient failures, including the first request.

## `useApifyProxy` (type: `boolean`):

Route browser requests through Apify Proxy. This does not bypass login or access controls.

## `proxyCountry` (type: `string`):

Optional ISO country code used with Apify Proxy, for example US or CN.

## Actor input object example

```json
{
  "searchQueries": [],
  "startUrls": [],
  "maxBooksPerQuery": 20,
  "maxSearchPages": 3,
  "includeChapterList": false,
  "maxChaptersPerBook": 100,
  "concurrency": 2,
  "requestDelayMs": 500,
  "maxRequestRetries": 2,
  "useApifyProxy": false,
  "proxyCountry": ""
}
```

# Actor output Schema

## `results` (type: `string`):

Default Dataset containing one successful public book row per canonical book URL.

## `runAudit` (type: `string`):

Sanitized discovered, collected, failed, and chapter-list counts with failure details.

## `searchAudits` (type: `string`):

One object per attempted keyword search page.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("research_master/webnovel-book-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("research_master/webnovel-book-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call research_master/webnovel-book-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,research_master/webnovel-book-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kcRFljV6Wd5cEaENH/builds/xeg0Vxo2LyXBdIs5u/openapi.json