# Web to Markdown for AI – Japanese & Chinese Ready (`kumo-data-tools/web-to-markdown-cjk`) Actor

Turn web pages or raw HTML into clean Markdown or text for LLMs and RAG. Decodes Shift_JIS, EUC-JP, GBK and Big5 correctly, handles furigana/ruby, respects robots.txt, and can report what changed since the last run. $0.80 per 1,000 pages.

- **URL**: https://apify.com/kumo-data-tools/web-to-markdown-cjk.md
- **Developed by:** [Kumo Data Tools](https://apify.com/kumo-data-tools) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web to Markdown for AI – Japanese & Chinese Ready

Turn web pages – or HTML you already have – into **clean Markdown or plain text** for LLMs, RAG and AI agents. Made with extra care for **Japanese and Chinese** pages, which generic converters often garble.

- 🈶 **No mojibake** – decodes Shift_JIS (incl. Windows-31J characters like ① ㈱ ～), EUC-JP, GB2312/GBK/GB18030 and Big5 the way browsers do, and recovers from the most common wrong charset headers. You can also force an encoding.
- 📖 **Furigana / ruby handled** – pages with `<ruby>` (furigana, pinyin) normally fall apart one character per line. Choose: drop the readings, put them in parentheses, or keep the HTML.
- 🧹 **Main content only** – menus, footers and ads removed; tables and links kept in Markdown.
- 📝 **Raw HTML input** – already scraped or generated the HTML? Send it directly; nothing is fetched.
- 🔁 **Change reports** – schedule it and get `new / changed / unchanged` plus a line diff for each page.
- 🤖 **Respects robots.txt** by default (pages it disallows are reported and not charged).
- 💸 **$0.80 per 1,000 pages** – failed or blocked pages are free.

### How to use

1. Put page URLs into **Page URLs** (up to 1,000), and/or objects `{"html": "...", "url": "https://..."}` into **Raw HTML documents**.
2. Choose **Output formats**: `markdown` (default), `text`, `html` (cleaned main content), `links`.
3. Run. Each page becomes one dataset item.

Leave both lists empty to try a **free bundled sample** (a Japanese page saved in Shift_JIS).

### Pricing

**$0.0008 per converted page** ($0.80 per 1,000). Pages that fail, return non-HTML content, or are disallowed by robots.txt are **not charged**. Each run also has Apify's standard start fee: $0.00005 per run up to 1 GB of memory, plus $0.00005 for each extra GB. Set **Max total charge** to cap your spend – pages beyond it are skipped and reported.

### Input

| Field | Default | Notes |
|---|---|---|
| `urls` | – | http(s), up to 1,000 per run, 5 MB per page |
| `htmlDocuments` | – | Up to 100 per run, 5 MB each; `url` optional (resolves relative links) |
| `formats` | `["markdown"]` | `markdown`, `text`, `html`, `links` |
| `mainContentOnly` | true | false keeps more of the page |
| `ruby` | `drop` | `drop`, `parentheses`, `keep` |
| `respectRobotsTxt` | true | Turn off only for sites you own or may fetch |
| `trackChanges` / `trackerName` | false / `web-markdown-tracker` | Tracker lives in your own Apify account |

### Output

```json
{
  "url": "https://www.aozora.gr.jp/cards/000148/files/773_14560.html",
  "status": "ok",
  "finalUrl": "https://www.aozora.gr.jp/cards/000148/files/773_14560.html",
  "encoding": "cp932",
  "title": "こころ",
  "description": null,
  "language": "ja",
  "publishedDate": "1995-01-01",
  "markdown": "私はその人を常に先生と呼んでいた。だからここでもただ先生と書くだけで本名は打ち明けない。これは世間を憚かる遠慮というよりも、その方が私にとっ…",
  "charCount": 162500,
  "contentHash": "20b7e8f8c172c79a43ed"
}
```

`status` is `ok`, `error`, `blocked_by_robots`, `unsupported` (not HTML or text) or `skipped` (max total charge reached) – only `ok` is charged.

### FAQ

**Does it run JavaScript?** No. It reads the HTML the server sends, which is fast and cheap and works for most articles, docs, blogs, news and shop pages. Pages that build their content only in the browser will come back mostly empty – send their rendered HTML through **Raw HTML documents** instead.

**Does it bypass logins, paywalls or bot protection?** No. It fetches public pages like a normal client and follows robots.txt by default. You are responsible for having the right to process the pages you send.

**Why does `encoding` say `cp932` for a Shift_JIS page?** Browsers decode "Shift_JIS" as Windows-31J (cp932), which includes characters such as ① and ㈱ that strict Shift_JIS lacks. We do the same, so nothing turns into `?`.

**Can two runs share a tracker?** Not at the same time. Runs that use the same Tracker name should not overlap (a daily schedule is fine); use different names for parallel watch lists.

**How is my data handled?** Pages are fetched and converted inside your run. Results stay in your Apify storage; the change tracker (if on) is a key-value store in your own account. We keep no copies.

### Notes

- Built on the open-source [trafilatura](https://github.com/adbar/trafilatura) (Apache 2.0) and [charset-normalizer](https://github.com/jawah/charset_normalizer) (MIT).
- The bundled sample page and its text are our own.

# Actor input Schema

## `urls` (type: `array`):

Web pages to convert (http/https), up to 1,000 per run. Leave empty (and HTML below empty) to try the free bundled Shift_JIS sample.

## `htmlDocuments` (type: `array`):

Already have the HTML? Pass objects `{"html": "<html>…", "url": "https://… (optional, used to resolve links)"}`. Nothing is fetched for these. Up to 100 per run, 5 MB each.

## `formats` (type: `array`):

Any of `markdown`, `text`, `html` (cleaned main content) and `links`.

## `mainContentOnly` (type: `boolean`):

Keep the article / main text and drop menus, footers and ads. Turn off to keep more of the page.

## `ruby` (type: `string`):

Pages with <ruby> (furigana, pinyin) otherwise break into one character per line. `drop` keeps the base text (私), `parentheses` adds the reading (私（わたくし）), `keep` leaves the HTML as is.

## `forceEncoding` (type: `string`):

Optional. Decode every fetched page with this encoding (e.g. `shift_jis`, `euc-jp`, `gbk`, `big5`, `utf-8`) instead of detecting it. Leave empty for automatic detection.

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site's robots.txt disallows (not charged). Turn off only for sites you own or have permission to fetch.

## `trackChanges` (type: `boolean`):

Remember each page's content in a named key-value store in your account and return `changeStatus` (new / changed / unchanged) plus a line diff. Ideal for a daily schedule.

## `trackerName` (type: `string`):

Name of the key-value store that remembers pages. Use different names for different watch lists.

## Actor input object example

```json
{
  "formats": [
    "markdown"
  ],
  "mainContentOnly": true,
  "ruby": "drop",
  "respectRobotsTxt": true,
  "trackChanges": false,
  "trackerName": "web-markdown-tracker"
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "formats": [
        "markdown"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("kumo-data-tools/web-to-markdown-cjk").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "formats": ["markdown"] }

# Run the Actor and wait for it to finish
run = client.actor("kumo-data-tools/web-to-markdown-cjk").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "formats": [
    "markdown"
  ]
}' |
apify call kumo-data-tools/web-to-markdown-cjk --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kumo-data-tools/web-to-markdown-cjk"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/V9VSbhTpt9yfUJF3P/builds/LyOkm94ThG0de7hUG/openapi.json