Web to Markdown for AI – Japanese & Chinese Ready

Turn web pages – or HTML you already have – into clean Markdown or plain text for LLMs, RAG and AI agents. Made with extra care for Japanese and Chinese pages, which generic converters often garble.

• 🈶 No mojibake – decodes Shift_JIS (incl. Windows-31J characters like ① ㈱ ～), EUC-JP, GB2312/GBK/GB18030 and Big5 the way browsers do, and recovers from the most common wrong charset headers. You can also force an encoding.
• 📖 Furigana / ruby handled – pages with <ruby> (furigana, pinyin) normally fall apart one character per line. Choose: drop the readings, put them in parentheses, or keep the HTML.
• 🧹 Main content only – menus, footers and ads removed; tables and links kept in Markdown.
• 📝 Raw HTML input – already scraped or generated the HTML? Send it directly; nothing is fetched.
• 🔁 Change reports – schedule it and get new / changed / unchanged plus a line diff for each page.
• 🤖 Respects robots.txt by default (pages it disallows are reported and not charged).
• 💸 $0.80 per 1,000 pages – failed or blocked pages are free.

How to use

1. Put page URLs into Page URLs (up to 1,000), and/or objects {"html": "...", "url": "https://..."} into Raw HTML documents.
2. Choose Output formats: markdown (default), text, html (cleaned main content), links.
3. Run. Each page becomes one dataset item.

Leave both lists empty to try a free bundled sample (a Japanese page saved in Shift_JIS).

Pricing

$0.0008 per converted page ($0.80 per 1,000). Pages that fail, return non-HTML content, or are disallowed by robots.txt are not charged. Each run also has Apify's standard start fee: $0.00005 per run up to 1 GB of memory, plus $0.00005 for each extra GB. Set Max total charge to cap your spend – pages beyond it are skipped and reported.

Input

Field	Default	Notes
urls	–	http(s), up to 1,000 per run, 5 MB per page
htmlDocuments	–	Up to 100 per run, 5 MB each; url optional (resolves relative links)
formats	["markdown"]	markdown, text, html, links
mainContentOnly	true	false keeps more of the page
ruby	drop	drop, parentheses, keep
respectRobotsTxt	true	Turn off only for sites you own or may fetch
trackChanges / trackerName	false / web-markdown-tracker	Tracker lives in your own Apify account

Output

{
  "url": "https://www.aozora.gr.jp/cards/000148/files/773_14560.html",
  "status": "ok",
  "finalUrl": "https://www.aozora.gr.jp/cards/000148/files/773_14560.html",
  "encoding": "cp932",
  "title": "こころ",
  "description": null,
  "language": "ja",
  "publishedDate": "1995-01-01",
  "markdown": "私はその人を常に先生と呼んでいた。だからここでもただ先生と書くだけで本名は打ち明けない。これは世間を憚かる遠慮というよりも、その方が私にとっ…",
  "charCount": 162500,
  "contentHash": "20b7e8f8c172c79a43ed"
}

status is ok, error, blocked_by_robots, unsupported (not HTML or text) or skipped (max total charge reached) – only ok is charged.

FAQ

Does it run JavaScript? No. It reads the HTML the server sends, which is fast and cheap and works for most articles, docs, blogs, news and shop pages. Pages that build their content only in the browser will come back mostly empty – send their rendered HTML through Raw HTML documents instead.

Does it bypass logins, paywalls or bot protection? No. It fetches public pages like a normal client and follows robots.txt by default. You are responsible for having the right to process the pages you send.

Why does encoding say cp932 for a Shift_JIS page? Browsers decode "Shift_JIS" as Windows-31J (cp932), which includes characters such as ① and ㈱ that strict Shift_JIS lacks. We do the same, so nothing turns into ?.

Can two runs share a tracker? Not at the same time. Runs that use the same Tracker name should not overlap (a daily schedule is fine); use different names for parallel watch lists.

How is my data handled? Pages are fetched and converted inside your run. Results stay in your Apify storage; the change tracker (if on) is a key-value store in your own account. We keep no copies.

Notes

• Built on the open-source trafilatura (Apache 2.0) and charset-normalizer (MIT).
• The bundled sample page and its text are our own.