# Web Page to Markdown for LLM & RAG – Chinese/Big5/GBK Ready (`raspberry_rondel/web-to-markdown-cjk`) Actor
Convert web pages to clean Markdown for LLMs & RAG: main content only, absolute links, images without trackers, intact code blocks, Big5/GBK/GB2312 Chinese decoding, optional RAG chunks. HTTP first, browser only for JavaScript pages. Blocked, failed and garbled pages are free. No proxies.
- **URL**: https://apify.com/raspberry_rondel/web-to-markdown-cjk.md
- **Developed by:** [星賢 陳](https://apify.com/raspberry_rondel) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
Pay per event
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event
## What's an Apify Actor?
An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.
For examples already wired to this Actor's own input schema, see the [API](#api) section below.
Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).
# README
## Web Page to Markdown for LLM & RAG – Chinese, Big5 & GBK ready
Convert web pages you are allowed to use into clean Markdown for LLM and RAG pipelines and AI agents. Typical sources are your own docs, public documentation, and openly licensed or permitted content. You get the article or document body only, with absolute links, useful images and code blocks kept intact. Chinese text is decoded from the page's declared encoding, including old **Big5, GBK and GB2312** sites. Optional **RAG chunks** come with heading paths and overlap.
You pay **per page converted**. Pages that fail, are blocked, are skipped, or come back garbled are **free**. Apify's small platform start fee is still charged for every run (see Pricing).
### Typical uses
- Turning your own product documentation, help centre or blog into a knowledge base for your chatbot or RAG index.
- Preparing public documentation (for example language or framework docs) and openly licensed pages for search, summarising or internal Q\&A.
- Research and internal reference on pages you are permitted to copy.
- Converting legacy Big5 or GBK pages that you own or maintain into clean, readable Markdown.
### What it does
- **Main content only.** Navigation, headers, footers, cookie banners, sidebars, "related articles" lists and share buttons are removed. You can turn this off to convert the whole page.
- **Links you can use.** Links are made absolute (`https://…` or `http://…`) and kept on one line. In an internal test (2026-10-17, build 0.2.6, 24 converted pages out of 30), our output had 1,672 Markdown links, none relative or broken across lines; one of them had an empty target. Same-page anchors, "edit" links and heading permalinks (¶) are dropped.
- **Images without junk.** Content images are kept as ``. Tracking pixels, spacers, icons, `data:` URIs and duplicate sizes of the same image are removed.
- **Chinese encodings handled.** Big5 (including the ETEN extensions), GBK/GB2312 (read as GB18030) and UTF-8 are detected from the HTTP header and the page's `` blocks become fenced code blocks with their line breaks kept, including code inside list items. Inline code stays inline.
- **Wikipedia-style infoboxes** become a compact `- **Key**: value` list instead of scattered fragments.
- **Fast and cheap by default.** Each page is fetched over plain HTTP first. In `auto` mode, a headless Chromium browser is used only when the server HTML is a JavaScript app shell with little text (for example an empty `` or a `` asking for JavaScript), or when the plain HTTP connection fails (see below). Short pages without such signals are converted from the server HTML and charged as a normal page. If rendering does not add meaningful content, the page is charged as a normal page. You can also force HTTP-only or browser rendering for every page.
- **Metadata.** Title, description, author, published date, site name, language, canonical URL, declared encodings.
- **RAG chunks (optional).** Set `chunkSize` to also get a `chunks` array per page. Pages are split at headings, then paragraphs, then sentence ends (。！？ as well as . ! ?). Code blocks and tables stay whole when they fit. Each chunk carries its heading path, and you can add an overlap.
### Responsible crawling (what we do *not* do)
- **robots.txt is respected** for every URL and every redirect (crawler name `RondelBot`). Disallowed pages are skipped and free. If a site's robots.txt cannot be fetched because of a server error, rate limit or timeout, the site is treated as disallowed. You can switch this off with `respectRobotsTxt` only if you have the site owner's permission. If you do, the row of each URL the Actor processes (including invalid URLs) is marked `robotsTxtIgnored: true` with a note that robots.txt was not checked. Rows for duplicates and for URLs skipped because of `maxUrlsPerDomain` or the spending limit are not fetched and do not carry this flag.
- robots.txt is evaluated for `RondelBot`, or for `*` when there is no `RondelBot` group. Site owners can opt out with `User-agent: RondelBot` / `Disallow: /`.
- **AI-crawler opt-outs are respected by default** (`respectAiOptOut`). If a site's robots.txt has a group for a known AI crawler (for example GPTBot, CCBot, Google-Extended, anthropic-ai, ClaudeBot or PerplexityBot) that disallows the page's path, the page is skipped: status `skipped-ai-crawler-opt-out`, free. This is checked for the URL you give and again after redirects. Only groups that name such a crawler count; a missing robots.txt means no opt-out. The list of crawler names is ours and may not cover every AI crawler, so check the site's rules yourself if your use involves AI.
- **Paywalled pages are skipped by default** (`skipPaywalledContent`). See "Paywalls" under Tips & limits.
- **No proxies, no disguise, no anti-bot bypass.** We use an honest user agent. If a site shows a bot challenge or blocks cloud servers, the page is reported as `blocked-by-anti-bot` and **not charged**; it is not retried in a browser to get around the block.
- If the plain HTTP request fails with a connection error, the page is tried again in a normal headless browser (same honest user agent, same robots.txt rules). Anti-bot challenges are not retried.
- Page requests to the same site are spaced at least 1 second apart, or more if the site's robots.txt `Crawl-delay` asks for it. Pages whose robots.txt asks for a Crawl-delay above 30 seconds are skipped (status `blocked-by-robots-txt`, free), in both plain HTTP and browser mode.
- On HTTP 429 or 503, the Actor waits and retries once only if the site's `Retry-After` is 30 seconds or less; otherwise it backs off and the page is free.
- You cannot supply cookies, headers or credentials. Cookies that a site sets during a run may be kept for the rest of that run, as with an ordinary HTTP client; this is not a login feature.
- Private and internal network addresses are refused.
### Input
| Field | Default | What it does |
|---|---|---|
| `urls` | – | Pages to convert, one URL per line. Links are not followed. |
| `renderMode` | `auto` | `auto`: HTTP first; browser only when the server HTML is a JavaScript app shell with little text, or after a connection failure. `http`: no browser. `browser`: render every page in the browser. |
| `mainContentOnly` | `true` | Remove boilerplate and keep the main content. |
| `includeLinks` / `includeImages` | `true` | Keep links and images as Markdown. |
| `chunkSize` | `0` (off) | 200–20,000 characters. Adds a `chunks` array per page. |
| `chunkOverlap` | `0` | Characters repeated from the end of the previous chunk (at most half the chunk size). |
| `maxMarkdownChars` | `0` (no limit) | Cut each page's Markdown to this length. The source line (see `addSourceLine`) is added afterwards, so the output can be slightly longer. |
| `minContentChars` | `50` | Pages with less text are reported as `empty-content` and are free. |
| `browserWaitMs` | `1000` | Extra wait after load for rendered pages (max 5,000). |
| `requestTimeoutSecs` | `30` | Per-page timeout (10–40 s). |
| `maxConcurrency` | `10` | Pages processed in parallel (1–20). |
| `respectRobotsTxt` | `true` | See above. |
| `locale` | `en-US` | Browser language for rendered pages, e.g. `zh-TW`. |
| `respectAiOptOut` | `true` | Skip a page when the site's robots.txt has a group for a known AI crawler (for example GPTBot, CCBot, Google-Extended, anthropic-ai, ClaudeBot, PerplexityBot) that disallows the page's path, also after redirects. Status `skipped-ai-crawler-opt-out`, `aiOptOut: true`, free. |
| `skipPaywalledContent` | `true` | Skip a page that declares paid content (schema.org `isAccessibleForFree: false`, `article:content_tier` = `locked`) or contains a common paywall marker (an element whose class or id is, for example, `paywall` or `regwall`). Every fetched page has `paywallDetected`. Status `skipped-paywall`, free. Heuristic. |
| `addSourceLine` | `true` | Start each converted page's Markdown with one line: `Source: <url> (<site name>)`. |
| `maxUrlsPerDomain` | `0` (no limit) | At most this many URLs per site (host name without `www.`) per run. The rest get status `skipped-domain-limit`, free. |
```json
{ "urls": ["https://docs.python.org/3/library/asyncio-task.html", "https://zh.wikipedia.org/zh-tw/臺灣鐵路"],
"renderMode": "auto", "chunkSize": 800, "chunkOverlap": 100 }
```
### Output
One dataset row per URL (shortened example from a Big5 page):
```json
{
"url": "https://www.webpage.idv.tw/maillist/maillist3/new/10/10.htm",
"status": "ok",
"httpStatus": 200,
"renderedWithBrowser": false,
"title": "網頁設計中 HTML 常犯的幾個錯誤",
"encoding": { "header": null, "meta": "big5" },
"markdown": "Source: 網頁設計中 HTML 常犯的幾個錯誤 <https://www.webpage.idv.tw/maillist/maillist3/new/10/10.htm> (webpage.idv.tw)\n\n# 網頁設計中 HTML 常犯的幾個錯誤\n\n壹、前言：\n\n在我學習超過十年的 HTML 標籤歲月裡，不論是網頁製作或教學中，經常看到一些不應該發生的 HTML 錯誤。\n\n…",
"stats": { "markdownChars": 3130, "textChars": 2784, "cjkChars": 2255, "links": 0, "images": 0, "extraction": "main-content" },
"quality": { "replacementChars": 0, "mojibakeHits": 0, "badRatio": 0.0 },
"paywallDetected": false,
"aiOptOut": false,
"charged": true,
"chargedEvent": "page",
"timings": { "httpSecs": 1.4 }
}
```
Each row also carries `url`, `canonicalUrl`, `author` and `siteName` where the page provides them, so you can keep the source with the text. By default the Markdown itself also starts with a `Source:` line (title, URL and site name).
With `chunkSize` set, OK rows also contain:
````json
"chunks": [
{ "index": 1, "chars": 813, "headings": ["Coroutines and tasks", "Coroutines"],
"text": "To actually run a coroutine, asyncio provides the following mechanisms:\n\n- The [`asyncio.run()`](https://docs.python.org/3/library/asyncio-runner.html#asyncio.run) function …\n\n```\nimport asyncio\nimport time\n…\n```" }
]
````
**Status values:** `ok` (charged) · `blocked-by-robots-txt` · `blocked-by-anti-bot` · `rate-limited` · `http-error` · `unreachable` · `not-html` (PDFs, images) · `empty-content` · `failed` (including `failureReason: "garbled-text"`) · `error` · `invalid-url` · `skipped-spending-limit` · `skipped-ai-crawler-opt-out` · `skipped-paywall` · `skipped-domain-limit` · `duplicate`.
- `duplicate` means the same URL appeared earlier in your list after removing `#fragment`, tracking parameters (such as `utm_*`, `fbclid`, `gclid`), host case and the default port. It is not fetched; the row has `duplicateOf`.
- `skipped-domain-limit` means the site already reached `maxUrlsPerDomain` in this run.
- `unreachable` means the URL is well-formed but the site could not be reached: the domain does not exist, the DNS lookup failed, or the connection failed.
- `invalid-url` means the URL itself is malformed, not http(s), or points to a private/local address.
- `rate-limited` means the site answered 429/503 with a `Retry-After` above 30 seconds.
**Only `ok` rows are charged.** Every row has a `charged` flag and an `error` text explaining what happened. A run summary is stored in the key-value store as `RUN_SUMMARY`.
### Pricing (pay per event)
| Event | Price | When |
|---|---|---|
| `page` | **$0.001 per page** ($1 per 1,000) | Page converted from the server HTML (no browser needed) |
| `browser-page` | **$0.004 per page** ($4 per 1,000) | Page that was rendered in a headless browser and returned from the browser version (JavaScript app shell with little text in `auto` mode, `browser` mode, or a browser retry after a connection failure) |
| Actor start (`apify-actor-start`, Apify's standard start event) | $0.0005 per GB of memory (runs use 1 GB, so $0.0005 per run) | At the start of each run, including runs where every URL is invalid, skipped or free |
A page is charged **either** `page` **or** `browser-page`, not both. In `auto` mode the browser is used only when the server HTML is a JavaScript app shell with little text (for example an empty `<div id="root">` or a `<noscript>` asking for JavaScript), or when the plain HTTP connection failed. Short pages without such signals are converted from the server HTML and charged as a normal `page`. If rendering does not add meaningful content (at least 20% and at least 200 characters more Markdown than the server-HTML version), the server-HTML version is returned and charged as a normal `page`.
**Free, not charged:**
- pages disallowed by robots.txt;
- anti-bot challenges and blocks;
- HTTP 4xx/5xx errors, rate-limited pages, timeouts and network errors;
- non-HTML files;
- pages with almost no text (`empty-content`);
- **garbled pages**;
- pages skipped because of an AI-crawler opt-out (`skipped-ai-crawler-opt-out`) or a paywall (`skipped-paywall`);
- URLs over `maxUrlsPerDomain` (`skipped-domain-limit`) and duplicate URLs (`duplicate`).
The only charge that applies even when every page is free is Apify's platform start fee (US$0.0005 per GB of memory; runs use 1 GB, so US$0.0005 per run). It is charged even if every input URL is invalid.
A page counts as garbled when the converted text contains many U+FFFD replacement characters or mojibake patterns, for example UTF-8 read as Latin-1, GBK/Big5 read as Latin-1, or broken private-use glyphs. Such a page gets `status: "failed"`, `failureReason: "garbled-text"` and `charged: false`; the Markdown is still included so you can inspect it.
Chunking does not change the price: you pay per page, not per chunk. Set **Maximum cost per run** in the run options and the Actor stops before exceeding it; skipped URLs are marked and free.
### Measured results in our test
*Internal test, 2026-10-17, version 0.2.6 with the current defaults (`Source:` line, AI-crawler opt-out skip, paywall skip). 30 pages: 15 Chinese (Traditional/Taiwan, Simplified, one Big5, one GBK, one GB2312) and 15 international (docs, blogs, news, wiki, e-commerce). Results may differ for other pages, settings, versions or dates.*
- **Pages converted:** 24 of 30. The other 6 were skipped (free) because the site's robots.txt disallows AI crawlers for that path (status `skipped-ai-crawler-opt-out`). No page was skipped as paywalled.
- **Legacy Chinese encodings:** the Big5, GBK and GB2312 pages were converted with 0 U+FFFD replacement characters. Across all converted pages our checker found 0 replacement characters and 0 mojibake patterns.
- **Links:** on the 24 converted pages, our output had 1,672 Markdown links (counted by our link checker in the returned Markdown): 1,585 `https://`, 86 `http://` and 1 with an empty target (`[asyncio.AbstractEventLoop]()` on a docs.python.org page). None were relative, broken across lines or same-page anchors. The per-page `stats.links` field adds up to 1,664 for the same pages, because it does not count the 8 links whose text contains square brackets or whose target is empty.
- **Main content:** the 24 converted pages produced 457,132 Markdown characters in total (including about 2,600 characters of `Source:` lines).
- **JavaScript page:** one JavaScript-only category page was rendered in a browser by auto mode and returned the page text (charged as `browser-page`).
**How we measured.** The URL list was fixed on 2026-10-17 and not changed. The test was one run on the Apify platform with only the 30 URLs as input, so every default was on (`renderMode: auto`, `respectAiOptOut`, `skipPaywalledContent`, `addSourceLine`). The output was scored with our own script and link checker. The 6 pages skipped for AI-crawler opt-outs were on twreporter.org, ithome.com.tw, cna.com.tw, github.com, bbc.com and theverge.com.
### Tips & limits
- Single pages only; it does not crawl or follow links. Use it after a crawler or sitemap step.
- PDFs and Office files are reported as `not-html` (free). Use a document converter for those.
- Some e-commerce product pages contain very little text in their HTML. The output is then short but honest.
- Sites that block cloud servers will show up as `blocked-by-anti-bot`, at no cost.
- **Paywalls.** The Actor has no login, cookie, header or proxy options and does not try to get past paywalls. With `skipPaywalledContent` on (the default), a page is skipped (status `skipped-paywall`, not converted, free) when it declares paid content (schema.org `isAccessibleForFree: false`, `article:content_tier` = `locked`) or contains a common paywall marker. The page's HTML is still downloaded to check this. Every fetched page has `paywallDetected`. Detection is heuristic: metered paywalls and paywalls without such markers are not detected. Some publishers include full articles in their HTML and hide them with scripts; you are responsible for not using the Actor to obtain content you are not entitled to read or reuse.
- Runs use 1 GB of memory; for large lists keep `maxConcurrency` at the default.
### Responsible use and legal notice
This Actor only visits the URLs you give it, from Apify's servers, with an honest user agent (`RondelBot`). By default it obeys robots.txt (including redirects), skips pages whose robots.txt opts out of known AI crawlers, skips pages marked as paywalled, and waits at least 1 second between page requests to the same site. It has **no login, cookie, header or proxy options** and does not try to get past logins, paywalls, CAPTCHAs or other anti-bot protections.
**You are responsible for how you use it and for what you do with the results.** Before running it, make sure you have the right to access and copy the pages you submit, and that your use complies with the websites' terms, copyright law and data-protection law (for example GDPR, CCPA, or Taiwan's Personal Data Protection Act). Turning off `respectRobotsTxt` is your decision and your responsibility; do it only for sites you own or have written permission to access. The same applies to turning off `respectAiOptOut` or `skipPaywalledContent`.
Results can contain personal data (for example names or contact details shown on a page). Do not collect personal data unless you have a lawful basis to do so. Your inputs and results are stored only in your own Apify account (the run's dataset and key-value store); the Actor does not send them to the developer or to third parties.
**Copyright and AI use.** The Markdown output is a copy of other people's content. Most articles, documentation and blog posts are protected by copyright. Do not republish, resell or redistribute the output without the owner's permission. Using it to train or fine-tune AI models, or to build a searchable RAG index, may require the rights holder's permission depending on your country; you are responsible for obtaining the permission you need. Keep the source URL and author with the text (the default `Source:` line helps with this).
**Paywalls.** Do not use this Actor to obtain content you are not entitled to read or reuse. The default paywall skip is heuristic and does not catch every paywall. Some publishers include the full text of paywalled articles in their HTML and hide it with scripts; collecting such content may breach the publisher's terms or the law.
Site owners can opt out by adding `User-agent: RondelBot` and `Disallow: /` to their robots.txt; the change applies from the Actor's next visit to the site.
This notice is not legal advice. If you are unsure whether your use is lawful, consult a lawyer.
### 負責任使用與法律聲明
本 Actor 只會從 Apify 伺服器造訪您提供的網址，並以誠實的使用者代理名稱（`RondelBot`）表明身分。預設遵守 robots.txt（含轉址），略過 robots.txt 拒絕已知 AI 爬蟲的頁面，略過標示為付費內容的頁面，並且對同一網站的頁面請求之間至少間隔 1 秒。本 Actor **不提供登入、cookie、自訂標頭或 proxy 功能**，也不會嘗試繞過登入、付費牆、驗證碼或其他防爬機制。
**使用方式與結果的用途由您自行負責。** 執行前，請確認您有權存取及複製所提交的網頁，且您的使用符合各網站的使用條款、著作權法與個人資料保護法規（例如 GDPR、CCPA 或我國《個人資料保護法》）。關閉 `respectRobotsTxt` 是您的決定與責任，請僅用於您擁有或已取得書面許可的網站；關閉後，本 Actor 實際處理的每個網址（包括無效網址）的結果都會標示 `robotsTxtIgnored: true` 並註明未檢查 robots.txt；重複網址，以及因 `maxUrlsPerDomain` 或費用上限而略過的網址不會被擷取，也不會帶有這個標示。關閉 `respectAiOptOut` 或 `skipPaywalledContent` 也同樣由您自行負責。
結果可能含有個人資料（例如網頁上的姓名或聯絡方式）。除非您有合法依據，請勿蒐集個人資料。您的輸入與結果只存放在您自己的 Apify 帳號中（該次執行的 dataset 與 key-value store），本 Actor 不會將其傳送給開發者或第三方。
**著作權與 AI 用途。** 本 Actor 輸出的 Markdown 是他人內容的重製物。多數文章、技術文件與部落格文章受著作權保護。未經權利人同意，請勿轉載、轉售或散布輸出內容。將輸出用於訓練或微調 AI 模型、或建立可檢索的 RAG 索引，依各國法律可能需要取得權利人授權；如有需要，請自行取得。請保留文字的來源網址與作者資訊（預設輸出開頭的 `Source:` 行可協助保留來源）。
**付費牆。** 請勿使用本 Actor 取得您無權閱讀或再利用的內容。預設的付費牆略過功能是依標記判斷，無法辨識所有付費牆。部分出版者會把付費文章全文放在網頁 HTML 中再以程式隱藏；擷取這類內容可能違反出版者條款或法律。
網站管理者可在 robots.txt 加入 `User-agent: RondelBot` 與 `Disallow: /` 拒絕本 Actor 造訪；自本 Actor 下次造訪該網站起生效。
本聲明並非法律意見；如不確定您的用途是否合法，請諮詢律師。
### Support
Open an issue on the Actor's Issues tab with the URL and the run ID.
# Actor input Schema
## `urls` (type: `array`):
Web pages to convert to Markdown, one URL per line. Each URL is fetched once; links are not followed.
## `renderMode` (type: `string`):
Auto fetches the server HTML and only opens a headless browser when the page has little or no content without JavaScript. Browser-rendered pages are charged as the separate browser-page event.
## `mainContentOnly` (type: `boolean`):
Remove navigation, headers, footers, cookie banners, sidebars and ads, and keep the article or document body. Turn off to convert the whole page.
## `includeLinks` (type: `boolean`):
Keep hyperlinks as Markdown links with absolute URLs. Turn off to get plain text with no URLs.
## `includeImages` (type: `boolean`):
Keep content images as Markdown image references (absolute URLs). Tracking pixels, spacers, icons and data: URIs are always removed.
## `maxMarkdownChars` (type: `integer`):
Cut each page's Markdown to this many characters (0 = no limit). Useful to cap LLM context size.
## `chunkSize` (type: `integer`):
0 = off. When set (200-20000), each page also gets a "chunks" array: pieces of at most this many characters, split at headings, then paragraphs, then sentence ends (。！？ and . ! ?), keeping code blocks and tables whole when they fit. Each chunk carries its heading path. Chunking does not change the price: you pay per page, not per chunk.
## `chunkOverlap` (type: `integer`):
Characters from the end of the previous chunk repeated at the start of the next one (capped at half the chunk size). 0 = no overlap.
## `minContentChars` (type: `integer`):
Pages with less text than this are reported as empty-content and not charged.
## `browserWaitMs` (type: `integer`):
When a page is rendered in a browser, wait this long after the load event for late content.
## `requestTimeoutSecs` (type: `integer`):
Maximum time to load one page (10-40 s). A page that times out is reported as failed and not charged.
## `maxConcurrency` (type: `integer`):
How many pages are processed in parallel. Requests to the same website are always spaced at least 1 second apart (or the site's robots.txt Crawl-delay).
## `respectRobotsTxt` (type: `boolean`):
Skip pages that the site's robots.txt disallows for our crawler (RondelBot). Skipped pages are not charged.
## `respectAiOptOut` (type: `boolean`):
Skip pages whose robots.txt has a rule for a known AI crawler (for example GPTBot, CCBot, Google-Extended, anthropic-ai, ClaudeBot, PerplexityBot) that disallows the page's path. Skipped pages get status "skipped-ai-crawler-opt-out" and are not charged. Turn off only if you have the right to use the content.
## `skipPaywalledContent` (type: `boolean`):
Skip pages that declare paid content (schema.org isAccessibleForFree=false, article:content_tier=locked) or contain common paywall markers (elements with class/id such as "paywall" or "regwall"). Every row has paywallDetected. Skipped pages get status "skipped-paywall" and are not charged. Detection is heuristic. When off, detected pages are converted and charged normally and keep paywallDetected=true.
## `addSourceLine` (type: `boolean`):
Start the Markdown of each converted page with one line naming the source: "Source: <title> <url> (<site name>)". Turn off to get the page content only. The line is added after maxMarkdownChars is applied.
## `maxUrlsPerDomain` (type: `integer`):
Optional limit on how many URLs of the same site (host name, without www.) are processed in one run. URLs above the limit get status "skipped-domain-limit" and are not charged. 0 or empty = no limit.
## `locale` (type: `string`):
Language used by the browser for rendered pages, e.g. zh-TW, zh-CN or en-US.
## Actor input object example
```json
{
"urls": [
"https://docs.python.org/3/tutorial/introduction.html",
"https://zh.wikipedia.org/zh-tw/臺灣鐵路"
],
"renderMode": "auto",
"mainContentOnly": true,
"includeLinks": true,
"includeImages": true,
"maxMarkdownChars": 0,
"chunkSize": 0,
"chunkOverlap": 0,
"minContentChars": 50,
"browserWaitMs": 1000,
"requestTimeoutSecs": 30,
"maxConcurrency": 10,
"respectRobotsTxt": true,
"respectAiOptOut": true,
"skipPaywalledContent": true,
"addSourceLine": true,
"locale": "en-US"
}
```
# Actor output Schema
## `pages` (type: `string`):
No description
## `summary` (type: `string`):
No description
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
token: '<YOUR_API_TOKEN>',
});
// Prepare Actor input
const input = {
"urls": [
"https://docs.python.org/3/tutorial/introduction.html",
"https://zh.wikipedia.org/zh-tw/臺灣鐵路"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("raspberry_rondel/web-to-markdown-cjk").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")
# Prepare the Actor input
run_input = { "urls": [
"https://docs.python.org/3/tutorial/introduction.html",
"https://zh.wikipedia.org/zh-tw/臺灣鐵路",
] }
# Run the Actor and wait for it to finish
run = client.actor("raspberry_rondel/web-to-markdown-cjk").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"urls": [
"https://docs.python.org/3/tutorial/introduction.html",
"https://zh.wikipedia.org/zh-tw/臺灣鐵路"
]
}' |
apify call raspberry_rondel/web-to-markdown-cjk --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,raspberry_rondel/web-to-markdown-cjk"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/d0cljCN2NPq3bXfoC/builds/6ammgFvzilvZnPvSL/openapi.json