Web Page to Markdown for LLM & RAG – Chinese, Big5 & GBK ready

Convert web pages you are allowed to use into clean Markdown for LLM and RAG pipelines and AI agents. Typical sources are your own docs, public documentation, and openly licensed or permitted content. You get the article or document body only, with absolute links, useful images and code blocks kept intact. Chinese text is decoded from the page's declared encoding, including old Big5, GBK and GB2312 sites. Optional RAG chunks come with heading paths and overlap.

You pay per page converted. Pages that fail, are blocked, are skipped, or come back garbled are free. Apify's small platform start fee is still charged for every run (see Pricing).

Typical uses

• Turning your own product documentation, help centre or blog into a knowledge base for your chatbot or RAG index.
• Preparing public documentation (for example language or framework docs) and openly licensed pages for search, summarising or internal Q&A.
• Research and internal reference on pages you are permitted to copy.
• Converting legacy Big5 or GBK pages that you own or maintain into clean, readable Markdown.

What it does

• Main content only. Navigation, headers, footers, cookie banners, sidebars, "related articles" lists and share buttons are removed. You can turn this off to convert the whole page.
• Links you can use. Links are made absolute (https://… or http://…) and kept on one line. In an internal test (2026-10-17, build 0.2.6, 24 converted pages out of 30), our output had 1,672 Markdown links, none relative or broken across lines; one of them had an empty target. Same-page anchors, "edit" links and heading permalinks (¶) are dropped.
• Images without junk. Content images are kept as ![alt](https://…). Tracking pixels, spacers, icons, data: URIs and duplicate sizes of the same image are removed.
• Chinese encodings handled. Big5 (including the ETEN extensions), GBK/GB2312 (read as GB18030) and UTF-8 are detected from the HTTP header and the page's <meta charset>, even when the two disagree. Traditional and Simplified Chinese stay exactly as the site wrote them; nothing is converted.
• Code blocks preserved. <pre> blocks become fenced code blocks with their line breaks kept, including code inside list items. Inline code stays inline.
• Wikipedia-style infoboxes become a compact - **Key**: value list instead of scattered fragments.
• Fast and cheap by default. Each page is fetched over plain HTTP first. In auto mode, a headless Chromium browser is used only when the server HTML is a JavaScript app shell with little text (for example an empty <div id="root"> or a <noscript> asking for JavaScript), or when the plain HTTP connection fails (see below). Short pages without such signals are converted from the server HTML and charged as a normal page. If rendering does not add meaningful content, the page is charged as a normal page. You can also force HTTP-only or browser rendering for every page.
• Metadata. Title, description, author, published date, site name, language, canonical URL, declared encodings.
• RAG chunks (optional). Set chunkSize to also get a chunks array per page. Pages are split at headings, then paragraphs, then sentence ends (。！？ as well as . ! ?). Code blocks and tables stay whole when they fit. Each chunk carries its heading path, and you can add an overlap.

Responsible crawling (what we do not do)

• robots.txt is respected for every URL and every redirect (crawler name RondelBot). Disallowed pages are skipped and free. If a site's robots.txt cannot be fetched because of a server error, rate limit or timeout, the site is treated as disallowed. You can switch this off with respectRobotsTxt only if you have the site owner's permission. If you do, the row of each URL the Actor processes (including invalid URLs) is marked robotsTxtIgnored: true with a note that robots.txt was not checked. Rows for duplicates and for URLs skipped because of maxUrlsPerDomain or the spending limit are not fetched and do not carry this flag.
• robots.txt is evaluated for RondelBot, or for * when there is no RondelBot group. Site owners can opt out with User-agent: RondelBot / Disallow: /.
• AI-crawler opt-outs are respected by default (respectAiOptOut). If a site's robots.txt has a group for a known AI crawler (for example GPTBot, CCBot, Google-Extended, anthropic-ai, ClaudeBot or PerplexityBot) that disallows the page's path, the page is skipped: status skipped-ai-crawler-opt-out, free. This is checked for the URL you give and again after redirects. Only groups that name such a crawler count; a missing robots.txt means no opt-out. The list of crawler names is ours and may not cover every AI crawler, so check the site's rules yourself if your use involves AI.
• Paywalled pages are skipped by default (skipPaywalledContent). See "Paywalls" under Tips & limits.
• No proxies, no disguise, no anti-bot bypass. We use an honest user agent. If a site shows a bot challenge or blocks cloud servers, the page is reported as blocked-by-anti-bot and not charged; it is not retried in a browser to get around the block.
• If the plain HTTP request fails with a connection error, the page is tried again in a normal headless browser (same honest user agent, same robots.txt rules). Anti-bot challenges are not retried.
• Page requests to the same site are spaced at least 1 second apart, or more if the site's robots.txt Crawl-delay asks for it. Pages whose robots.txt asks for a Crawl-delay above 30 seconds are skipped (status blocked-by-robots-txt, free), in both plain HTTP and browser mode.
• On HTTP 429 or 503, the Actor waits and retries once only if the site's Retry-After is 30 seconds or less; otherwise it backs off and the page is free.
• You cannot supply cookies, headers or credentials. Cookies that a site sets during a run may be kept for the rest of that run, as with an ordinary HTTP client; this is not a login feature.
• Private and internal network addresses are refused.

Input

Field	Default	What it does
urls	–	Pages to convert, one URL per line. Links are not followed.
renderMode	auto	auto: HTTP first; browser only when the server HTML is a JavaScript app shell with little text, or after a connection failure. http: no browser. browser: render every page in the browser.
mainContentOnly	true	Remove boilerplate and keep the main content.
includeLinks / includeImages	true	Keep links and images as Markdown.
chunkSize	0 (off)	200–20,000 characters. Adds a chunks array per page.
chunkOverlap	0	Characters repeated from the end of the previous chunk (at most half the chunk size).
maxMarkdownChars	0 (no limit)	Cut each page's Markdown to this length. The source line (see addSourceLine) is added afterwards, so the output can be slightly longer.
minContentChars	50	Pages with less text are reported as empty-content and are free.
browserWaitMs	1000	Extra wait after load for rendered pages (max 5,000).
requestTimeoutSecs	30	Per-page timeout (10–40 s).
maxConcurrency	10	Pages processed in parallel (1–20).
respectRobotsTxt	true	See above.
locale	en-US	Browser language for rendered pages, e.g. zh-TW.
respectAiOptOut	true	Skip a page when the site's robots.txt has a group for a known AI crawler (for example GPTBot, CCBot, Google-Extended, anthropic-ai, ClaudeBot, PerplexityBot) that disallows the page's path, also after redirects. Status skipped-ai-crawler-opt-out, aiOptOut: true, free.
skipPaywalledContent	true	Skip a page that declares paid content (schema.org isAccessibleForFree: false, article:content_tier = locked) or contains a common paywall marker (an element whose class or id is, for example, paywall or regwall). Every fetched page has paywallDetected. Status skipped-paywall, free. Heuristic.
addSourceLine	true	Start each converted page's Markdown with one line: Source: <title> <url> (<site name>).
maxUrlsPerDomain	0 (no limit)	At most this many URLs per site (host name without www.) per run. The rest get status skipped-domain-limit, free.

{ "urls": ["https://docs.python.org/3/library/asyncio-task.html", "https://zh.wikipedia.org/zh-tw/臺灣鐵路"],
  "renderMode": "auto", "chunkSize": 800, "chunkOverlap": 100 }

Output

One dataset row per URL (shortened example from a Big5 page):

{
  "url": "https://www.webpage.idv.tw/maillist/maillist3/new/10/10.htm",
  "status": "ok",
  "httpStatus": 200,
  "renderedWithBrowser": false,
  "title": "網頁設計中 HTML 常犯的幾個錯誤",
  "encoding": { "header": null, "meta": "big5" },
  "markdown": "Source: 網頁設計中 HTML 常犯的幾個錯誤 <https://www.webpage.idv.tw/maillist/maillist3/new/10/10.htm> (webpage.idv.tw)\n\n# 網頁設計中 HTML 常犯的幾個錯誤\n\n壹、前言：\n\n在我學習超過十年的 HTML 標籤歲月裡，不論是網頁製作或教學中，經常看到一些不應該發生的 HTML 錯誤。\n\n…",
  "stats": { "markdownChars": 3130, "textChars": 2784, "cjkChars": 2255, "links": 0, "images": 0, "extraction": "main-content" },
  "quality": { "replacementChars": 0, "mojibakeHits": 0, "badRatio": 0.0 },
  "paywallDetected": false,
  "aiOptOut": false,
  "charged": true,
  "chargedEvent": "page",
  "timings": { "httpSecs": 1.4 }
}

Each row also carries url, canonicalUrl, author and siteName where the page provides them, so you can keep the source with the text. By default the Markdown itself also starts with a Source: line (title, URL and site name).

With chunkSize set, OK rows also contain:

"chunks": [
  { "index": 1, "chars": 813, "headings": ["Coroutines and tasks", "Coroutines"],
    "text": "To actually run a coroutine, asyncio provides the following mechanisms:\n\n- The [`asyncio.run()`](https://docs.python.org/3/library/asyncio-runner.html#asyncio.run) function …\n\n```\nimport asyncio\nimport time\n…\n```" }
]

Status values: ok (charged) · blocked-by-robots-txt · blocked-by-anti-bot · rate-limited · http-error · unreachable · not-html (PDFs, images) · empty-content · failed (including failureReason: "garbled-text") · error · invalid-url · skipped-spending-limit · skipped-ai-crawler-opt-out · skipped-paywall · skipped-domain-limit · duplicate.

• duplicate means the same URL appeared earlier in your list after removing #fragment, tracking parameters (such as utm_*, fbclid, gclid), host case and the default port. It is not fetched; the row has duplicateOf.
• skipped-domain-limit means the site already reached maxUrlsPerDomain in this run.
• unreachable means the URL is well-formed but the site could not be reached: the domain does not exist, the DNS lookup failed, or the connection failed.
• invalid-url means the URL itself is malformed, not http(s), or points to a private/local address.
• rate-limited means the site answered 429/503 with a Retry-After above 30 seconds.

Only ok rows are charged. Every row has a charged flag and an error text explaining what happened. A run summary is stored in the key-value store as RUN_SUMMARY.

Pricing (pay per event)

Event	Price	When
page	$0.001 per page ($1 per 1,000)	Page converted from the server HTML (no browser needed)
browser-page	$0.004 per page ($4 per 1,000)	Page that was rendered in a headless browser and returned from the browser version (JavaScript app shell with little text in auto mode, browser mode, or a browser retry after a connection failure)
Actor start (apify-actor-start, Apify's standard start event)	$0.0005 per GB of memory (runs use 1 GB, so $0.0005 per run)	At the start of each run, including runs where every URL is invalid, skipped or free

A page is charged either page or browser-page, not both. In auto mode the browser is used only when the server HTML is a JavaScript app shell with little text (for example an empty <div id="root"> or a <noscript> asking for JavaScript), or when the plain HTTP connection failed. Short pages without such signals are converted from the server HTML and charged as a normal page. If rendering does not add meaningful content (at least 20% and at least 200 characters more Markdown than the server-HTML version), the server-HTML version is returned and charged as a normal page.

Free, not charged:

• pages disallowed by robots.txt;
• anti-bot challenges and blocks;
• HTTP 4xx/5xx errors, rate-limited pages, timeouts and network errors;
• non-HTML files;
• pages with almost no text (empty-content);
• garbled pages;
• pages skipped because of an AI-crawler opt-out (skipped-ai-crawler-opt-out) or a paywall (skipped-paywall);
• URLs over maxUrlsPerDomain (skipped-domain-limit) and duplicate URLs (duplicate).

The only charge that applies even when every page is free is Apify's platform start fee (US$0.0005 per GB of memory; runs use 1 GB, so US$0.0005 per run). It is charged even if every input URL is invalid.

A page counts as garbled when the converted text contains many U+FFFD replacement characters or mojibake patterns, for example UTF-8 read as Latin-1, GBK/Big5 read as Latin-1, or broken private-use glyphs. Such a page gets status: "failed", failureReason: "garbled-text" and charged: false; the Markdown is still included so you can inspect it.

Chunking does not change the price: you pay per page, not per chunk. Set Maximum cost per run in the run options and the Actor stops before exceeding it; skipped URLs are marked and free.

Measured results in our test

Internal test, 2026-10-17, version 0.2.6 with the current defaults (Source: line, AI-crawler opt-out skip, paywall skip). 30 pages: 15 Chinese (Traditional/Taiwan, Simplified, one Big5, one GBK, one GB2312) and 15 international (docs, blogs, news, wiki, e-commerce). Results may differ for other pages, settings, versions or dates.

• Pages converted: 24 of 30. The other 6 were skipped (free) because the site's robots.txt disallows AI crawlers for that path (status skipped-ai-crawler-opt-out). No page was skipped as paywalled.
• Legacy Chinese encodings: the Big5, GBK and GB2312 pages were converted with 0 U+FFFD replacement characters. Across all converted pages our checker found 0 replacement characters and 0 mojibake patterns.
• Links: on the 24 converted pages, our output had 1,672 Markdown links (counted by our link checker in the returned Markdown): 1,585 https://, 86 http:// and 1 with an empty target ([asyncio.AbstractEventLoop]() on a docs.python.org page). None were relative, broken across lines or same-page anchors. The per-page stats.links field adds up to 1,664 for the same pages, because it does not count the 8 links whose text contains square brackets or whose target is empty.
• Main content: the 24 converted pages produced 457,132 Markdown characters in total (including about 2,600 characters of Source: lines).
• JavaScript page: one JavaScript-only category page was rendered in a browser by auto mode and returned the page text (charged as browser-page).

How we measured. The URL list was fixed on 2026-10-17 and not changed. The test was one run on the Apify platform with only the 30 URLs as input, so every default was on (renderMode: auto, respectAiOptOut, skipPaywalledContent, addSourceLine). The output was scored with our own script and link checker. The 6 pages skipped for AI-crawler opt-outs were on twreporter.org, ithome.com.tw, cna.com.tw, github.com, bbc.com and theverge.com.

Tips & limits

• Single pages only; it does not crawl or follow links. Use it after a crawler or sitemap step.
• PDFs and Office files are reported as not-html (free). Use a document converter for those.
• Some e-commerce product pages contain very little text in their HTML. The output is then short but honest.
• Sites that block cloud servers will show up as blocked-by-anti-bot, at no cost.
• Paywalls. The Actor has no login, cookie, header or proxy options and does not try to get past paywalls. With skipPaywalledContent on (the default), a page is skipped (status skipped-paywall, not converted, free) when it declares paid content (schema.org isAccessibleForFree: false, article:content_tier = locked) or contains a common paywall marker. The page's HTML is still downloaded to check this. Every fetched page has paywallDetected. Detection is heuristic: metered paywalls and paywalls without such markers are not detected. Some publishers include full articles in their HTML and hide them with scripts; you are responsible for not using the Actor to obtain content you are not entitled to read or reuse.
• Runs use 1 GB of memory; for large lists keep maxConcurrency at the default.

Responsible use and legal notice

This Actor only visits the URLs you give it, from Apify's servers, with an honest user agent (RondelBot). By default it obeys robots.txt (including redirects), skips pages whose robots.txt opts out of known AI crawlers, skips pages marked as paywalled, and waits at least 1 second between page requests to the same site. It has no login, cookie, header or proxy options and does not try to get past logins, paywalls, CAPTCHAs or other anti-bot protections.

You are responsible for how you use it and for what you do with the results. Before running it, make sure you have the right to access and copy the pages you submit, and that your use complies with the websites' terms, copyright law and data-protection law (for example GDPR, CCPA, or Taiwan's Personal Data Protection Act). Turning off respectRobotsTxt is your decision and your responsibility; do it only for sites you own or have written permission to access. The same applies to turning off respectAiOptOut or skipPaywalledContent.

Results can contain personal data (for example names or contact details shown on a page). Do not collect personal data unless you have a lawful basis to do so. Your inputs and results are stored only in your own Apify account (the run's dataset and key-value store); the Actor does not send them to the developer or to third parties.

Copyright and AI use. The Markdown output is a copy of other people's content. Most articles, documentation and blog posts are protected by copyright. Do not republish, resell or redistribute the output without the owner's permission. Using it to train or fine-tune AI models, or to build a searchable RAG index, may require the rights holder's permission depending on your country; you are responsible for obtaining the permission you need. Keep the source URL and author with the text (the default Source: line helps with this).

Paywalls. Do not use this Actor to obtain content you are not entitled to read or reuse. The default paywall skip is heuristic and does not catch every paywall. Some publishers include the full text of paywalled articles in their HTML and hide it with scripts; collecting such content may breach the publisher's terms or the law.

Site owners can opt out by adding User-agent: RondelBot and Disallow: / to their robots.txt; the change applies from the Actor's next visit to the site.

This notice is not legal advice. If you are unsure whether your use is lawful, consult a lawyer.

負責任使用與法律聲明

本 Actor 只會從 Apify 伺服器造訪您提供的網址，並以誠實的使用者代理名稱（RondelBot）表明身分。預設遵守 robots.txt（含轉址），略過 robots.txt 拒絕已知 AI 爬蟲的頁面，略過標示為付費內容的頁面，並且對同一網站的頁面請求之間至少間隔 1 秒。本 Actor 不提供登入、cookie、自訂標頭或 proxy 功能，也不會嘗試繞過登入、付費牆、驗證碼或其他防爬機制。

使用方式與結果的用途由您自行負責。 執行前，請確認您有權存取及複製所提交的網頁，且您的使用符合各網站的使用條款、著作權法與個人資料保護法規（例如 GDPR、CCPA 或我國《個人資料保護法》）。關閉 respectRobotsTxt 是您的決定與責任，請僅用於您擁有或已取得書面許可的網站；關閉後，本 Actor 實際處理的每個網址（包括無效網址）的結果都會標示 robotsTxtIgnored: true 並註明未檢查 robots.txt；重複網址，以及因 maxUrlsPerDomain 或費用上限而略過的網址不會被擷取，也不會帶有這個標示。關閉 respectAiOptOut 或 skipPaywalledContent 也同樣由您自行負責。

結果可能含有個人資料（例如網頁上的姓名或聯絡方式）。除非您有合法依據，請勿蒐集個人資料。您的輸入與結果只存放在您自己的 Apify 帳號中（該次執行的 dataset 與 key-value store），本 Actor 不會將其傳送給開發者或第三方。

著作權與 AI 用途。 本 Actor 輸出的 Markdown 是他人內容的重製物。多數文章、技術文件與部落格文章受著作權保護。未經權利人同意，請勿轉載、轉售或散布輸出內容。將輸出用於訓練或微調 AI 模型、或建立可檢索的 RAG 索引，依各國法律可能需要取得權利人授權；如有需要，請自行取得。請保留文字的來源網址與作者資訊（預設輸出開頭的 Source: 行可協助保留來源）。

付費牆。 請勿使用本 Actor 取得您無權閱讀或再利用的內容。預設的付費牆略過功能是依標記判斷，無法辨識所有付費牆。部分出版者會把付費文章全文放在網頁 HTML 中再以程式隱藏；擷取這類內容可能違反出版者條款或法律。

網站管理者可在 robots.txt 加入 User-agent: RondelBot 與 Disallow: / 拒絕本 Actor 造訪；自本 Actor 下次造訪該網站起生效。

本聲明並非法律意見；如不確定您的用途是否合法，請諮詢律師。

Support

Open an issue on the Actor's Issues tab with the URL and the run ID.