Web to Markdown — Clean, LLM-Ready Markdown from Web Pages avatar

Web to Markdown — Clean, LLM-Ready Markdown from Web Pages

Pricing

from $3.00 / 1,000 page converteds

Go to Apify Store
Web to Markdown — Clean, LLM-Ready Markdown from Web Pages

Web to Markdown — Clean, LLM-Ready Markdown from Web Pages

Pricing

from $3.00 / 1,000 page converteds

Rating

0.0

(0)

Developer

Abdelkader BENAISSI

Abdelkader BENAISSI

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

a month ago

Last modified

Categories

Features

Limited permissionsSingle-tenantServer interface

Share

Web to Markdown (LLM-Ready)

Turn one URL, a documentation site, or a sitemap into clean Markdown for LLMs, RAG pipelines, and knowledge bases — with metadata and optional token-aware chunks.

Convert any web page to clean Markdown for $0.003 per page (static) or $0.004 (browser-rendered), plus platform usage of roughly $0.50–0.70 per 1,000 pages measured — or call it as a live HTTP API from standby.

Not for: login walls, CAPTCHAs, or anti-bot-protected news sites.

Value proposition: one predictable dataset row per page: markdown, metadata, optional chunks, and an error field. Failed URLs are isolated, so one bad page does not stop the run. You pay only for successfully converted pages.

The Actor is built on the Python web2md library in this repository: HTTP fetching, optional Playwright rendering, boilerplate removal, Markdown conversion, robots.txt, rate limits, retries, and whole-site crawling.


Three use cases

1. RAG over documentation and knowledge bases

Crawl a docs site, keep the main content, and get tiktoken cl100k_base chunks with overlap and token counts. Embed the chunks[].text values directly instead of re-chunking HTML in your ingestion pipeline.

2. Clean context for LLM agents and research

Convert articles, news pages, forum threads, product pages, and PDFs to Markdown with title, language, source URL, word count, and token count. Feed the Markdown to an LLM or store it as a knowledge file without navigation, ads, cookie banners, or inline SVG noise. PDFs are split into ## Page N sections with a page_count metadata field.

3. Content migration and site auditing

Crawl a blog or documentation site into a dataset of Markdown pages. Use the url, title, metadata, and error fields to audit missing pages, compare content, or seed a static-site migration. Failed URLs remain visible as their own rows.


Quick start

Input

{
"startUrls": [
{"url": "https://docs.apify.com/platform/actors/development"}
],
"crawlMode": "single",
"maxPages": 10,
"renderingMode": "auto",
"extractMainContentOnly": true,
"includeLinks": true,
"includeImages": false,
"outputFormat": "markdown+metadata",
"chunking": {
"enabled": true,
"maxTokens": 800,
"overlap": 100
},
"respectRobotsTxt": true,
"proxyConfiguration": {"useApifyProxy": false},
"maxConcurrency": 10,
"requestTimeoutSecs": 30,
"maxRetries": 3
}
  • crawlMode: single, site, or sitemap.
  • renderingMode: auto (static first, Playwright fallback), static, or browser (always Playwright).
  • chunking: optional; maxTokens is a hard cap using cl100k_base.
  • proxyConfiguration: optional Apify Proxy. Disabled by default.

Output

Each dataset row has this shape (trimmed):

{
"url": "https://docs.python.org/3/library/asyncio-task.html",
"title": "Coroutines and tasks",
"markdown": "---\ntitle: Coroutines and tasks\nlanguage: en\nsource: https://docs.python.org/3/library/asyncio-task.html\n---\n\n# Coroutines and tasks\n\nThis section outlines ...",
"metadata": {
"title": "Coroutines and tasks",
"language": "en",
"source": "https://docs.python.org/3/library/asyncio-task.html",
"word_count": 8242,
"wordCount": 8242,
"tokenCount": 14342,
"tokenizer": "cl100k_base",
"rendered": false,
"statusCode": 200
},
"chunks": [
{
"index": 0,
"text": "# Coroutines and tasks\n\nThis section outlines ...",
"tokenCount": 211,
"wordCount": 64,
"overlapTokens": 0
}
],
"chunkCount": 22,
"wordCount": 8242,
"tokenCount": 14342,
"rendered": false,
"error": null
}

chunks is [] when chunking is disabled. Chunk text excludes YAML front matter. A failed page still gets a row, with error set, empty Markdown, and zero counts.

Pages with fewer than 50 words, or pages that match a known anti-bot challenge phrase, are still published but carry metadata.lowContent: true, metadata.lowContentReason, and metadata.warnings: ["lowContent"]. They are not charged a page-conversion event. Examples include eBay's “Pardon Our Interruption” splash and Cloudflare-style “verify you are human” pages.


Example inputs

A documentation page — returned as clean, structured Markdown:

{
"startUrls": [{"url": "https://docs.apify.com/platform/actors/development"}],
"crawlMode": "single"
}

A JavaScript-rendered page — browser mode renders it before converting:

{
"startUrls": [{"url": "https://quotes.toscrape.com/js/"}],
"crawlMode": "single",
"renderingMode": "browser"
}

An Arabic article — returned as clean article text without navigation or layout junk:

{
"startUrls": [{"url": "https://ar.wikipedia.org/wiki/الذكاء_الاصطناعي"}],
"crawlMode": "single"
}

Measured performance

These are local numbers on one docs site, not an Apify platform benchmark. Measured on 2026-10-20 on 4 vCPU / 7.8 GiB RAM, Python 3.12.3, with 100 URLs from https://docs.apify.com/sitemap.xml, no proxy, chunking off. Peak RSS is the Python process tree.

ModeURLsFailuresRenderedWall timesec/pagePeak RSS
static1000059.39 s0.59177.1 MB
auto1000057.30 s0.57181.6 MB
browser1000100252.83 s2.531,517.1 MB

Browser mode was 4.26x slower and used 8.57x more peak memory than static mode. Auto rendered 0 pages on this static/SSR docs set, so it behaved like static here; it is not proof for JavaScript-heavy sites. The full method, URL set, and before/after examples are in BENCHMARK.md.

On a separate fixed 30-URL mixed benchmark with the default 30 s timeout and 3 retries, the Actor converted 20/30 pages and failed 10/30 (403 bot blocks, robots.txt blocks and genuine 404s), rendering 6 pages in 35.1 s. Two converted rows were low-content (eBay challenge, 5-word PDF) and were exempted from charge events. The failures remain in the dataset and are listed by cause in BENCHMARK.md; they were not replaced.

Platform verification (2026-10-20, live Actor, settled usage): 100 static Wikipedia articles converted in 14.5 min for $0.0493 platform usage with all 100 pages charged; 20 browser-rendered pages for $0.0142. The local numbers above remain a baseline, not a capacity guarantee.


Pricing

The Actor uses pay-per-event (PPE) pricing. The prices are configured in Apify Console, not in the source code:

EventWhen it is chargedLive price
page-convertedSuccessful static (non-browser) page$0.003 / page ($3.00 per 1,000)
browser-page-convertedSuccessful Playwright-rendered page$0.004 / page ($4.00 per 1,000)
  • Failed pages, skipped pages, error rows, and lowContent challenge/thin pages are not charged.
  • Chunking is included; there is no separate chunking event.
  • At these prices, 100 successful static pages cost $0.30 in PPE and 100 browser pages cost $0.40, before Apify platform usage.
  • Platform usage is paid by the user running the Actor (pay-per-event terms). Measured on the platform (2026-10-20, very large Wikipedia articles): ~$0.49 of platform usage per 1,000 static pages and ~$0.68 per 1,000 browser pages, excluding run startup. Typical pages are far smaller and cost less.
  • If a spending limit is reached, the Actor stops publishing/charging new pages and exits cleanly. A page that was charged successfully is published.

If you change the prices, keep the event names exactly as above; the code calls those names.


Input reference

InputDefaultNotes
startUrlsrequiredOne or more URLs. In site mode the first is the seed; in sitemap mode its domain is used.
crawlModesinglesingle, site, or sitemap.
maxPages10Hard cap, 1–10,000.
maxDepth2Link hops in site mode, 0–10.
includeGlobs / excludeGlobs[]URL glob filters for site crawling.
renderingModeautostatic, auto, or browser.
extractMainContentOnlytrueRemove navigation, sidebars, footers, cookie banners, ads.
includeLinkstrueKeep absolute Markdown links.
includeImagesfalseKeep absolute Markdown images.
outputFormatmarkdown+metadatamarkdown or markdown+metadata.
chunking.enabledfalseEnable RAG chunks.
chunking.maxTokens800Hard token cap per chunk.
chunking.overlap100Token overlap; must be smaller than maxTokens.
respectRobotsTxttrueKeep on unless you have explicit permission.
proxyConfigurationdisabledOptional Apify Proxy.
maxConcurrency101–50; browser concurrency is capped lower internally.
requestTimeoutSecs30Per-page timeout, 5–120 s.
maxRetries3Transient network/HTTP retries, 0–10.

Integrations

All snippets call this Actor as abdou_009/apify-web-to-markdown.

Python (apify-client)

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_API_TOKEN"])
run = client.actor("abdou_009/apify-web-to-markdown").call(
run_input={
"startUrls": [{"url": "https://docs.apify.com/platform/actors/development"}],
"crawlMode": "single",
"renderingMode": "auto",
"chunking": {"enabled": True, "maxTokens": 800, "overlap": 100},
}
)
items = client.dataset(run["defaultDatasetId"]).list_items().items
for page in items:
print(page["url"], page["tokenCount"], page["chunkCount"])
for chunk in page["chunks"]:
print(chunk["index"], chunk["tokenCount"], chunk["text"][:80])

JavaScript (apify-client)

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_API_TOKEN });
const run = await client
.actor('abdou_009/apify-web-to-markdown')
.call({
startUrls: [{ url: 'https://docs.apify.com/platform/actors/development' }],
crawlMode: 'single',
renderingMode: 'auto',
chunking: { enabled: true, maxTokens: 800, overlap: 100 },
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const page of items) {
console.log(page.url, page.tokenCount, page.chunkCount);
}

n8n (HTTP Request node)

Import this minimal workflow and set APIFY_TOKEN in the n8n environment. The workflow already targets abdou_009/apify-web-to-markdown.

{
"name": "Web to Markdown",
"nodes": [
{
"parameters": {
"method": "POST",
"url": "https://api.apify.com/v2/acts/abdou_009~apify-web-to-markdown/run-sync-get-dataset-items",
"sendQuery": true,
"queryParameters": {
"parameters": [
{"name": "token", "value": "={{ $env.APIFY_TOKEN }}"}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={\n \"startUrls\": [{\"url\": \"https://docs.apify.com/platform/actors/development\"}],\n \"crawlMode\": \"single\",\n \"renderingMode\": \"auto\",\n \"chunking\": {\"enabled\": true, \"maxTokens\": 800, \"overlap\": 100}\n}",
"options": {"timeout": 300000}
},
"name": "Convert to Markdown",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [0, 0]
}
],
"connections": {}
}

run-sync-get-dataset-items is convenient for small jobs. For large crawls or browser runs that may exceed the synchronous limit, start an async run and poll the run/dataset endpoints instead.

LangChain

import os
from langchain_community.utilities import ApifyWrapper
from langchain_core.documents import Document
os.environ["APIFY_API_TOKEN"] = "YOUR_APIFY_TOKEN"
apify = ApifyWrapper()
loader = apify.call_actor(
actor_id="abdou_009/apify-web-to-markdown",
run_input={
"startUrls": [{"url": "https://docs.apify.com/platform/actors/development"}],
"crawlMode": "single",
"renderingMode": "auto",
"chunking": {"enabled": True, "maxTokens": 800, "overlap": 100},
},
dataset_mapping_function=lambda item: Document(
page_content=item["markdown"],
metadata={
"source": item["url"],
"title": item["title"],
"token_count": item["tokenCount"],
},
),
)
documents = loader.load()

The Actor already returns chunks. If you want one LangChain Document per chunk, read item["chunks"] from the dataset and map each chunk's text and index instead of using the whole markdown.


Standby API (HTTP)

The Actor is Standby-enabled (usesStandbyMode: true and an OpenAPI schema in .actor/web_server_schema.json). After it is deployed, the Endpoints tab shows the hostname. Authentication is the normal Apify API token:

curl -H "Authorization: Bearer $APIFY_TOKEN" \
"https://YOUR_STANDBY_HOST/convert?url=https%3A%2F%2Fexample.com&chunking=true"

The response is one dataset-row-shaped object:

{
"url": "https://example.com/",
"title": "Example Domain",
"markdown": "---\ntitle: Example Domain\n...\n\n# Example Domain\n...",
"metadata": {"language": "en", "tokenCount": 28, "tokenizer": "cl100k_base"},
"chunks": [{"index": 0, "text": "...", "tokenCount": 28, "wordCount": 25, "overlapTokens": 0}],
"chunkCount": 1,
"wordCount": 25,
"tokenCount": 28,
"rendered": false,
"error": null
}
Query parameterDefaultMeaning
urlrequiredOne absolute public http(s) URL.
renderingModeautoauto, static, or browser.
chunkingfalseReturn chunks using cl100k_base.
maxTokens800Hard chunk cap; 50–8,000.
overlap100Chunk overlap; smaller than maxTokens.
includeLinkstrueKeep absolute links.
includeImagesfalseKeep absolute images.

Other behavior:

  • GET /health returns {"status": "ok"}; the platform readiness probe is answered on /.
  • Missing/invalid parameters return 400; conversion failures return 502 with an error JSON body; a PPE spending-limit rejection returns 402.
  • Successful requests use the same page-converted / browser-page-converted events as batch mode.
  • The Actor and Standby endpoint enable SSRF protection: hostnames are resolved before the request, every DNS answer must be public, and each redirect hop is resolved and checked again. Loopback, private, link-local and cloud-metadata addresses are rejected. DNS rebinding between the lookup and the connection remains a residual risk—use network policy if you expose the endpoint beyond trusted callers. The Python library/CLI keeps private hosts available for local development and tests.
  • Standby v1 does not read proxyConfiguration; use batch mode when you need Apify Proxy.
  • Standby runs stay alive between requests. Configure the idle timeout and memory in the Endpoints tab; idle runs consume platform resources.

Local verification (not a platform run) returned 200 from /health, ready from the readiness probe, and a valid Markdown row from /convert for https://example.com/ and the TodoMVC React SPA.

FAQ

Does it render JavaScript?

Yes, with Playwright. browser always renders. auto starts with HTTP and only falls back when the HTML looks JavaScript-driven; the fallback is conservative, so a JS-heavy site may need renderingMode: "browser".

Does it crawl a whole site or sitemap?

Yes. site follows same-domain links with depth/page limits and include/exclude globs. sitemap discovers sitemap.xml and nested sitemap indexes.

Does it respect robots.txt?

Yes, by default, including disallow rules and crawl delays. Keep it enabled unless you have explicit permission to ignore it.

What happens when a URL fails?

The run continues. The failed URL gets its own dataset row with error set, empty Markdown, and zero counts. It is not charged a page-conversion event.

How are chunks counted?

With the tiktoken cl100k_base tokenizer, not an approximate character count. Every chunk is at or below maxTokens. Chunks prefer heading/paragraph boundaries, fall back to token windows for oversized blocks, and keep fenced code blocks valid. Arabic/RTL text is preserved in the tested cases.

Can I still use the old v1 input (urls, maxChars)?

Yes. Inputs that use the legacy urls field (one URL, or one URL per line) keep working: they are mapped to startUrls, and legacy mode restores the v1 defaults — includeImages on and maxChars 20000, which truncates each page's Markdown at 20,000 characters exactly as v1 did. maxChars is deprecated: in modern startUrls inputs it applies only when you set it explicitly; there is no default cap. Note that on the Apify platform the Console pre-fills schema defaults, so set includeImages explicitly if a legacy integration depends on it.

Is Apify Proxy supported?

The Actor passes proxyConfiguration to both HTTP and Playwright requests. The same SDK path was tested locally with a custom proxy, but a live Apify Proxy run still needs an authenticated Apify account.

Is this a replacement for Website Content Crawler?

Not yet, and no head-to-head run has been done. Website Content Crawler is much more mature and has anti-bot/session rotation, Firefox, cookies/login support, file downloads, screenshots, AI summaries, incremental crawling, and broad integrations. This Actor is smaller and adds built-in token chunks and a simple Python dataset contract. See BENCHMARK.md for the honest comparison and the blocked WCC measurement.

Can I self-host it?

Yes. The Docker entrypoint and Python library are in this repository. PPE charging is enabled only when the Actor runs with pay-per-event pricing; local runs do not charge.

Is there a page limit?

The input cap is 10,000 pages. Browser runs consume much more memory and time than static runs; start small. The largest verified local run is 100 pages.


Limits and known gaps

  • No CAPTCHA solving, login-wall bypass, PII harvesting, or anti-bot evasion. Hard bot-protected sites stay blocked: for example AP News returns HTTP 403 to this Actor direct, via Apify datacenter proxy, and via Apify residential proxy (all verified 2026-10-20). There is no anti-blocking stack; if a site blocks automated browsers, expect an error row, not a bypass.
  • PDFs with a text layer are supported up to 5 MB and 100 pages; larger PDFs produce an error row with a clear limit message. Scanned PDFs need OCR, which is not included. DOC/XLS files, screenshots, infinite scroll, and click-to-expand are not supported.
  • Anti-bot challenge pages are flagged as lowContent, not bypassed.
  • auto rendering can miss content that appears only after a delay; use browser plus a longer timeout for those sites.
  • Complex tables with rowspan/colspan can be flattened or simplified. The measured Wikipedia table now emits its caption above the table without an empty header row.
  • Permalink-only headings (for example a heading containing only ) are removed instead of producing an empty ##.
  • Browser mode measured 1.5 GB peak RSS and 2.53 s/page locally; memory and concurrency need tuning on the platform.
  • Web page responses are capped at 8 MB; larger responses produce an error row. PDFs are capped separately at 5 MB / 100 pages (see above).

Responsible use

Use this Actor for publicly available content you have the right to access and process. Respect each site's terms of service, robots.txt, rate limits, and copyright. Do not use it to bypass paywalls, logins, CAPTCHAs, or technical access controls, and do not collect personal data without a lawful basis. Crawled content remains the property of its owners; do not republish it without permission. This is not legal advice.


Python library and CLI

The same repository ships the underlying web2md Python library and CLI.

Install

pip install . # core
pip install '.[render]' # + Playwright support
python -m playwright install chromium

Requires Python 3.10+.

One page

web2md https://docs.python.org/3/library/pathlib.html
web2md https://example.com/article -o article.md
web2md https://example.com/app -o app.md --render always --wait-for "main"

Crawl a site

web2md crawl https://example.com -o site-md \
--max-pages 200 --max-depth 4 \
--rate-limit 0.5 --concurrency 5 \
--sitemap --exclude '/blog/tags/' --skip-existing

Crawl output includes manifest.json, crawl.ndjson, and crawl-index.md in addition to the Markdown files. --skip-existing resumes a crawl; --dry-run lists seeds without fetching.

Library

import asyncio
from web2md import Fetcher, PageOptions, fetch_page, html_to_markdown
markdown = html_to_markdown(open("page.html", "rb").read(), "https://example.com/page")
async def main():
async with Fetcher(respect_robots=True) as fetcher:
page = await fetch_page(
"https://example.com/article",
fetcher=fetcher,
options=PageOptions(render="auto"),
)
print(page.document)
print(page.title, page.words)
asyncio.run(main())

Development

python -m venv .venv && . .venv/bin/activate
pip install -e '.[dev]'
make test # pytest: 73 passed, 1 skipped at the last verification
make lint # ruff check + format check
make test-render # opt-in real-browser test (needs Playwright + Chromium)
make clean

The tests use a local HTTP fixture site and cover conversion, extraction, robots, sitemap parsing, end-to-end crawls, Actor input validation, proxy handling, chunking, and PPE policy.

License

MIT