Web to Markdown — Clean, LLM-Ready Markdown from Web Pages
Pricing
from $3.00 / 1,000 page converteds
Web to Markdown — Clean, LLM-Ready Markdown from Web Pages
Pricing
from $3.00 / 1,000 page converteds
Rating
0.0
(0)
Developer
Abdelkader BENAISSI
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a month ago
Last modified
Categories
Features
Share
Web to Markdown (LLM-Ready)
Turn one URL, a documentation site, or a sitemap into clean Markdown for LLMs, RAG pipelines, and knowledge bases — with metadata and optional token-aware chunks.
Convert any web page to clean Markdown for $0.003 per page (static) or $0.004 (browser-rendered), plus platform usage of roughly $0.50–0.70 per 1,000 pages measured — or call it as a live HTTP API from standby.
Not for: login walls, CAPTCHAs, or anti-bot-protected news sites.
Value proposition: one predictable dataset row per page: markdown,
metadata, optional chunks, and an error field. Failed URLs are isolated,
so one bad page does not stop the run. You pay only for successfully converted
pages.
The Actor is built on the Python web2md library in this repository: HTTP
fetching, optional Playwright rendering, boilerplate removal, Markdown
conversion, robots.txt, rate limits, retries, and whole-site crawling.
Three use cases
1. RAG over documentation and knowledge bases
Crawl a docs site, keep the main content, and get tiktoken cl100k_base
chunks with overlap and token counts. Embed the chunks[].text values directly
instead of re-chunking HTML in your ingestion pipeline.
2. Clean context for LLM agents and research
Convert articles, news pages, forum threads, product pages, and PDFs to Markdown
with title, language, source URL, word count, and token count. Feed the Markdown
to an LLM or store it as a knowledge file without navigation, ads, cookie
banners, or inline SVG noise. PDFs are split into ## Page N sections with a
page_count metadata field.
3. Content migration and site auditing
Crawl a blog or documentation site into a dataset of Markdown pages. Use the
url, title, metadata, and error fields to audit missing pages, compare
content, or seed a static-site migration. Failed URLs remain visible as their own
rows.
Quick start
Input
{"startUrls": [{"url": "https://docs.apify.com/platform/actors/development"}],"crawlMode": "single","maxPages": 10,"renderingMode": "auto","extractMainContentOnly": true,"includeLinks": true,"includeImages": false,"outputFormat": "markdown+metadata","chunking": {"enabled": true,"maxTokens": 800,"overlap": 100},"respectRobotsTxt": true,"proxyConfiguration": {"useApifyProxy": false},"maxConcurrency": 10,"requestTimeoutSecs": 30,"maxRetries": 3}
crawlMode:single,site, orsitemap.renderingMode:auto(static first, Playwright fallback),static, orbrowser(always Playwright).chunking: optional;maxTokensis a hard cap usingcl100k_base.proxyConfiguration: optional Apify Proxy. Disabled by default.
Output
Each dataset row has this shape (trimmed):
{"url": "https://docs.python.org/3/library/asyncio-task.html","title": "Coroutines and tasks","markdown": "---\ntitle: Coroutines and tasks\nlanguage: en\nsource: https://docs.python.org/3/library/asyncio-task.html\n---\n\n# Coroutines and tasks\n\nThis section outlines ...","metadata": {"title": "Coroutines and tasks","language": "en","source": "https://docs.python.org/3/library/asyncio-task.html","word_count": 8242,"wordCount": 8242,"tokenCount": 14342,"tokenizer": "cl100k_base","rendered": false,"statusCode": 200},"chunks": [{"index": 0,"text": "# Coroutines and tasks\n\nThis section outlines ...","tokenCount": 211,"wordCount": 64,"overlapTokens": 0}],"chunkCount": 22,"wordCount": 8242,"tokenCount": 14342,"rendered": false,"error": null}
chunks is [] when chunking is disabled. Chunk text excludes YAML front
matter. A failed page still gets a row, with error set, empty Markdown, and
zero counts.
Pages with fewer than 50 words, or pages that match a known anti-bot challenge
phrase, are still published but carry metadata.lowContent: true,
metadata.lowContentReason, and metadata.warnings: ["lowContent"]. They are
not charged a page-conversion event. Examples include eBay's “Pardon Our
Interruption” splash and Cloudflare-style “verify you are human” pages.
Example inputs
A documentation page — returned as clean, structured Markdown:
{"startUrls": [{"url": "https://docs.apify.com/platform/actors/development"}],"crawlMode": "single"}
A JavaScript-rendered page — browser mode renders it before converting:
{"startUrls": [{"url": "https://quotes.toscrape.com/js/"}],"crawlMode": "single","renderingMode": "browser"}
An Arabic article — returned as clean article text without navigation or layout junk:
{"startUrls": [{"url": "https://ar.wikipedia.org/wiki/الذكاء_الاصطناعي"}],"crawlMode": "single"}
Measured performance
These are local numbers on one docs site, not an Apify platform
benchmark. Measured on 2026-10-20 on 4 vCPU / 7.8 GiB RAM, Python 3.12.3, with
100 URLs from https://docs.apify.com/sitemap.xml, no proxy, chunking off.
Peak RSS is the Python process tree.
| Mode | URLs | Failures | Rendered | Wall time | sec/page | Peak RSS |
|---|---|---|---|---|---|---|
| static | 100 | 0 | 0 | 59.39 s | 0.59 | 177.1 MB |
| auto | 100 | 0 | 0 | 57.30 s | 0.57 | 181.6 MB |
| browser | 100 | 0 | 100 | 252.83 s | 2.53 | 1,517.1 MB |
Browser mode was 4.26x slower and used 8.57x more peak memory than static mode. Auto rendered 0 pages on this static/SSR docs set, so it behaved like static here; it is not proof for JavaScript-heavy sites. The full method, URL set, and before/after examples are in BENCHMARK.md.
On a separate fixed 30-URL mixed benchmark with the default 30 s timeout and 3 retries, the Actor converted 20/30 pages and failed 10/30 (403 bot blocks, robots.txt blocks and genuine 404s), rendering 6 pages in 35.1 s. Two converted rows were low-content (eBay challenge, 5-word PDF) and were exempted from charge events. The failures remain in the dataset and are listed by cause in BENCHMARK.md; they were not replaced.
Platform verification (2026-10-20, live Actor, settled usage): 100 static Wikipedia articles converted in 14.5 min for $0.0493 platform usage with all 100 pages charged; 20 browser-rendered pages for $0.0142. The local numbers above remain a baseline, not a capacity guarantee.
Pricing
The Actor uses pay-per-event (PPE) pricing. The prices are configured in Apify Console, not in the source code:
| Event | When it is charged | Live price |
|---|---|---|
page-converted | Successful static (non-browser) page | $0.003 / page ($3.00 per 1,000) |
browser-page-converted | Successful Playwright-rendered page | $0.004 / page ($4.00 per 1,000) |
- Failed pages, skipped pages, error rows, and
lowContentchallenge/thin pages are not charged. - Chunking is included; there is no separate chunking event.
- At these prices, 100 successful static pages cost $0.30 in PPE and 100 browser pages cost $0.40, before Apify platform usage.
- Platform usage is paid by the user running the Actor (pay-per-event terms). Measured on the platform (2026-10-20, very large Wikipedia articles): ~$0.49 of platform usage per 1,000 static pages and ~$0.68 per 1,000 browser pages, excluding run startup. Typical pages are far smaller and cost less.
- If a spending limit is reached, the Actor stops publishing/charging new pages and exits cleanly. A page that was charged successfully is published.
If you change the prices, keep the event names exactly as above; the code calls those names.
Input reference
| Input | Default | Notes |
|---|---|---|
startUrls | required | One or more URLs. In site mode the first is the seed; in sitemap mode its domain is used. |
crawlMode | single | single, site, or sitemap. |
maxPages | 10 | Hard cap, 1–10,000. |
maxDepth | 2 | Link hops in site mode, 0–10. |
includeGlobs / excludeGlobs | [] | URL glob filters for site crawling. |
renderingMode | auto | static, auto, or browser. |
extractMainContentOnly | true | Remove navigation, sidebars, footers, cookie banners, ads. |
includeLinks | true | Keep absolute Markdown links. |
includeImages | false | Keep absolute Markdown images. |
outputFormat | markdown+metadata | markdown or markdown+metadata. |
chunking.enabled | false | Enable RAG chunks. |
chunking.maxTokens | 800 | Hard token cap per chunk. |
chunking.overlap | 100 | Token overlap; must be smaller than maxTokens. |
respectRobotsTxt | true | Keep on unless you have explicit permission. |
proxyConfiguration | disabled | Optional Apify Proxy. |
maxConcurrency | 10 | 1–50; browser concurrency is capped lower internally. |
requestTimeoutSecs | 30 | Per-page timeout, 5–120 s. |
maxRetries | 3 | Transient network/HTTP retries, 0–10. |
Integrations
All snippets call this Actor as abdou_009/apify-web-to-markdown.
Python (apify-client)
import osfrom apify_client import ApifyClientclient = ApifyClient(os.environ["APIFY_API_TOKEN"])run = client.actor("abdou_009/apify-web-to-markdown").call(run_input={"startUrls": [{"url": "https://docs.apify.com/platform/actors/development"}],"crawlMode": "single","renderingMode": "auto","chunking": {"enabled": True, "maxTokens": 800, "overlap": 100},})items = client.dataset(run["defaultDatasetId"]).list_items().itemsfor page in items:print(page["url"], page["tokenCount"], page["chunkCount"])for chunk in page["chunks"]:print(chunk["index"], chunk["tokenCount"], chunk["text"][:80])
JavaScript (apify-client)
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: process.env.APIFY_API_TOKEN });const run = await client.actor('abdou_009/apify-web-to-markdown').call({startUrls: [{ url: 'https://docs.apify.com/platform/actors/development' }],crawlMode: 'single',renderingMode: 'auto',chunking: { enabled: true, maxTokens: 800, overlap: 100 },});const { items } = await client.dataset(run.defaultDatasetId).listItems();for (const page of items) {console.log(page.url, page.tokenCount, page.chunkCount);}
n8n (HTTP Request node)
Import this minimal workflow and set APIFY_TOKEN in the n8n environment.
The workflow already targets abdou_009/apify-web-to-markdown.
{"name": "Web to Markdown","nodes": [{"parameters": {"method": "POST","url": "https://api.apify.com/v2/acts/abdou_009~apify-web-to-markdown/run-sync-get-dataset-items","sendQuery": true,"queryParameters": {"parameters": [{"name": "token", "value": "={{ $env.APIFY_TOKEN }}"}]},"sendBody": true,"specifyBody": "json","jsonBody": "={\n \"startUrls\": [{\"url\": \"https://docs.apify.com/platform/actors/development\"}],\n \"crawlMode\": \"single\",\n \"renderingMode\": \"auto\",\n \"chunking\": {\"enabled\": true, \"maxTokens\": 800, \"overlap\": 100}\n}","options": {"timeout": 300000}},"name": "Convert to Markdown","type": "n8n-nodes-base.httpRequest","typeVersion": 4.2,"position": [0, 0]}],"connections": {}}
run-sync-get-dataset-items is convenient for small jobs. For large crawls or
browser runs that may exceed the synchronous limit, start an async run and poll
the run/dataset endpoints instead.
LangChain
import osfrom langchain_community.utilities import ApifyWrapperfrom langchain_core.documents import Documentos.environ["APIFY_API_TOKEN"] = "YOUR_APIFY_TOKEN"apify = ApifyWrapper()loader = apify.call_actor(actor_id="abdou_009/apify-web-to-markdown",run_input={"startUrls": [{"url": "https://docs.apify.com/platform/actors/development"}],"crawlMode": "single","renderingMode": "auto","chunking": {"enabled": True, "maxTokens": 800, "overlap": 100},},dataset_mapping_function=lambda item: Document(page_content=item["markdown"],metadata={"source": item["url"],"title": item["title"],"token_count": item["tokenCount"],},),)documents = loader.load()
The Actor already returns chunks. If you want one LangChain Document per
chunk, read item["chunks"] from the dataset and map each chunk's text and
index instead of using the whole markdown.
Standby API (HTTP)
The Actor is Standby-enabled (usesStandbyMode: true and an OpenAPI schema
in .actor/web_server_schema.json). After it is deployed, the Endpoints tab
shows the hostname. Authentication is the normal Apify API token:
curl -H "Authorization: Bearer $APIFY_TOKEN" \"https://YOUR_STANDBY_HOST/convert?url=https%3A%2F%2Fexample.com&chunking=true"
The response is one dataset-row-shaped object:
{"url": "https://example.com/","title": "Example Domain","markdown": "---\ntitle: Example Domain\n...\n\n# Example Domain\n...","metadata": {"language": "en", "tokenCount": 28, "tokenizer": "cl100k_base"},"chunks": [{"index": 0, "text": "...", "tokenCount": 28, "wordCount": 25, "overlapTokens": 0}],"chunkCount": 1,"wordCount": 25,"tokenCount": 28,"rendered": false,"error": null}
| Query parameter | Default | Meaning |
|---|---|---|
url | required | One absolute public http(s) URL. |
renderingMode | auto | auto, static, or browser. |
chunking | false | Return chunks using cl100k_base. |
maxTokens | 800 | Hard chunk cap; 50–8,000. |
overlap | 100 | Chunk overlap; smaller than maxTokens. |
includeLinks | true | Keep absolute links. |
includeImages | false | Keep absolute images. |
Other behavior:
GET /healthreturns{"status": "ok"}; the platform readiness probe is answered on/.- Missing/invalid parameters return
400; conversion failures return502with anerrorJSON body; a PPE spending-limit rejection returns402. - Successful requests use the same
page-converted/browser-page-convertedevents as batch mode. - The Actor and Standby endpoint enable SSRF protection: hostnames are resolved before the request, every DNS answer must be public, and each redirect hop is resolved and checked again. Loopback, private, link-local and cloud-metadata addresses are rejected. DNS rebinding between the lookup and the connection remains a residual risk—use network policy if you expose the endpoint beyond trusted callers. The Python library/CLI keeps private hosts available for local development and tests.
- Standby v1 does not read
proxyConfiguration; use batch mode when you need Apify Proxy. - Standby runs stay alive between requests. Configure the idle timeout and memory in the Endpoints tab; idle runs consume platform resources.
Local verification (not a platform run) returned 200 from /health, ready
from the readiness probe, and a valid Markdown row from /convert for
https://example.com/ and the TodoMVC React SPA.
FAQ
Does it render JavaScript?
Yes, with Playwright. browser always renders. auto starts with HTTP and only
falls back when the HTML looks JavaScript-driven; the fallback is conservative,
so a JS-heavy site may need renderingMode: "browser".
Does it crawl a whole site or sitemap?
Yes. site follows same-domain links with depth/page limits and include/exclude
globs. sitemap discovers sitemap.xml and nested sitemap indexes.
Does it respect robots.txt?
Yes, by default, including disallow rules and crawl delays. Keep it enabled unless you have explicit permission to ignore it.
What happens when a URL fails?
The run continues. The failed URL gets its own dataset row with error set,
empty Markdown, and zero counts. It is not charged a page-conversion event.
How are chunks counted?
With the tiktoken cl100k_base tokenizer, not an approximate character
count. Every chunk is at or below maxTokens. Chunks prefer heading/paragraph
boundaries, fall back to token windows for oversized blocks, and keep fenced
code blocks valid. Arabic/RTL text is preserved in the tested cases.
Can I still use the old v1 input (urls, maxChars)?
Yes. Inputs that use the legacy urls field (one URL, or one URL per line)
keep working: they are mapped to startUrls, and legacy mode restores the v1
defaults — includeImages on and maxChars 20000, which truncates each
page's Markdown at 20,000 characters exactly as v1 did. maxChars is
deprecated: in modern startUrls inputs it applies only when you set it
explicitly; there is no default cap. Note that on the Apify platform the
Console pre-fills schema defaults, so set includeImages explicitly if a
legacy integration depends on it.
Is Apify Proxy supported?
The Actor passes proxyConfiguration to both HTTP and Playwright requests. The
same SDK path was tested locally with a custom proxy, but a live Apify Proxy run
still needs an authenticated Apify account.
Is this a replacement for Website Content Crawler?
Not yet, and no head-to-head run has been done. Website Content Crawler is much more mature and has anti-bot/session rotation, Firefox, cookies/login support, file downloads, screenshots, AI summaries, incremental crawling, and broad integrations. This Actor is smaller and adds built-in token chunks and a simple Python dataset contract. See BENCHMARK.md for the honest comparison and the blocked WCC measurement.
Can I self-host it?
Yes. The Docker entrypoint and Python library are in this repository. PPE charging is enabled only when the Actor runs with pay-per-event pricing; local runs do not charge.
Is there a page limit?
The input cap is 10,000 pages. Browser runs consume much more memory and time than static runs; start small. The largest verified local run is 100 pages.
Limits and known gaps
- No CAPTCHA solving, login-wall bypass, PII harvesting, or anti-bot evasion. Hard bot-protected sites stay blocked: for example AP News returns HTTP 403 to this Actor direct, via Apify datacenter proxy, and via Apify residential proxy (all verified 2026-10-20). There is no anti-blocking stack; if a site blocks automated browsers, expect an error row, not a bypass.
- PDFs with a text layer are supported up to 5 MB and 100 pages; larger PDFs produce an error row with a clear limit message. Scanned PDFs need OCR, which is not included. DOC/XLS files, screenshots, infinite scroll, and click-to-expand are not supported.
- Anti-bot challenge pages are flagged as
lowContent, not bypassed. autorendering can miss content that appears only after a delay; usebrowserplus a longer timeout for those sites.- Complex tables with rowspan/colspan can be flattened or simplified. The measured Wikipedia table now emits its caption above the table without an empty header row.
- Permalink-only headings (for example a heading containing only
¶) are removed instead of producing an empty##. - Browser mode measured 1.5 GB peak RSS and 2.53 s/page locally; memory and concurrency need tuning on the platform.
- Web page responses are capped at 8 MB; larger responses produce an error row. PDFs are capped separately at 5 MB / 100 pages (see above).
Responsible use
Use this Actor for publicly available content you have the right to access and
process. Respect each site's terms of service, robots.txt, rate limits, and
copyright. Do not use it to bypass paywalls, logins, CAPTCHAs, or technical
access controls, and do not collect personal data without a lawful basis.
Crawled content remains the property of its owners; do not republish it without
permission. This is not legal advice.
Python library and CLI
The same repository ships the underlying web2md Python library and CLI.
Install
pip install . # corepip install '.[render]' # + Playwright supportpython -m playwright install chromium
Requires Python 3.10+.
One page
web2md https://docs.python.org/3/library/pathlib.htmlweb2md https://example.com/article -o article.mdweb2md https://example.com/app -o app.md --render always --wait-for "main"
Crawl a site
web2md crawl https://example.com -o site-md \--max-pages 200 --max-depth 4 \--rate-limit 0.5 --concurrency 5 \--sitemap --exclude '/blog/tags/' --skip-existing
Crawl output includes manifest.json, crawl.ndjson, and crawl-index.md in
addition to the Markdown files. --skip-existing resumes a crawl;
--dry-run lists seeds without fetching.
Library
import asynciofrom web2md import Fetcher, PageOptions, fetch_page, html_to_markdownmarkdown = html_to_markdown(open("page.html", "rb").read(), "https://example.com/page")async def main():async with Fetcher(respect_robots=True) as fetcher:page = await fetch_page("https://example.com/article",fetcher=fetcher,options=PageOptions(render="auto"),)print(page.document)print(page.title, page.words)asyncio.run(main())
Development
python -m venv .venv && . .venv/bin/activatepip install -e '.[dev]'make test # pytest: 73 passed, 1 skipped at the last verificationmake lint # ruff check + format checkmake test-render # opt-in real-browser test (needs Playwright + Chromium)make clean
The tests use a local HTTP fixture site and cover conversion, extraction, robots, sitemap parsing, end-to-end crawls, Actor input validation, proxy handling, chunking, and PPE policy.
License
MIT