Web Page to Markdown for LLM & RAG – clean main content, no browser

Give it web page URLs and get the main content of each page as clean Markdown, with title, language, word and token counts and optional heading-aware chunks for RAG, one record per page. It talks plain HTTP, with no browser: results are fast, cheap and easy to schedule. Boilerplate (menus, headers, footers, cookie notices) is left out, links and images get absolute addresses, tables stay tables, and a page that cannot be read is a free record that tells you why.

Price: $0.001 per page read (that is $1.00 per 1,000 pages), one flat price per page. Pages that are blocked, empty, not HTML, forbidden by robots.txt, unreachable, repeated or too big are free.

Who it is for

• RAG and AI-agent builders who need web pages as text a model can read, already cut into chunks that follow the headings.
• Teams that index or monitor documentation, help centers, blogs and news pages on a schedule.
• Data and no-code users (Clay, Make, n8n, Zapier...) who want a URL in and Markdown out through the Apify API.
• Anyone paying for a browser just to read text: if the text is in the HTML, a browser is not needed.

What you get for each page

• markdown: the main content, starting with the title as a level-1 heading. Headings, lists, bold and italic, links (absolute, without the repeated title text), code blocks as fenced blocks with their lines, indentation and language (also from syntax highlighters that write one div per line, and pre blocks without code), and data tables as GFM pipe tables.
• title, language, description, wordCount, tokenEstimate, httpStatus, finalUrl (after redirects), charset.
• Optional chunks for RAG: pieces of about the size you choose, cut at paragraph, list, table and code-block edges. A heading starts a new chunk once the current one is half full, and each chunk says which headings it sits under.
• Optional links: the links of the content as data (text and absolute address), to carry on from.
• extraction: how the content was found (readability, your own selector, or fallback_body).

How the main content is found: Mozilla Readability (the engine behind Firefox Reader View) finds the article; if it cannot, the page body without navigation, headers, footers and asides is converted and the record says fallback_body. Permalink marks in headings (¶, #) are removed. Infoboxes and navigation boxes of Wikipedia-style pages are left out, and so are, by default, their reference lists and notes (removeReferences). You can also give a CSS selector for the part you want, and removeSelectors for parts to cut out first.

Quick start

1. Paste the page addresses into Page URLs (or give a datasetId of another run, see below).
2. Click Start.
3. Open Pages for the list, Markdown for the content, RAG chunks, Links, or download the dataset as JSON, CSV or Excel.

Example input

This is the example prefilled in the input form (three real, free pages):

{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://developer.mozilla.org/en-US/docs/Web/HTML/Element/table",
    "https://docs.python.org/3/library/json.html"
  ],
  "chunkTokens": 400,
  "chunkOverlapTokens": 40
}

Run result (a real run): 3 records (3 ok), 3 charged = 3 x $0.001 = $0.003, in about 2 seconds.

Example output (a real run, shortened)

The record of the third page, a Sphinx documentation page: its code examples come out as fenced blocks with their lines, indentation and language intact. Every field is always present; fields that do not apply are null:

{
  "url": "https://docs.python.org/3/library/json.html",
  "finalUrl": "https://docs.python.org/3/library/json.html",
  "status": "ok",
  "reason": null,
  "charged": true,
  "httpStatus": 200,
  "title": "json — JSON encoder and decoder",
  "language": "en",
  "description": "Source code: Lib/json/__init__.py JSON (JavaScript Object Notation), specified by RFC 7159(which obsoletes RFC 4627) and by ECMA-404, is a lightweight data interchange format inspired by JavaScript...",
  "extraction": "readability",
  "charset": "utf-8",
  "markdown": "# json — JSON encoder and decoder\n\n**Source code:** [Lib/json/\\_\\_init\\_\\_.py](https://github.com/python/cpython/tree/3.15/Lib/json/__init__.py)\n\n---\n\n[JSON (JavaScript Object Notation)](https://json.org/), specified by [**RFC 7159**](https://datatracker.ietf.org/doc/html/rfc7159.html) (which obsoletes [**RFC 4627**](https://datatracker.ietf.org/doc/html/rfc4627.html)) and by [ECMA-404](https://ecma-international.org/publications-and-standards/standards/ecma-404/), is a lightweight data interchange format inspired by [JavaScript](https://en.wikipedia.org/wiki/JavaScript) object literal syntax (although it is not a strict subset of JavaScript \\[1\\] ).\n\n[...]\n\nEncoding basic Python object hierarchies:\n\n```python\n>>> import json\n>>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])\n'[\"foo\", {\"bar\": [\"baz\", null, 1.0, 2]}]'\n>>> print(json.dumps(\"\\\"foo\\bar\"))\n\"\\\"foo\\bar\"\n>>> print(json.dumps('\\u1234'))\n\"\\u1234\"\n>>> print(json.dumps('\\\\'))\n\"\\\\\"\n>>> print(json.dumps({\"c\": 0, \"b\": 0, \"a\":\n\n[...]",
  "wordCount": 3872,
  "tokenEstimate": 7774,
  "truncated": false,
  "links": null,
  "chunks": [
    {
      "index": 2,
      "headings": [
        "json — JSON encoder and decoder"
      ],
      "text": "Customizing JSON object decoding:\n\n```python\n>>> import json\n>>> def as_complex(dct):\n...     if '__complex__' in dct:\n...         return complex(dct['real'], dct['imag'])\n...     return dct\n...\n>>> json.loads('{\"__complex__\": tru [...]",
      "tokenEstimate": 398
    }
  ],
  "warnings": [],
  "extractedAt": "2026-10-23T10:08:23.883Z"
}

([...] marks text cut for this README. The page has 23 chunks in the real record; chunks shows the third.)

Chain it after another Actor

Put the ID of a dataset in datasetId and the name of its address field in datasetUrlField (url by default; a dotted path such as page.url works). Those addresses are added to urls. This is how you read all the pages a crawler or a sitemap reader found, without a script.

Input

Option	Default	What it does
urls	3 sample pages	Page addresses (http or https). A repeated address is reported once as duplicate.
datasetId	empty	Dataset of another run to read addresses from.
datasetUrlField	url	Field of that dataset with the address.
selector	empty	CSS selector of the part to convert. Empty: automatic. No match on a page: automatic for that page, with a warning.
removeSelectors	empty	CSS selectors cut out before reading (cookie banner, related posts...).
includeLinks	on	Links as Markdown links with absolute addresses. Off: link text only.
includeImages	off	Images as .
includeTables	on	Data tables as GFM tables. Off: tables left out.
removeReferences	on	On Wikipedia-style pages, drop reference lists, notes, bibliography, further reading and external links. Footnote markers are always dropped.
outputLinks	off	Add the links array to every record.
chunkTokens	0 (no chunks)	Chunk size in estimated tokens (50 to 8000).
chunkOverlapTokens	0	The end of a full chunk is repeated at the start of the next, up to this many tokens (at most half the chunk size).
maxPages	1000	Pages read in this run; the rest is counted in the SUMMARY, not fetched.
minWords	10	Fewer words of main content: empty (free).
maxPageMegabytes	5	Bigger pages are not read (too_large, free).
respectRobotsTxt	on	Read robots.txt of every site first and skip what it forbids to this bot.
requestsPerSecond	2	Requests per second to one site (1 or 2). Different sites run in parallel.
maxConcurrency	5	Pages read at the same time.
timeoutSecs	30	Longest wait for a site.

Output fields

Field	Meaning
url	The address as given (credentials removed, fragment dropped).
finalUrl	Address after redirects.
status	See the table below.
reason	One sentence saying why a page has no content; null for ok.
charged	true when this record was charged as one page event.
httpStatus	HTTP status of the final answer.
title	The headline when the <title> is the headline plus the site name; otherwise the <title>.
language	Language tag from the page (en, pt-BR...), or null.
description	The meta description, or the first paragraph of the text.
extraction	readability, selector or fallback_body.
charset	Character set the page was decoded with (byte-order mark, Content-Type header, <meta>, then UTF-8 or windows-1252 by inspection).
markdown	The main content as Markdown.
wordCount	Words of text (approximate; scripts without spaces are segmented).
tokenEstimate	Estimated LLM tokens (about 4 characters per token, one per CJK character; give or take 25 %). Not a tokenizer count.
truncated	true when the Markdown passed 1,500,000 characters and was cut at a paragraph boundary.
links	With outputLinks: {text, url} of each link of the content, each address once, at most 1000.
chunks	With chunkTokens: {index, headings, text, tokenEstimate}. headings is the path where the chunk starts.
warnings	Things worth knowing (credentials removed from the address, unknown character set, fallback extraction...).
extractedAt	When the record was made.

Statuses

status	Meaning	Charged
ok	The page was read and has a main content of at least minWords words.	yes
empty	Valid HTML with no readable text, or fewer words than minWords. A page whose text is built by JavaScript looks like this: there is no browser here.	no
blocked	The site answered 401, 403, 429 or 202, or served a bot-challenge page. Nothing is done to get around it.	no
robots_disallowed	robots.txt forbids this address to this bot.	no
not_html	PDF, image, JSON, XML... (see the sister Actors below for PDF and Office files).	no
too_large	Over maxPageMegabytes, over 150,000 HTML elements (a data dump, not a page), or too heavy to read in 40 seconds or in the memory of the run.	no
http_error	404, 410, 5xx and other HTTP errors.	no
unreachable	DNS, connection, TLS or timeout failure.	no
duplicate	A repeated input address, or a page already delivered under another address in this run (for example behind a redirect).	no
invalid	Not a usable public web address (ftp:, mailto:, private or local network...).	no

Pricing

One event, page: $0.001. It is charged only when a record has status ok. The spending limit you set on a run is respected: when it is reached the run stops, does not fetch the next pages, and says so in the status message and in the SUMMARY record (stopReason: "spending limit reached", with the addresses it did not start). A run restarted by the platform does not deliver or charge a page twice.

Safe and polite by design

• robots.txt first. The robots.txt of every site is read before the first request there, and every redirect hop is checked again. Crawl-delay is honoured up to 10 seconds.
• Honest bot. One fixed User-Agent that names the bot (Mozilla/5.0 (compatible; WebPageToMarkdownBot/0.1)), no proxies, no disguise, no attempt to solve a challenge. The name is documented here rather than a contact link inside the header, because some large news sites answer 403 to any User-Agent that carries a URL. A wall is reported as blocked and left alone (the only retries are one more try after a dropped connection or a 5xx, and one wait of up to 30 seconds when a site says Retry-After on a 429).
• Gentle. At most 2 requests per second to one site.
• No window into your network. Private, loopback and link-local addresses are refused, and the check is made on the address the connection really uses (so a name that answers a private address after the first lookup is refused too), on every redirect hop.
• No personal data collected on purpose. There is no author or byline field. The text of the page is the text of the page.
• Credentials written inside an address are removed from the record and the log, and never used.

Limits, honestly

• No JavaScript is run. Pages whose text appears only after scripts run (many single-page apps) come back empty, and you are not charged. For those you need a browser-based Actor.
• No login, no paywall, no CAPTCHA, no infinite scroll.
• One address is one record: this Actor does not follow links by itself. Use outputLinks, or chain it after a crawler or a sitemap reader with datasetId.
• Main-content detection is a heuristic. On unusual layouts it can keep a sidebar, drop a section, or even take a banner for the page. Two guards help: when the content found is only a sliver (under 10 %) of the text of a long page, the whole page body is converted instead (extraction says fallback_body, with a warning); and minWords keeps near-empty results free. For pages you know, use selector and removeSelectors, and look at extraction and warnings.
• tokenEstimate and the chunk sizes are estimates (no tokenizer is shipped).
• Memory: the default 512 MB reads typical pages (a few hundred KB) many at a time; measured peaks are about 190 MB for ordinary documentation pages and about 440 MB for pages of 1 MB. A page of several MB is many times its size in memory while it is read and needs 1 GB or more; one that does not fit comes back too_large (free), and a bigger memory setting lets it through.

FAQ

Why is the title also the first line of the Markdown? Readability removes the page headline from the article body. It is put back as a level-1 heading so each record, and the first chunk, stands alone.

A page I see in my browser came back empty. Its text is probably created by JavaScript. Open the page, view source: if the text is not there, this Actor cannot see it.

Can I get the pictures? Set includeImages: they come as Markdown images with absolute addresses (alt text kept). Images are not downloaded or described.

Is the same page charged twice if two of my addresses lead to it? No. The second is a free duplicate.

What about PDF and Office files? Use the sister Actors: far003/pdf-to-text-markdown-extractor and far003/office-docs-markdown. A PDF sent here comes back not_html for free.

I need the page metadata (Open Graph, JSON-LD, schema.org). That is far003/metadata-structured-data. This Actor is about the readable content.

Does it work in Chinese, Japanese, Arabic, Cyrillic...? Yes: the character set is detected and words are counted with a script-aware splitter. The token estimate counts one token per CJK character.

Integration

Call it from your code with the Apify client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('far003/web-page-to-markdown').call({
    urls: ['https://blog.apify.com/what-is-web-scraping/'],
    chunkTokens: 400,
    chunkOverlapTokens: 40,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const page of items.filter((p) => p.status === 'ok')) {
    for (const chunk of page.chunks) console.log(page.url, chunk.headings.at(-1), chunk.tokenEstimate);
}

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("far003/web-page-to-markdown").call(run_input={
    "urls": ["https://blog.apify.com/what-is-web-scraping/"],
    "chunkTokens": 400,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page["status"] == "ok":
        print(page["title"], page["wordCount"], page["tokenEstimate"])

The run also stores a SUMMARY record in the key-value store: counts per status, how many pages were charged, requests made, and the addresses it did not reach if it stopped early.

Built with

Plain Node.js with the following open-source libraries, all under permissive licenses:

Library	License	Used for
@mozilla/readability	Apache-2.0	Finding the main content of a page
linkedom	ISC	Reading HTML into a document tree
undici	MIT	HTTP client with the connection-time address check
turndown	MIT	HTML to Markdown
turndown-plugin-gfm	MIT	Strikethrough and task lists
@mixmark-io/domino (dependency of turndown)	BSD-2-Clause	HTML parser used by turndown
apify	Apache-2.0	Apify platform SDK

Related

• PDF to Text & Markdown Extractor – the same job for PDFs.
• Office Documents to Markdown – the same job for DOCX, PPTX and XLSX.
• Metadata & Structured Data Extractor – JSON-LD, Open Graph and the SEO metadata of a page.
• Sitemap URL Extractor + Robots.txt & llms.txt Validator – list the pages of a site, then feed them here with datasetId.