Webpage to Markdown – Clean Text for LLM & RAG

Give it URLs, get back clean Markdown: the article only, with the navigation, header, footer, cookie bars and share buttons stripped out. Each row carries a token estimate, so you can fill a context window without guessing.

Most pages never need a browser. This Actor reads the plain HTML first and only opens Chromium when a page turns out to be a JavaScript shell, which makes the common case fast and cheap. Failed pages are free.

Quick start

1. Paste your links into URLs (or upload a text file with one URL per line).
2. Click Start.
3. The Output tab has one row per page: markdown, title, estimatedTokens and words.

Input example:

{
  "urls": [
    { "url": "https://en.wikipedia.org/wiki/Markdown" },
    { "url": "https://example.com" }
  ],
  "mainContentOnly": true,
  "render": "auto"
}

Output row (shortened):

{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "status": "ok",
  "title": "Markdown",
  "markdown": "From Wikipedia, the free encyclopedia\n\n| Markdown |\n| --- |\n...",
  "words": 3315,
  "estimatedTokens": 11770,
  "renderedWith": "http",
  "mainContentFound": true,
  "truncated": false
}

Run it after a scraper

On your scraper or its task, open Integrations, search for this Actor and set its input:

{
  "datasetId": "{{resource.defaultDatasetId}}"
}

Every time the scraper finishes, the page behind each row comes back as Markdown. By default only the row's own address is read — a top-level url, link or loadedUrl — so a scraper's outbound links are never fetched by accident. Set URL field (results.url, links.0.href) when you do want a list of links read. Each row keeps recordIndexes, the dataset rows the URL came from.

Settings worth knowing

Setting	What it does
Main content only	On by default. Keeps the article, drops nav, header, footer and sidebars. Turn it off to convert the whole page.
Run JavaScript	auto reads the HTML and only opens a browser when the page is empty without scripts. never is the fastest, always forces the browser.
Keep links	Keeps [text](url) with relative links resolved to full URLs. Turn off for plain prose.
Keep images	Off by default, because image lines waste tokens in a prompt.
Max characters per page	Cuts each page to fit your context window. The row says truncated: true when it did.

Why this one

• Fewer tokens for the same content. Dropping the page chrome took a blog index from 4,521 tokens down to 3,043 — 33% smaller — while keeping all 40 of its post links. Same information, smaller prompt.
• The article, not the chrome. Menus, cookie bars and "related posts" never reach your prompt — and a listing page still keeps its list, not just the intro paragraph.
• A browser only when needed. Static pages cost a plain HTTP read; only JavaScript pages pay for Chromium.
• Tables survive as GitHub-flavoured Markdown instead of collapsing into a wall of text.
• Links are absolute, so the Markdown still works once it leaves the page.
• Token estimate per page, so you can batch to a context window.
• Failures are free and every failed URL comes back with a plain-language reason.

Pricing

$1 per 1,000 pages read as plain HTML, $3 per 1,000 pages that need a browser.

Charged only after a page is converted. Leaving Run JavaScript on auto keeps most pages on the cheap path. Pages that fail are never charged, and there is no fee just for starting a run.

Call it from your code

curl -X POST "https://api.apify.com/v2/acts/tidyrun~webpage-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls":[{"url":"https://example.com"}]}'

Tips

• Feeding a RAG index? Leave Keep images off and Keep links on: the links give the model somewhere to point.
• A page comes back short or empty? Set Run JavaScript to always; some sites render everything client-side.
• Want the raw page including the menus? Turn Main content only off.
• Running with Run JavaScript set to never? 1 GB of memory is enough and runs just as fast as 2 GB.
• PDFs are not web pages: use PDF to Text (tidyrun/pdf-to-text) for those.

Responsible use

Reads public pages only, as an ordinary visitor. It does not log in, does not solve CAPTCHAs and does not collect personal data. Check a site's terms and robots rules before bulk-reading it, and keep the volume polite.