# Webpage to Markdown – HTML to Markdown for LLM & RAG (`tidyrun/webpage-to-markdown`) Actor

Turn web pages into clean Markdown for LLMs and RAG: main content only, no nav or footer, links resolved to full URLs, token count per page. Opens a browser only for JavaScript pages. $1 per 1,000 pages; failed pages are free.

- **URL**: https://apify.com/tidyrun/webpage-to-markdown.md
- **Developed by:** [Tidyrun](https://apify.com/tidyrun) (community)
- **Categories:** Developer tools, AI, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Webpage to Markdown – Clean Text for LLM & RAG

Give it URLs, get back clean Markdown: the article only, with the navigation, header, footer, cookie bars and share buttons stripped out. Each row carries a token estimate, so you can fill a context window without guessing.

**Most pages never need a browser.** This Actor reads the plain HTML first and only opens Chromium when a page turns out to be a JavaScript shell, which makes the common case fast and cheap. Failed pages are free.

### Quick start

1. Paste your links into **URLs** (or upload a text file with one URL per line).
2. Click **Start**.
3. The **Output** tab has one row per page: `markdown`, `title`, `estimatedTokens` and `words`.

Input example:

```json
{
  "urls": [
    { "url": "https://en.wikipedia.org/wiki/Markdown" },
    { "url": "https://example.com" }
  ],
  "mainContentOnly": true,
  "render": "auto"
}
```

Output row (shortened):

```json
{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "status": "ok",
  "title": "Markdown",
  "markdown": "From Wikipedia, the free encyclopedia\n\n| Markdown |\n| --- |\n...",
  "words": 3315,
  "estimatedTokens": 11770,
  "renderedWith": "http",
  "mainContentFound": true,
  "truncated": false
}
```

### Run it after a scraper

On your scraper or its task, open **Integrations**, search for this Actor and set its input:

```json
{
  "datasetId": "{{resource.defaultDatasetId}}"
}
```

Every time the scraper finishes, the page behind each row comes back as Markdown. By default only the row's own address is read — a top-level `url`, `link` or `loadedUrl` — so a scraper's outbound links are never fetched by accident. Set **URL field** (`results.url`, `links.0.href`) when you do want a list of links read. Each row keeps `recordIndexes`, the dataset rows the URL came from.

### Settings worth knowing

| Setting | What it does |
| --- | --- |
| **Main content only** | On by default. Keeps the article, drops nav, header, footer and sidebars. Turn it off to convert the whole page. |
| **Run JavaScript** | `auto` reads the HTML and only opens a browser when the page is empty without scripts. `never` is the fastest, `always` forces the browser. |
| **Keep links** | Keeps `[text](url)` with relative links resolved to full URLs. Turn off for plain prose. |
| **Keep images** | Off by default, because image lines waste tokens in a prompt. |
| **Max characters per page** | Cuts each page to fit your context window. The row says `truncated: true` when it did. |

### Why this one

- **Fewer tokens for the same content.** Dropping the page chrome took a blog index from 4,521 tokens down to **3,043 — 33% smaller — while keeping all 40 of its post links**. Same information, smaller prompt.
- **The article, not the chrome.** Menus, cookie bars and "related posts" never reach your prompt — and a listing page still keeps its list, not just the intro paragraph.
- **A browser only when needed.** Static pages cost a plain HTTP read; only JavaScript pages pay for Chromium.
- **Tables survive** as GitHub-flavoured Markdown instead of collapsing into a wall of text.
- **Links are absolute**, so the Markdown still works once it leaves the page.
- **Token estimate per page**, so you can batch to a context window.
- **Failures are free** and every failed URL comes back with a plain-language reason.

### Pricing

**$1 per 1,000 pages** read as plain HTML, **$3 per 1,000 pages** that need a browser.

Charged only after a page is converted. Leaving **Run JavaScript** on `auto` keeps most pages on the cheap path. Pages that fail are never charged, and there is no fee just for starting a run.

### Call it from your code

```bash
curl -X POST "https://api.apify.com/v2/acts/tidyrun~webpage-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls":[{"url":"https://example.com"}]}'
```

### Tips

- Feeding a RAG index? Leave **Keep images** off and **Keep links** on: the links give the model somewhere to point.
- A page comes back short or empty? Set **Run JavaScript** to `always`; some sites render everything client-side.
- Want the raw page including the menus? Turn **Main content only** off.
- Running with **Run JavaScript** set to `never`? 1 GB of memory is enough and runs just as fast as 2 GB.
- PDFs are not web pages: use **PDF to Text** (`tidyrun/pdf-to-text`) for those.

### Responsible use

Reads public pages only, as an ordinary visitor. It does not log in, does not solve CAPTCHAs and does not collect personal data. Check a site's terms and robots rules before bulk-reading it, and keep the volume polite.

# Actor input Schema

## `urls` (type: `array`):

Pages to convert. You can also upload a text file with one URL per line.

## `datasetId` (type: `string`):

Optional dataset, for example from a scraper run. Its page links are read together with the URLs above, and each row keeps the dataset record index. When connected as an integration after a scraper, use {{resource.defaultDatasetId}}.

## `urlField` (type: `string`):

Field that holds the page link: a URL, a list of URLs, or objects with url/href. Dot paths work: results.0.url, or results.url for every link in a list. Leave empty and only the row’s own address is used (a top-level <code>url</code>, <code>link</code> or <code>loadedUrl</code>), so a scraper’s outbound links are never read by accident.

## `mainContentOnly` (type: `boolean`):

Keep the article and drop the navigation, header, footer, sidebars and share buttons. Turn this off to convert the whole page.

## `render` (type: `string`):

Most pages ship their text in the HTML and need no browser, which is much faster and cheaper. Auto starts there and opens a browser only when the page turns out to be a JavaScript shell.

## `includeLinks` (type: `boolean`):

Keep [text](url) links, with relative URLs resolved to full ones. Turn off for plain prose.

## `includeImages` (type: `boolean`):

Keep ![alt](url) images. Off by default because image lines waste tokens in an LLM prompt.

## `maxCharacters` (type: `integer`):

Cut each page's Markdown to this length to fit a context window. Leave empty for the whole page.

## `dedupe` (type: `boolean`):

Read a repeated URL once instead of paying for it twice. The row lists every dataset record it came from.

## `timeoutSecs` (type: `integer`):

How long to wait for one page.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://en.wikipedia.org/wiki/Markdown"
    },
    {
      "url": "https://blog.cloudflare.com/"
    }
  ],
  "mainContentOnly": true,
  "render": "auto",
  "includeLinks": true,
  "includeImages": false,
  "dedupe": true,
  "timeoutSecs": 45
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL: markdown, title, byline, excerpt, word and token counts, whether a browser was needed, and any error.

## `summary` (type: `string`):

Pages converted, how many needed a browser, total characters and tokens, and failures by reason.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://en.wikipedia.org/wiki/Markdown"
        },
        {
            "url": "https://blog.cloudflare.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidyrun/webpage-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        { "url": "https://en.wikipedia.org/wiki/Markdown" },
        { "url": "https://blog.cloudflare.com/" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("tidyrun/webpage-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://en.wikipedia.org/wiki/Markdown"
    },
    {
      "url": "https://blog.cloudflare.com/"
    }
  ]
}' |
apify call tidyrun/webpage-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidyrun/webpage-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sHtmVBMYKBmqUKy1l/builds/E5av96N6I6mKVTcX1/openapi.json