# Web Page → Markdown Converter (Trafilatura, LLM-ready) (`gochujang/web-to-markdown`) Actor

Convert any URL to clean Markdown plus structured metadata (title, author, date, lang, image, tags). Uses trafilatura — the same library Common Crawl uses. LLM-ready output. Batch up to 500 URLs. $0.005 per URL.

- **URL**: https://apify.com/gochujang/web-to-markdown.md
- **Developed by:** [Hojun Lee](https://apify.com/gochujang) (community)
- **Categories:** Developer tools, Automation, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## Web Page → Markdown Converter

> Convert any URL to clean Markdown plus structured metadata (title, author, date, lang, image, tags). **Uses trafilatura — the same library Common Crawl uses.** LLM-ready output. Batch up to 500 URLs. **$0.005 per URL.**

---

### Why this exists

Most LLM pipelines need clean article-body text — but raw HTML is 60-90% boilerplate (nav, footer, ads, JS, related stories). Existing solutions:
- **Browserless / Puppeteer**: complex setup, $30+/mo
- **Mercury Parser**: deprecated
- **Diffbot**: $299/mo minimum
- **Readability.js**: requires running Node

This actor wraps **trafilatura** — the gold-standard Python library used by Common Crawl and most LLM training pipelines — into a one-call API. Pass a URL list, get clean Markdown + metadata back.

---

### What you get per row

| Field | Example | Notes |
|---|---|---|
| `url` | `https://...` | input URL |
| `ok` | `true` | did extraction succeed |
| `title` | `Bitcoin — Wikipedia` | from `<title>` or og |
| `author` | `Wikipedia contributors` | |
| `description` | `Bitcoin is a cryptocurrency...` | |
| `date_published` | `2025-12-01` | |
| `language` | `en` | auto-detected |
| `sitename` | `Wikipedia` | |
| `tags` | `["cryptocurrency", "blockchain"]` | |
| `categories` | `["Technology"]` | |
| `image` | `https://...` | hero image |
| `markdown` | `# Bitcoin\n\nBitcoin is...` | clean body |
| `char_count` | `48230` | |
| `word_count` | `7842` | |

---

### Quick start

#### Single URL
```json
{
  "url": "https://en.wikipedia.org/wiki/Bitcoin"
}
````

#### Batch of URLs

```json
{
  "urls": [
    "https://techcrunch.com/article-1",
    "https://www.theverge.com/article-2",
    "https://www.wired.com/article-3"
  ],
  "includeTables": true,
  "deduplicate": true
}
```

#### Custom User-Agent (some sites require it)

```json
{
  "url": "https://...",
  "userAgent": "Mozilla/5.0 (compatible; YourBot/1.0; +https://yourdomain.com/bot)"
}
```

***

### Pricing

**Pay-Per-Event**: `$0.005 per URL processed.`

| Run | URLs | Cost |
|---|---|---|
| Single article | 1 | $0.005 |
| Batch of 100 | 100 | $0.50 |
| Daily crawl of 1K URLs | 1000 | $5.00 |

Vs Diffbot ($299/mo), Mercury ($199/mo for similar tier), this is **40-60x cheaper** for typical volumes.

***

### Common pipeline patterns

#### Feed to Claude / GPT for summarization

```bash
## 1. Extract clean text
curl -X POST "https://api.apify.com/v2/acts/gochujang~web-to-markdown/runs?token=$T" \
  -d '{"url":"..."}'

## 2. Pipe markdown to Claude
curl -X POST https://api.anthropic.com/v1/messages \
  -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Summarize: $MARKDOWN\"}]}"
```

#### RSS-style aggregator

1. [Sitemap URL Discovery](https://apify.com/gochujang/sitemap-url-discovery) to get all article URLs
2. Filter by lastmod (recent only)
3. This actor to convert each to Markdown
4. Store in your DB / Notion / Obsidian

#### Personal read-it-later

Schedule this actor with your "saved articles" Google Sheet → get clean markdown into Obsidian / Logseq daily.

***

### Use cases

1. **LLM input prep** — Clean text for RAG / fine-tuning / summarization
2. **Content curation** — Newsletter / digest aggregation
3. **SEO research** — Compare clean content across competitors
4. **Archiving** — Read-it-later in Markdown format
5. **Translation pipelines** — Strip boilerplate before sending to MT

***

### Data source / engine

- **Engine**: [trafilatura](https://trafilatura.readthedocs.io/) — actively maintained, used by Common Crawl
- **Fallback**: Returns `ok: false` with error message if a page can't be extracted (paywall, JS-heavy SPA without SSR, etc.)

***

### Limitations

- **JS-only sites**: Pages that render entirely in client-side JS may return empty markdown. For those, use a browser-rendering actor (Playwright/Puppeteer-based).
- **Paywalls**: This actor doesn't bypass paywalls.
- **Comments / discussion sections**: Off by default; enable with `includeComments: true`.

***

### Related actors (same author)

- [HTML Metadata Extractor](https://apify.com/gochujang/html-metadata-extractor) — Just metadata (OG, Twitter, JSON-LD) without article body
- [Sitemap URL Discovery](https://apify.com/gochujang/sitemap-url-discovery) — Find all URLs to feed into this actor
- [PDF Text Extractor](https://apify.com/gochujang/pdf-text-extractor) — PDF version
- [JSON Schema Generator](https://apify.com/gochujang/json-schema-generator)

***

### Feedback

A short review helps content/AI engineers find it: [Leave a review on Apify Store](https://apify.com/gochujang/web-to-markdown#reviews)

# Actor input Schema

## `urls` (type: `array`):

List of URLs to convert. Each is billed separately.

## `url` (type: `string`):

Single URL (used when 'urls' is empty).

## `includeComments` (type: `boolean`):

Include reader comments / discussion sections.

## `includeTables` (type: `boolean`):

Render HTML tables as Markdown tables.

## `deduplicate` (type: `boolean`):

Drop repeated boilerplate (nav, footer, ads).

## `userAgent` (type: `string`):

Custom UA string. Default looks like a desktop browser.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Bitcoin"
  ],
  "url": "",
  "includeComments": false,
  "includeTables": true,
  "deduplicate": true,
  "userAgent": ""
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Bitcoin"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("gochujang/web-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://en.wikipedia.org/wiki/Bitcoin"] }

# Run the Actor and wait for it to finish
run = client.actor("gochujang/web-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Bitcoin"
  ]
}' |
apify call gochujang/web-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=gochujang/web-to-markdown",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Web Page → Markdown Converter (Trafilatura, LLM-ready)",
        "description": "Convert any URL to clean Markdown plus structured metadata (title, author, date, lang, image, tags). Uses trafilatura — the same library Common Crawl uses. LLM-ready output. Batch up to 500 URLs. $0.005 per URL.",
        "version": "0.1",
        "x-build-id": "LWY8yZDtHrftLXu0l"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/gochujang~web-to-markdown/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-gochujang-web-to-markdown",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/gochujang~web-to-markdown/runs": {
            "post": {
                "operationId": "runs-sync-gochujang-web-to-markdown",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/gochujang~web-to-markdown/run-sync": {
            "post": {
                "operationId": "run-sync-gochujang-web-to-markdown",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "urls": {
                        "title": "URLs",
                        "type": "array",
                        "description": "List of URLs to convert. Each is billed separately.",
                        "default": [],
                        "items": {
                            "type": "string"
                        }
                    },
                    "url": {
                        "title": "Single URL (shortcut)",
                        "type": "string",
                        "description": "Single URL (used when 'urls' is empty).",
                        "default": ""
                    },
                    "includeComments": {
                        "title": "Include comments",
                        "type": "boolean",
                        "description": "Include reader comments / discussion sections.",
                        "default": false
                    },
                    "includeTables": {
                        "title": "Include tables",
                        "type": "boolean",
                        "description": "Render HTML tables as Markdown tables.",
                        "default": true
                    },
                    "deduplicate": {
                        "title": "Deduplicate boilerplate",
                        "type": "boolean",
                        "description": "Drop repeated boilerplate (nav, footer, ads).",
                        "default": true
                    },
                    "userAgent": {
                        "title": "User-Agent override",
                        "type": "string",
                        "description": "Custom UA string. Default looks like a desktop browser.",
                        "default": ""
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```