# Web Page to Markdown & Link Preview (OG, favicon) (`keith-apis/web-page-to-markdown`) Actor

Turn a list of URLs into clean Markdown for LLMs and RAG plus link preview metadata: title, description, Open Graph and Twitter card images, favicons. No browser, SSRF-safe, honours robots.txt for Markdown. $0.0015 per page; failed pages are free.

- **URL**: https://apify.com/keith-apis/web-page-to-markdown.md
- **Developed by:** [Lai Keith](https://apify.com/keith-apis) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 page processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Page to Markdown & Link Preview: clean Markdown for LLMs plus Open Graph, Twitter card and favicons

**Web Page to Markdown & Link Preview is an Apify Actor that turns a list of URLs into one dataset row per page: the page's main content as clean Markdown (for ChatGPT, Claude, agents and RAG pipelines) and its link preview metadata (title, description, Open Graph and Twitter card images, favicons, author and dates).** It fetches without a headless browser, refuses private-network and cloud-metadata addresses, honours robots.txt for Markdown, and you pay only for pages that return a result ($0.0015 each, or $1.50 per 1,000).

### What does Web Page to Markdown & Link Preview do?

You give it a list of public URLs and pick a mode:

- **`both`** (default): one row with the preview and the Markdown, at the price of one page.
- **`preview`:** link preview only, the same data Slack or Discord show when you paste a link.
- **`markdown`:** the main content only, as Markdown.

For each URL it returns:

- **Preview:** `title`, `description`, `site_name`, `canonical`, `lang`, `type`, `image` and `images[]` (absolute URLs, with `width`, `height` and `alt` when declared), `videos[]`, `favicons[]`, `theme_color`, `author`, `published_time`, `modified_time`, `keywords`, `oembed` and `twitter` card fields. It falls back from Open Graph to Twitter cards to plain HTML (`<title>`, meta description, first paragraph, first real image), so pages without social tags still give a usable card.
- **Markdown:** the article, docs page or README without menus, footers, cookie banners or ads, plus `word_count`, `reading_time_minutes`, `truncated` and, with `includeLinks`, a `links` list.
- **Fetch details:** `final_url` after redirects and the HTTP `status`.

Results land in the run's dataset, ready to export as JSON, CSV or Excel or to pass to the next Actor.

### When should I use it?

- **LLM and RAG pipelines:** turn documentation, blog posts or a list of sources into Markdown chunks for embeddings or prompts.
- **Link previews in bulk:** enrich a bookmark list, a newsletter, a CMS or a chat export with titles, images and favicons.
- **Content audits and SEO:** check Open Graph and Twitter card tags, canonical URLs and missing images across many pages.
- **After a crawler or search Actor:** pass its URLs here to get clean text and metadata.

If you need one page at a time from your own code, use the [Web Page API](https://web-page-api.fly.dev/) instead. It runs the same code behind a REST endpoint.

### Input

```json
{
  "urls": ["https://example.com/", "https://en.wikipedia.org/wiki/Markdown", "https://www.python.org/"],
  "mode": "both",
  "includeLinks": false,
  "includeImages": false,
  "maxChars": 100000
}
```

| Field | Default | Description |
|---|---|---|
| `urls` | required | Public http(s) URLs (ports 80 and 443). Up to 1,000 per run; duplicates are removed |
| `mode` | `both` | `both`, `preview` or `markdown` |
| `includeLinks` | `false` | Keep hyperlinks in the Markdown and add a `links` list |
| `includeImages` | `false` | Keep images in the Markdown as `![alt](url)` |
| `maxChars` | `100000` | Truncate each page's Markdown to this many characters (100 to 1,000,000) |

### Output

One row per page. This is the real row for `https://www.python.org/` (abridged; checked 2026-10-24):

```json
{
  "index": 2,
  "url": "https://www.python.org/",
  "mode": "both",
  "final_url": "https://www.python.org/",
  "status": 200,
  "title": "Welcome to Python.org",
  "description": "The official home of the Python Programming Language",
  "image": "https://www.python.org/static/opengraph-icon-200x200.png",
  "favicon": "https://www.python.org/static/favicon.ico",
  "lang": "en",
  "preview": { "site_name": "…", "canonical": "…", "images": ["…"], "favicons": ["…"], "twitter": "…", "fetch_ms": "…" },
  "markdown": "# Welcome to Python.org

### Get Started

Whether you're new to programming or an experienced developer, it's easy to learn …",
  "word_count": 269,
  "reading_time_minutes": 2,
  "truncated": false,
  "links": null,
  "errors": null
}
```

In `both` mode, if one part fails (for example robots.txt disallows the Markdown) the row still has the other part, and `errors` says what failed. URLs where nothing worked are not in the dataset: they are listed with their error code in the key-value store record `FAILED` and are not charged.

### Is it safe to give it any URL?

Yes. It uses the same fetching code as the Web Page API, which is built for untrusted URLs:

- Private, loopback, link-local, CGNAT, reserved and cloud-metadata addresses (such as `169.254.169.254`) are refused in every IPv4 and IPv6 spelling, as are internal names such as `localhost` and `*.internal`.
- DNS is resolved once, checked, and the connection goes to that exact IP, so DNS rebinding does not work.
- Every redirect hop (at most 5) is checked again. Only ports 80 and 443 are allowed.
- One 8-second deadline covers each fetch, and pages over 3 MB are refused.

### Does it respect robots.txt?

Markdown does: it honours robots.txt for the token `WebPageAPI` (else `*`), and a disallowed page gets the error `robots_disallowed`. Link previews do not, the same as Slack, Discord and iMessage link unfurlers, because a preview reads only public head metadata.

### How much does it cost?

It's **pay per result: $0.0015 per page** ($1.50 per 1,000 pages), plus a little Apify platform usage. `both` mode costs the same as one mode. Invalid, blocked, unreachable and robots-disallowed URLs are **not charged**. If you set a maximum spend on the run, it stops when that limit is reached.

### FAQ

**How do I convert a list of web pages to Markdown for ChatGPT, Claude or a RAG pipeline?**
Paste the URLs into `urls`, set `mode` to `markdown` (or keep `both`), run the Actor and export the dataset. Each row's `markdown` field holds the main content only.

**How do I get link previews (title, image, favicon) for many URLs at once?**
Run it with `mode` set to `preview`. Each row has `title`, `description`, `image` and `favicon`, and the full Open Graph and Twitter card data in `preview`.

**Does it render JavaScript?**
No. There is no headless browser, which keeps it fast and cheap. Pages that build their content with JavaScript return what the server sends; that is usually enough for a preview, but single-page apps may give little Markdown. For those, use a browser-based crawler.

**How does it compare with other URL-to-Markdown tools?**
Most Apify Actors and APIs do either Markdown or link previews; this one returns both in one row for one price. Firecrawl and Jina Reader focus on Markdown, Microlink on previews and screenshots.

**Is my data stored?**
The results stay in your own Apify storage, under your account's retention settings. Pages are fetched and parsed inside the run; nothing is sent to another service.

### Related

- [Web Page API](https://web-page-api.fly.dev/): the same link preview and Markdown extraction as a REST API, one URL per call.
- [Free URL to Markdown converter](https://web-page-api.fly.dev/url-to-markdown): try the extractor on a single page in your browser, no account needed.
- [keith-apis-mcp](https://github.com/Keithhk/keith-apis-mcp): an MCP server that lets Claude, Cursor and other AI assistants read web pages as Markdown.

# Actor input Schema

## `urls` (type: `array`):

Public http(s) URLs, one per line (ports 80 and 443). Up to 1,000 per run; duplicates are removed. Charged per page that returns a result; invalid, blocked, unreachable or robots-disallowed URLs are skipped and not charged.

## `mode` (type: `string`):

preview: link preview metadata (title, description, images, favicons, Open Graph and Twitter card). markdown: the main content as clean Markdown (honours robots.txt). both: one row with the preview and the Markdown, same price.

## `includeLinks` (type: `boolean`):

Keep hyperlinks in the Markdown and add a links list to each row.

## `includeImages` (type: `boolean`):

Keep images as ![alt](url).

## `maxChars` (type: `integer`):

Truncate each page's Markdown to this many characters (100 to 1,000,000).

## Actor input object example

```json
{
  "urls": [
    "https://example.com/",
    "https://en.wikipedia.org/wiki/Markdown",
    "https://www.python.org/"
  ],
  "mode": "both",
  "includeLinks": false,
  "includeImages": false,
  "maxChars": 100000
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `failed` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com/",
        "https://en.wikipedia.org/wiki/Markdown",
        "https://www.python.org/"
    ],
    "mode": "both"
};

// Run the Actor and wait for it to finish
const run = await client.actor("keith-apis/web-page-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://example.com/",
        "https://en.wikipedia.org/wiki/Markdown",
        "https://www.python.org/",
    ],
    "mode": "both",
}

# Run the Actor and wait for it to finish
run = client.actor("keith-apis/web-page-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com/",
    "https://en.wikipedia.org/wiki/Markdown",
    "https://www.python.org/"
  ],
  "mode": "both"
}' |
apify call keith-apis/web-page-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,keith-apis/web-page-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qAbCTjr7NaozLApMX/builds/bZE0wIh8cMXBBx0yu/openapi.json