# URL to Markdown API — Clean Page Text for LLMs, RAG & Agents (`highbrow_fame/url-to-markdown-api`) Actor

Fetches any URL and returns clean Markdown (readability-extracted main content), title, metadata and links; optional headless browser for JS pages. Unofficial URL to Markdown API.

- **URL**: https://apify.com/highbrow_fame/url-to-markdown-api.md
- **Developed by:** [yestrue](https://apify.com/highbrow_fame) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown API — Clean Page Text for LLMs, RAG & Agents

Give it web page URLs and get each page back as clean Markdown and plain text — main content only, with title, description, language, canonical URL, author, publish date, headings, link count and word count. Built for LLM pipelines, RAG ingestion and AI agents that need to *read* a page, not scrape it.

**Try it in 1 minute:**

1. Press **Start** — the prefilled input reads a Wikipedia article and the Apify blog.
2. Wait a few seconds.
3. Open the **Output** tab: one record per URL with `markdown`, `text`, `title`, `wordCount` and the metadata, as JSON, CSV or Excel.

Price: $0.50 per 1,000 pages. Pages that cannot be read (404, not HTML, blocked by robots.txt, no text) are free.

**Use it to:**

- Feed web pages to an LLM as Markdown instead of raw HTML — 5–20× fewer tokens, structure kept.
- Build a RAG corpus: one clean document per URL with headings, word count and a stable canonical URL for deduplication.
- Give an AI agent a "read this page" tool through the Apify MCP server or the API.
- Archive articles and docs as Markdown with their title, author and publish date.
- Pull the text of hundreds of pages in one batch from a sheet of URLs.

### How it works

Each URL is fetched with browser-like headers and a 30-second timeout (one retry on a network error through the Apify proxy). Mozilla Readability — the engine behind Firefox Reader View — finds the main content and drops navigation, footer, sidebars and cookie notices; Turndown turns it into Markdown with headings, lists, links, images and GFM tables. Relative links and images are made absolute, so the Markdown works anywhere. Metadata comes from the page head: `<title>`, Open Graph, `<link rel=canonical>`, `lang`, JSON-LD.

Pages that build their text with JavaScript come back as `NO_CONTENT` from a plain fetch — turn on **Use a browser** to render them in headless Chromium first (slower, and worth more compute per page).

`robots.txt` is respected by default: a page the site disallows for bots is skipped with a free `ROBOTS_DISALLOWED` record.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` | — | Web pages to read, one per line (required) |
| `useBrowser` | `false` | Render in headless Chromium first — for JavaScript pages |
| `mainContentOnly` | `true` | Readability main content; off = whole page body |
| `includeLinks` | `false` | Add `links` (href + anchor text, up to 500) |
| `respectRobotsTxt` | `true` | Skip pages the site's robots.txt disallows |
| `waitUntil`, `extraWaitMs` | `load`, `0` | Browser mode: when the page counts as loaded, and an extra wait |
| `timeoutSecs` | `30` | Per-page timeout |
| `outputFormat` | `both` | `markdown`, `text` or both |
| `includeHtml` | `false` | Also deliver the extracted content as cleaned HTML |
| `maxTextLength` | `0` | Cut `markdown`/`text` after N characters |
| `maxConcurrency`, `proxyConfiguration` | `5`, Apify proxy | Speed and retry proxy |

### Output

One record per URL:

| Field | What it is |
|---|---|
| `url`, `canonicalUrl`, `httpStatus`, `contentType` | the page after redirects, its canonical link, HTTP status and content type |
| `title`, `description`, `siteName`, `language` | from the page head (Open Graph preferred) |
| `author`, `publishedAt`, `image` | byline, ISO 8601 publish time and lead image when the page declares them |
| `markdown`, `text` | the content as Markdown (starts with `# Title`) and as plain text |
| `wordCount`, `headings`, `linksCount`, `links` | size, structure and links |
| `mainContentDetected`, `renderedWithBrowser`, `truncated` | how the record was made |
| `input`, `status`, `error`, `scrapedAt` | what you gave, `OK` or the reason, and when |

Status codes: `OK`, `NOT_FOUND`, `NOT_HTML`, `NO_CONTENT`, `ROBOTS_DISALLOWED`, `TOO_LARGE`, `INVALID_INPUT`, `FAILED`. Only `OK` records are charged.

#### Example

```json
{
  "input": "https://en.wikipedia.org/wiki/Web_scraping",
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "httpStatus": 200,
  "title": "Web scraping",
  "description": null,
  "language": "en",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Web_scraping",
  "siteName": "Wikipedia",
  "markdown": "# Web scraping\n\nWeb scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites...\n\n## History\n\n...",
  "text": "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites...",
  "wordCount": 3120,
  "headings": [{ "level": 2, "text": "History" }, { "level": 2, "text": "Techniques" }],
  "linksCount": 412,
  "mainContentDetected": true,
  "renderedWithBrowser": false,
  "status": "OK",
  "scrapedAt": "2026-10-19T12:00:00.000Z"
}
```

### FAQ

**Is this an official API?** No, it's an unofficial API built on public pages.

**Why is a page `NO_CONTENT`?** Its HTML has no text — the content is rendered by JavaScript. Run it again with **Use a browser** on.

**Can it read pages behind a login or a paywall?** No. It reads public pages only and does not log in, solve captchas or bypass bot walls.

**How is this different from a plain fetch?** You get the *content*, not the page: Readability removes menus, footers and cookie notices, and the Markdown keeps headings, lists, tables and links — far fewer tokens for an LLM.

**Does it respect robots.txt?** Yes, by default. A disallowed page is a free `ROBOTS_DISALLOWED` record; you can turn the check off for pages you have the right to read.

**Like it?** A short review on the Store page helps other people find this Actor. Something missing or broken? Tell us on the Issues tab — we read every one.

### Related Actors

- [Sitemap Extractor & Scraper](https://apify.com/highbrow_fame/sitemap-url-extractor) — every URL, date and image from a sitemap
- [PDF Text Extractor & Scraper](https://apify.com/highbrow_fame/pdf-text-extractor) — page-by-page PDF text and metadata
- [YouTube Transcript & Subtitles Scraper](https://apify.com/highbrow_fame/youtube-transcript-fast) — captions as text, SRT or VTT
- [Web Fetch API](https://apify.com/highbrow_fame/web-fetch-api) — Fetch any URL as HTML, text, Markdown or JSON
- [Markdown Crawler API](https://apify.com/highbrow_fame/markdown-site-crawler) — crawl a whole website to Markdown for RAG

# Actor input Schema

## `urls` (type: `array`):

Web pages to read, one per line. Each becomes one record with clean Markdown, plain text, title and metadata.

## `useBrowser` (type: `boolean`):

Render the page in headless Chromium before reading it. Needed only for pages that build their text with JavaScript; slower and more expensive than the default plain fetch.

## `mainContentOnly` (type: `boolean`):

Keep the article or main text (Mozilla Readability) and drop navigation, footer, sidebars and cookie notices. Off: the whole page body.

## `includeLinks` (type: `boolean`):

Add a `links` list (absolute URL and anchor text, up to 500) with every link found on the page. The `linksCount` field is always there.

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site's robots.txt disallows for bots (a free ROBOTS_DISALLOWED record). Turn off only for pages you have the right to read.

## `waitUntil` (type: `string`):

With "Use a browser": when the page counts as loaded. `load` is the safe default; `networkidle` waits for background requests to stop (slower, better for single-page apps).

## `extraWaitMs` (type: `integer`):

With "Use a browser": wait this many milliseconds after loading, for late-rendering content. Up to 15000.

## `timeoutSecs` (type: `integer`):

Give up on a page after this long.

## `outputFormat` (type: `string`):

Which text fields to deliver: `markdown`, plain `text`, or `both`.

## `includeHtml` (type: `boolean`):

Also deliver the extracted content as cleaned HTML (`html`).

## `maxTextLength` (type: `integer`):

Cut `markdown` and `text` after this many characters (`truncated: true` is set). 0: no limit.

## `maxConcurrency` (type: `integer`):

How many pages to read at the same time.

## `proxyConfiguration` (type: `object`):

Used only to try again after a network error: the first try goes out directly.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://blog.apify.com/"
  ],
  "useBrowser": false,
  "mainContentOnly": true,
  "includeLinks": false,
  "respectRobotsTxt": true,
  "waitUntil": "load",
  "extraWaitMs": 0,
  "timeoutSecs": 30,
  "outputFormat": "both",
  "includeHtml": false,
  "maxTextLength": 0,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Every result as a JSON record, with a status for each input.

## `resultsCsv` (type: `string`):

The same records as CSV, for a spreadsheet.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://blog.apify.com/"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("highbrow_fame/url-to-markdown-api").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://blog.apify.com/",
    ],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("highbrow_fame/url-to-markdown-api").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://blog.apify.com/"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call highbrow_fame/url-to-markdown-api --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,highbrow_fame/url-to-markdown-api"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8HJE9XAfd5dfc0eaa/builds/kxIEMCV3TwyEfieQO/openapi.json