# Website Content to Markdown — LLM-ready text extractor (`fre.d/url-to-markdown`) Actor

Convert web page URLs into clean, LLM-ready Markdown and JSON. Returns one record per URL with title, markdown, text, links, metadata (author, publishedAt, lang, canonical), wordCount and status. Strips nav/ads/footers, optional JS rendering, respects robots.txt. Failed URLs are never charged.

- **URL**: https://apify.com/fre.d/url-to-markdown.md
- **Developed by:** [Yosuke Sakaguchi](https://apify.com/fre.d) (community)
- **Categories:** AI, Developer tools
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Content to Markdown — LLM-ready text extractor

Convert any list of web page URLs into clean Markdown and structured JSON that is ready to feed into an LLM, a RAG pipeline, or an AI agent. One record per URL: title, main-content Markdown, plain text, links, metadata, word count, and a per-page status. Navigation, ads, cookie banners and footers are stripped using Mozilla Readability (the engine behind Firefox Reader View).

### What you get

Each URL produces exactly one dataset record:

| Field | Type | Description |
|---|---|---|
| `url` | string | The URL you passed in |
| `finalUrl` | string | null | URL after redirects, when different |
| `title` | string | null | Article / page title |
| `markdown` | string | null | Main content as Markdown (ATX headings, fenced code blocks, pipe tables). `null` when `outputFormat` is `json` |
| `text` | string | null | Main content as plain text with paragraph breaks. `null` when `outputFormat` is `markdown` |
| `links` | `{ text, href }[]` | Absolute http(s) links found in the main content (empty when `includeLinks` is false) |
| `metadata` | object | `description`, `author`, `publishedAt` (ISO 8601), `lang`, `canonical`, `siteName`, `image` — from meta tags, Open Graph and JSON-LD |
| `wordCount` | integer | Whitespace-separated words in `text` (for CJK text this is closer to a sentence count) |
| `fetchedAt` | string | ISO 8601 timestamp of the fetch |
| `status` | `ok` | `failed` | `skipped` | `failed` = fetch/extraction error, `skipped` = disallowed by robots.txt or spending limit reached |
| `error` | string | Present only when `status` is not `ok` |

The run always finishes as **SUCCEEDED**, even if some URLs fail. Check `status` per record instead of the run status, so one dead link never breaks your pipeline.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | string\[] | required | Absolute http(s) URLs. Pass many at once; each page is a separate billable event |
| `outputFormat` | `markdown` | `json` | `both` | `both` | `markdown` returns only the `markdown` field; `json` returns `text` + structured fields without Markdown; `both` returns everything |
| `renderJs` | boolean | `false` | Load the page in headless Chromium before extraction. Use for single-page apps or content injected by JavaScript. Slower and billed at the higher rate |
| `includeLinks` | boolean | `true` | Collect links from the main content |
| `includeMetadata` | boolean | `true` | Extract page metadata |
| `respectRobotsTxt` | boolean | `true` | Skip URLs disallowed by the site's robots.txt |
| `maxConcurrency` | integer | `5` | Upper bound on parallel fetches (1–20). Same-host requests are additionally spaced out |
| `requestTimeoutSecs` | integer | `30` | Per-page load timeout (5–180) |

Example input:

```json
{
  "urls": [
    "https://example.com",
    "https://blog.apify.com/fixing-actors-invisible-to-agents/"
  ],
  "outputFormat": "both",
  "renderJs": false
}
```

### Output example

```json
{
  "url": "https://blog.apify.com/fixing-actors-invisible-to-agents/",
  "finalUrl": null,
  "title": "What Apify MCP does to your Actor before an AI agent sees it",
  "markdown": "This article was written by [Olivier Reynaud](https://apify.com/silentflow) as part of [Write for Apify](https://apify.com/resources/write-for-apify) ...\n\n## The MCP server rewrites your Actor\n\n...",
  "text": "This article was written by Olivier Reynaud as part of Write for Apify ...",
  "links": [
    { "text": "Olivier Reynaud", "href": "https://apify.com/silentflow" },
    { "text": "Apify Store", "href": "https://apify.com/store" }
  ],
  "metadata": {
    "description": "Olivier Reynaud audited his Apify Actors through the Apify MCP server to see how AI agents find and read them, and what developers should fix.",
    "author": "Olivier Reynaud",
    "publishedAt": "2026-09-24T13:25:20.000Z",
    "lang": "en",
    "canonical": "https://blog.apify.com/fixing-actors-invisible-to-agents/",
    "siteName": "Apify Blog",
    "image": "https://storage.ghost.io/.../i-audited-151-actors-for-ai-agents.png"
  },
  "wordCount": 3641,
  "fetchedAt": "2026-10-15T01:58:12.345Z",
  "status": "ok"
}
```

A failed URL looks like this and is **not charged**:

```json
{
  "url": "https://this-domain-does-not-exist.example/",
  "status": "failed",
  "error": "getaddrinfo ENOTFOUND this-domain-does-not-exist.example",
  "markdown": null,
  "text": null,
  "links": [],
  "wordCount": 0,
  "fetchedAt": "2026-10-15T01:58:12.345Z"
}
```

### Pricing

Pay per event. You are charged only for pages that return `status: "ok"`. Failed and skipped pages are free.

| Event | Price | When |
|---|---|---|
| `page-processed` | $0.001 per page | `renderJs: false` (default, plain HTTP fetch) |
| `page-processed-js` | $0.003 per page | `renderJs: true` (headless Chromium rendering) |

Example: converting 1,000 static pages costs $1.00; 1,000 JavaScript-rendered pages cost $3.00. You can cap spending per run with the *Maximum total charge* setting in Apify Console or the `maxTotalChargeUsd` API parameter; when the cap is reached, remaining URLs are returned with `status: "skipped"`.

### Usage

#### Apify Console

Open the Actor, paste your URLs into *Page URLs*, click *Start*, and download the dataset as JSON, CSV or Excel.

#### JavaScript (apify-client)

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('fre.d/url-to-markdown').call({
  urls: ['https://example.com', 'https://blog.apify.com/fixing-actors-invisible-to-agents/'],
  outputFormat: 'markdown',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const page of items) {
  if (page.status === 'ok') console.log(page.title, page.markdown.length);
  else console.warn(page.url, page.error);
}
```

#### Python (apify-client)

```python
from apify_client import ApifyClient

client = ApifyClient(token=os.environ["APIFY_TOKEN"])

run = client.actor("fre.d/url-to-markdown").call(run_input={
    "urls": ["https://example.com", "https://blog.apify.com/fixing-actors-invisible-to-agents/"],
    "outputFormat": "both",
})

for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page["status"] == "ok":
        print(page["title"], page["wordCount"])
    else:
        print("failed:", page["url"], page["error"])
```

#### AI agents via MCP

Add the Apify MCP server to your agent (Claude Desktop, Cursor, LangGraph, etc.) and allow this Actor:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=fre.d/url-to-markdown",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}
```

The agent can then call the tool with `{"urls": ["https://..."]}` and read `markdown` or `text` directly from the result. Because `urls` is an array, an agent can fetch a whole reading list in one tool call.

#### HTTP API

```bash
curl -X POST "https://api.apify.com/v2/acts/fredman~url-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"], "outputFormat": "markdown"}'
```

### How it works

1. URLs are validated, de-duplicated, and checked against the site's `robots.txt` (fetched once per host).
2. Pages are fetched over HTTP with Crawlee's CheerioCrawler, or rendered in headless Chromium when `renderJs` is on. Requests to the same host are spaced out and total concurrency is capped.
3. The main content is identified with Mozilla Readability. If Readability finds nothing, the Actor falls back to `<main>`, `<article>` or `<body>` with navigation and scripts removed.
4. The content is converted to Markdown with Turndown (fenced code blocks keep their language, simple tables become pipe tables) and to plain text with paragraph breaks.
5. Metadata is merged from `<meta>` tags, Open Graph, `<link rel="canonical">`, `<time datetime>`, and JSON-LD (`Article`, `BlogPosting`, ...).

The Actor identifies itself with the User-Agent `Mozilla/5.0 (compatible; WebsiteContentToMarkdown/0.1; +https://apify.com/fre.d/url-to-markdown)`. Site owners can block it with a `User-agent: WebsiteContentToMarkdown` rule in robots.txt.

### Limitations

- Only HTML pages are supported. PDFs, images and other binary resources return `status: "failed"`.
- Pages behind a login, paywall or CAPTCHA cannot be extracted.
- Readability is tuned for article-like pages. For product listings, search results or dashboards, consider a dedicated scraper.
- `wordCount` counts whitespace-separated tokens; for Japanese, Chinese or Korean text it will be much lower than the character count.

### robots.txt and responsible use

By default the Actor respects `robots.txt` and skips disallowed URLs. Turning `respectRobotsTxt` off is intended for sites you own or have permission to crawl.

The extracted text remains the property of the original publisher. You are responsible for how you store, redistribute or otherwise use the content you retrieve with this Actor, including compliance with the target site's terms of service and applicable copyright and data-protection law. Fetching a page for personal analysis or feeding it to an LLM as context is a different matter from republishing it; when in doubt, link to the source instead of redistributing the text.

### Support

Open an issue on the Actor's *Issues* tab in Apify Store, or email the maintainer listed on the Actor page. Feature requests (custom selectors, PDF support, chunking for embeddings) are welcome.

# Changelog

This Actor's version history is a separate document: https://apify.com/fre.d/url-to-markdown/changelog.md

# Actor input Schema

## `urls` (type: `array`):

List of absolute http(s) URLs to convert. One output record is produced per URL. Pass many URLs in one run; each page is a separate billable event.

## `outputFormat` (type: `string`):

Which content fields to include in each record. 'markdown' returns the markdown field only, 'json' returns text plus structured fields (links, metadata) without markdown, 'both' returns everything.

## `renderJs` (type: `boolean`):

Load pages in a headless Chromium browser before extraction. Enable for single-page apps or pages whose content is injected by JavaScript. Slower and billed at the higher 'page-processed-js' rate.

## `includeLinks` (type: `boolean`):

Collect outbound links found in the main content as an array of {text, href}.

## `includeMetadata` (type: `boolean`):

Extract page metadata (description, author, publishedAt, lang, canonical) from meta tags, Open Graph and JSON-LD.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs that the site's robots.txt disallows for this crawler's user agent. Skipped URLs are returned with status 'skipped' and are not charged. Keep enabled unless you own the target site.

## `maxConcurrency` (type: `integer`):

Upper bound on pages fetched in parallel across all hosts. Requests to the same host are additionally limited to a few at a time.

## `requestTimeoutSecs` (type: `integer`):

Maximum time to wait for one page to load before marking it as failed.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://blog.apify.com/fixing-actors-invisible-to-agents/"
  ],
  "outputFormat": "both",
  "renderJs": false,
  "includeLinks": true,
  "includeMetadata": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `pages` (type: `string`):

All page records as JSON. Each record has url, status, title, markdown, text, links, metadata, wordCount and fetchedAt.

## `pagesMarkdown` (type: `string`):

Same records with only url, title and markdown fields.

## `pagesCsv` (type: `string`):

Flat CSV with url, status, title, wordCount and error.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com",
        "https://blog.apify.com/fixing-actors-invisible-to-agents/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fre.d/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://example.com",
        "https://blog.apify.com/fixing-actors-invisible-to-agents/",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("fre.d/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com",
    "https://blog.apify.com/fixing-actors-invisible-to-agents/"
  ]
}' |
apify call fre.d/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fre.d/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/n9idCyazty8m6GEiQ/builds/bLieNg1TwpeUawVux/openapi.json