# Web Fetch - URL to Markdown for LLM, RAG & AI Agents (`martin10291/url-to-markdown-llm`) Actor

Convert web pages to clean Markdown for LLM prompts, RAG and AI agents. Menus and ads removed; returns title, meta description, language, date, headings, text, word count, optional links and images. Same-site crawl, only-new mode, robots.txt respected. No JavaScript rendering.

- **URL**: https://apify.com/martin10291/url-to-markdown-llm.md
- **Developed by:** [Martin Petrov](https://apify.com/martin10291) (community)
- **Categories:** AI, SEO tools, Agents
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Fetch - URL to Markdown for LLM, RAG & AI Agents

Feeding web pages to **ChatGPT, Claude or a RAG pipeline** is expensive when you paste raw HTML: menus, cookie banners, ads and scripts burn tokens and confuse the model. This actor takes a list of URLs and returns the **main content as clean Markdown**, plus title, meta description, language, author, date, headings, plain text and word count. Cheap, fast, and one call per page.

It uses plain HTTP and a content extractor (trafilatura, with a BeautifulSoup fallback). **No browser, no JavaScript rendering, no proxies.** It sends an honest `RadarFetch/1.0` User-Agent and **respects robots.txt**.

### What you get

- **markdown**: main article content with headings, lists, tables, code blocks and links. Boilerplate removed.
- **text**: the same content as plain text.
- **title, description, language, author, publishedDate, siteName** when the page provides them.
- **headings**: list of `{level, text}` for chunking and outlines.
- **wordCount**, **truncated**, **statusCode**, **finalUrl** (after redirects), **fetchedAt**.
- Optional **internalLinks / externalLinks** and **images** (URL + alt text).
- Optional **same-site crawl** and **only-new mode** (pay only for new or changed pages).

### Use cases

- Give an AI agent or chatbot the content of any public page ("read this URL").
- Build a RAG knowledge base from documentation, blogs or help centers.
- Summarise articles, competitor pages and news for newsletters.
- Monitor pages and get only the ones whose content changed.
- Cut LLM cost: use `maxCharsPerPage` to fit a context window.

### How to use it

1. Open the actor and add your **URLs** (one per line).
2. Optional: turn on **Follow links on the same site** and set **Max pages** and **Max crawl depth**.
3. Optional: set **Max characters per page** to truncate long pages.
4. Click **Start** and download the dataset as JSON, CSV or Excel, or call it from code.

### API example (curl)

Replace `YOUR_APIFY_TOKEN` with your token (Apify Console, Settings, Integrations). This runs the actor and returns the pages in one request:

```bash
curl -X POST "https://api.apify.com/v2/acts/martin10291~url-to-markdown-llm/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://en.wikipedia.org/wiki/Web_scraping"], "maxCharsPerPage": 8000}'
```

### MCP example (Claude, Cursor, any MCP client)

Apify exposes every actor as an MCP tool. Add the actor to the Apify MCP server:

```
https://mcp.apify.com?tools=martin10291/url-to-markdown-llm
```

In Claude Desktop or Cursor add it as a remote MCP server (OAuth login to Apify, or `Authorization: Bearer YOUR_APIFY_TOKEN`). Then ask: "Read https://example.com/docs and summarise it", and the agent calls this actor with `startUrls`.

### Input

| Field | Type | Description |
|---|---|---|
| `startUrls` | array | Pages to convert (http or https, public pages). |
| `maxPages` | integer | Hard limit on pages fetched per run (default 20). |
| `crawl` | boolean | Follow same-domain links from each start URL (default false). |
| `maxDepth` | integer | Crawl depth, 1 = start pages plus pages they link to. |
| `includeLinks` | boolean | Add `internalLinks` and `externalLinks`. |
| `includeImages` | boolean | Add `images` with URL and alt text. |
| `maxCharsPerPage` | integer | Cut markdown and text at this length (0 = no limit). |
| `onlyNew` | boolean | Return only pages that are new or changed since the last run. |
| `monitorId` | string | Separate memory for different watch lists. |
| `includeFailed` | boolean | Also save failed pages with an `error` (charged like normal results). |
| `timeoutSecs` | integer | Per-page timeout (default 20). |

#### Example input

```json
{
  "startUrls": ["https://en.wikipedia.org/wiki/Web_scraping", "https://blog.apify.com/introducing-x402-agentic-payments/"],
  "maxCharsPerPage": 20000,
  "includeLinks": false
}
```

### Output example

```json
{
  "url": "https://blog.apify.com/introducing-x402-agentic-payments/",
  "finalUrl": "https://blog.apify.com/introducing-x402-agentic-payments/",
  "statusCode": 200,
  "fetchedAt": "2026-11-01T22:09:29Z",
  "error": null,
  "title": "Your AI agent just got 20,000+ new tools on x402",
  "description": "...",
  "language": "en",
  "author": "...",
  "publishedDate": "2026-06-30",
  "siteName": "Apify Blog",
  "markdown": "The value of AI agents is proportional to the size of tasks ...",
  "text": "The value of AI agents is proportional to the size of tasks ...",
  "wordCount": 1659,
  "headings": [{"level": 2, "text": "..."}],
  "truncated": false,
  "extractor": "trafilatura"
}
```

A `RUN_SUMMARY` record in the key-value store lists counts (fetched, saved, unchanged, failed, blocked by robots.txt), the failed URLs and `lastError`.

### Pricing

Pay per result: **$1.00 per 1,000 pages** (one result = one page saved to the dataset). Pages that fail are not charged unless you turn on `includeFailed`. In only-new mode unchanged pages are not charged. `maxPages` caps the cost of a run.

### FAQ

**Does it respect robots.txt?** Yes. Pages disallowed for our User-Agent are skipped with the error `blocked by robots.txt`.

**Does it run JavaScript?** No. It reads the HTML the server returns. Pages that build their content only in the browser (some single-page apps) can return little or no text; the result then has an error like "no main content found". This is not a headless browser.

**Can it read pages behind a login, paywall or CAPTCHA?** No. Public pages only. It does not use proxies or try to bypass bot protection, so sites that block automated traffic will return an HTTP error.

**What about PDFs and images?** Only HTML and plain-text pages are converted.

**How polite is it?** At most 5 pages in parallel and at least 1 second between requests to the same domain.

**How does only-new mode work?** The actor stores a hash of each page content under your Monitor ID and returns a page only when the hash changes. Combine it with an Apify Schedule for change alerts.

**Is the Markdown always perfect?** Extraction is automatic and works best on articles, docs and blogs. Listing pages, dashboards and unusual layouts may be incomplete.

### Use it as an integration (chain it after another actor)

This actor can run automatically after any actor whose results contain a URL.

1. Open the source actor in Apify Console and go to the **Integrations** tab.
2. Click **Add integration** -> **Actor**, pick this actor and set **Dataset** to `{{resource.defaultDatasetId}}` (leave `startUrls` empty). Set your other options and save.
3. When the source run finishes, this actor reads its dataset and processes every row. The field is detected automatically; set **Dataset field** if your column has another name.

You can also start it by hand with `datasetId` (pick a dataset or paste its ID) to process an existing dataset.

Good sources to chain from:

- apify/website-content-crawler
- martin10291/medium-articles-scraper
- martin10291/github-trending-scraper
- martin10291/reddit-posts-comments-extractor (link)
- any actor with a url column (news, blogs, docs)

# Actor input Schema

## `startUrls` (type: `array`):

Web pages to convert to clean Markdown, one full URL per entry (http or https). Public pages only, no login pages.

## `maxPages` (type: `integer`):

Hard limit on pages fetched in one run (also limits what you pay for). Failed pages count too.

## `crawl` (type: `boolean`):

When on, the actor also follows links that stay on the same domain as each start URL, up to Max crawl depth and Max pages.

## `maxDepth` (type: `integer`):

Only used when crawling. 1 = start URLs plus the pages they link to.

## `includeLinks` (type: `boolean`):

Add the page links to every result as internalLinks and externalLinks lists (absolute URLs, up to 200 each).

## `includeImages` (type: `boolean`):

Add the page images (absolute URL and alt text, up to 100) to every result.

## `maxCharsPerPage` (type: `integer`):

Cut markdown and text after this many characters to fit LLM context windows. 0 = no limit. The truncated flag tells you when a page was cut.

## `onlyNew` (type: `boolean`):

For monitoring: return (and pay for) a page only when its content is new or changed since the last run with the same Monitor ID.

## `monitorId` (type: `string`):

Separate memory for different watch lists (used with only-new mode).

## `includeFailed` (type: `boolean`):

When on, pages that fail (blocked by robots.txt, HTTP error, timeout) are also saved with an error message and are charged like normal results. When off, failures are only listed in the RUN_SUMMARY record.

## `timeoutSecs` (type: `integer`):

Give up on a page that does not answer within this time.

## `datasetId` (type: `string`):

Optional. Pick a dataset (or paste its ID) to read the input from, for example the result of Google Maps Scraper. When this actor runs as an integration, the dataset of the triggering run is used automatically (payload.resource.defaultDatasetId), so you can leave this empty. If set, the values found in the dataset replace the main input list.

## `datasetField` (type: `string`):

Optional. The item field that holds the value, for example website, url or videoUrl (dot paths like author.url work). Empty = detected automatically.

## `datasetMaxItems` (type: `integer`):

Upper limit of unique values taken from the dataset, to keep the cost under control.

## Actor input object example

```json
{
  "startUrls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://blog.apify.com/introducing-x402-agentic-payments/"
  ],
  "maxPages": 20,
  "crawl": false,
  "maxDepth": 1,
  "includeLinks": false,
  "includeImages": false,
  "maxCharsPerPage": 0,
  "onlyNew": false,
  "monitorId": "default",
  "includeFailed": false,
  "timeoutSecs": 20,
  "datasetMaxItems": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

One item per page: title, metadata, clean Markdown, text, headings and optional links and images.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://blog.apify.com/introducing-x402-agentic-payments/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("martin10291/url-to-markdown-llm").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://blog.apify.com/introducing-x402-agentic-payments/",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("martin10291/url-to-markdown-llm").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://blog.apify.com/introducing-x402-agentic-payments/"
  ]
}' |
apify call martin10291/url-to-markdown-llm --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,martin10291/url-to-markdown-llm"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IeNQsk2ox3EuNjuOP/builds/yxLtHHVeTfujfiI1V/openapi.json