# URL to Markdown for AI Agents and RAG (`itayozer9/url-to-markdown`) Actor

Turn any list of web page URLs into clean Markdown for LLMs and RAG: main content only, plus title, language, word and token counts, links and a needsJavaScript flag. Plain HTTP, obeys robots.txt, no browser.

- **URL**: https://apify.com/itayozer9/url-to-markdown.md
- **Developed by:** [Itay Ozer](https://apify.com/itayozer9) (community)
- **Categories:** AI, Agents, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown for AI Agents and RAG

Give it a list of web page URLs. Get back each page's main content as clean Markdown, ready to paste into an LLM prompt, a vector database or a RAG pipeline. Navigation, cookie bars, scripts and styling are stripped out.

It reads pages over plain HTTP, the way a search engine bot does. There is no headless browser and no proxy in the loop, so it is fast (30 Wikipedia articles in about 10 seconds in our tests) and cheap: **$0.40 per 1,000 pages**, and pages that fail cost nothing.

It is a polite crawler: it reads each site's `robots.txt` and skips any page the site disallows, following RFC 9309, the internet standard for robots.txt.

### What you get

One result per URL you send (a URL listed twice is read and charged once):

- `markdown`: the main content of the page as Markdown, with headings, lists, tables, code blocks and links (all links made absolute). It starts with the page title as a heading.
- `title`, `description`, `language` (as the page declares it, such as `en` or `fr-CA`) and `publishedAt` when the page states a publish date
- `wordCount` and `tokenEstimate`, so you can budget LLM context before you send anything
- `links`: up to 200 unique absolute links found on the whole page, handy for deciding what to read next
- `needsJavaScript`: `true` when the page shows little or no text without a browser (single-page apps)
- `status` and `error`: `ok`, or a plain-language reason the page was not converted
- `url`, `finalUrl` (after redirects), `statusCode`, `contentType`, `fetchedAt`

### Input

| Field | Required | Example | Notes |
|---|---|---|---|
| `urls` | yes | `["https://en.wikipedia.org/wiki/Markdown"]` | Full addresses, up to 10,000 per run. One result per unique URL |

```json
{
    "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://www.rfc-editor.org/rfc/rfc9309.html"
    ]
}
```

Plain text pages (`text/plain`, `text/markdown`) come back as they are. PDFs, images and other files are skipped with status `unsupported-content-type`.

### Output example

A real result, with the Markdown and links shortened:

```json
{
    "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
    "finalUrl": "https://www.rfc-editor.org/rfc/rfc9309.html",
    "status": "ok",
    "statusCode": 200,
    "error": null,
    "contentType": "text/html",
    "title": "RFC 9309: Robots Exclusion Protocol",
    "description": "This document specifies and extends the \"Robots Exclusion Protocol\" method originally defined by Martijn Koster in 1994 ...",
    "language": "en",
    "publishedAt": null,
    "markdown": "# RFC 9309: Robots Exclusion Protocol\n\n| RFC 9309 | Robots Exclusion Protocol (REP) | September 2022 |\n| --- | --- | --- |\n...\n\n## Abstract\n\nThis document specifies and extends the \"Robots Exclusion Protocol\" method ...",
    "wordCount": 2940,
    "tokenEstimate": 5998,
    "links": ["https://www.rfc-editor.org/rfc/rfc9309", "https://www.rfc-editor.org/info/rfc9309", "..."],
    "needsJavaScript": false,
    "fetchedAt": "2026-10-31T16:05:12.016Z"
}
```

A page the site's robots.txt disallows still gets a row, so you always know what happened to every URL:

```json
{
    "url": "https://en.wikipedia.org/wiki/Special:Random",
    "status": "blocked-by-robots",
    "error": "https://en.wikipedia.org/robots.txt disallows this page for crawlers, so it was skipped.",
    "markdown": null,
    "wordCount": 0
}
```

| `status` | Meaning | Charged |
|---|---|---|
| `ok` | Converted | yes |
| `needs-javascript` | Little or no text without a browser | no |
| `empty` | The page has no readable text | no |
| `blocked-by-robots` | The site's robots.txt disallows it, or robots.txt had a server error | no |
| `http-error` | The site answered 4xx or 5xx (for example 403 or 404) | no |
| `unsupported-content-type` | Not HTML or plain text (PDF, image, ...) | no |
| `too-many-redirects` | More than 5 redirects | no |
| `fetch-failed` | No answer (timeout, unknown host) | no |
| `invalid-url` | Not a public http or https address | no |

### Pricing

Pay per event:

| Event | Price | When |
|---|---|---|
| Page converted | $0.0004 | Every result with status `ok` |
| Actor start | $0.00005 | Once per run |

Example: a run with 1,000 URLs where 950 pages convert costs 950 x $0.0004 + $0.00005 = $0.38005.

The actor checks your run's maximum charge before it fetches anything and only reads as many URLs as that covers, so a run never goes over the limit you set.

### Limits

- Pages are read without running JavaScript. Sites that build their content in the browser come back with `needsJavaScript: true` and are not charged. Use a browser-based actor for those.
- Sites that block bots (for example with a 403 page or a challenge wall) return `http-error`. There is no proxy rotation or unblocking.
- Pages are read up to 4 MB of HTML. Very large pages (over 5,000 HTML elements) use a faster extraction: the page's main area with menus, sidebars and footers removed.
- `tokenEstimate` is an estimate, not a tokenizer: within about 6% of GPT-4o's tokenizer on English pages in our tests, rougher for other scripts.
- It reads the URLs you send. It does not crawl a whole site or run a web search.
- Up to 8 pages are fetched at a time. A URL listed more than once is read and charged once.

### FAQ

**How is this different from a full web crawler or a browser-based scraper?**
It does one job: URL in, clean Markdown out. No browser means it starts fast and costs a fraction of browser-based tools. The trade-off is that pages that only render with JavaScript are flagged instead of converted.

**Does it respect robots.txt?**
Yes, always, and it cannot be turned off. It identifies itself as `UrlToMarkdownBot` with a link to this page, reads each site's robots.txt once per run, and follows RFC 9309: a missing robots.txt means everything is allowed, and a robots.txt that answers with a server error means the site is skipped.

**Why did a URL come back with status `http-error` 403?**
The site refused automated requests. Nothing was charged for it.

**Can I call it from an AI agent, MCP client or the API?**
Yes. Input is a single list of URLs, every field in the output has a fixed type, and it runs with limited permissions, so it needs no access to your other Apify data. It works from the Apify API, schedules, integrations and MCP clients.

**Is there personal data in the results?**
The actor returns what is on the pages you choose. It targets no site by default and adds nothing of its own. You are responsible for having the right to use the pages you send it.

**Something broke or a page converted badly. What do I do?**
Open an issue on the Issues tab with your run link and the URL. Issues are answered within a few days.

# Actor input Schema

## `urls` (type: `array`):

Web pages to convert, one full address per line (https://...). Up to 10,000 per run. Each URL gives one result. You pay only for pages that come back with content; failed, blocked and JavaScript-only pages are free.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

One item per URL: Markdown, title, language, word and token counts, links, status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://www.rfc-editor.org/rfc/rfc9309.html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("itayozer9/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://www.rfc-editor.org/rfc/rfc9309.html",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("itayozer9/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://www.rfc-editor.org/rfc/rfc9309.html"
  ]
}' |
apify call itayozer9/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,itayozer9/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kvbvbTN7LGcOfvXd2/builds/gT3sox7wAEBE3zNsR/openapi.json