# URL to Markdown for AI Agents: Clean LLM Context (`edgef-4/url-to-markdown-for-ai-agents`) Actor

Clean main-content Markdown from up to 200 URLs a run, for AI agents and RAG: title, token estimate, optional chunks. $2.00 per 1,000 pages converted, failures free. Plain HTTP, no browser: a JavaScript-rendered page returns needs_browser, a PDF not_html. No crawl: it reads only the URLs you give.

- **URL**: https://apify.com/edgef-4/url-to-markdown-for-ai-agents.md
- **Developed by:** [Edge](https://apify.com/edgef-4) (community)
- **Categories:** AI, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Clean main-content Markdown from up to 200 URLs a run, for AI agents and RAG: title, token estimate, optional chunks. $2.00 per 1,000 pages converted, failures free. Plain HTTP, no browser: a JavaScript-rendered page returns needs_browser, a PDF not_html. No crawl: it reads only the URLs you give.

**Price (pay per event):** $0.002 per page converted ($2.00 per 1,000 pages), plus the platform's $0.00005 start event per run. Every status other than ok has no page charge.

**What it does not do:** No browser, no PDF text, no crawl: it reads only the URLs you give (sitemap-url-extractor lists a site's URLs).

**Quickstart** (with your Apify token):

- API: `curl -X POST "https://api.apify.com/v2/acts/edgef-4~url-to-markdown-for-ai-agents/run-sync-get-dataset-items" -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" -d '{"urls":["https://example.com/article"]}'`
- MCP `call-actor`: `{"actor":"edgef-4/url-to-markdown-for-ai-agents","input":{"urls":["https://example.com/article"]}}`
- Python: `ApifyClient(os.environ["APIFY_TOKEN"]).actor("edgef-4/url-to-markdown-for-ai-agents").call(run_input={"urls":["https://example.com/article"]})`

Turn web pages into **clean, main-content-only Markdown** sized for LLM context. Give up to 200 URLs per run and get
one dataset item per URL with the title, canonical URL, language, publish date, word count, token estimate and the
Markdown. You can also cap the length, drop links, or get 800 to 1,200 token chunks for RAG.

**The per-page charge applies only to pages that convert.** SPA shells, PDFs, 404s, robots.txt blocks and timeouts
are reported with a clear status and have no page charge.

### For AI agents

Call it synchronously and get the items back in one HTTP request:

```bash
curl -s -X POST "https://api.apify.com/v2/acts/edgef-4~url-to-markdown-for-ai-agents/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"], "maxChars": 20000, "includeLinks": false}'
```

- The response is a JSON array with one object per unique input URL, in completion order (not input order): match
  results to inputs by `url`. Spellings of one page (host case, default port, fragment, missing scheme) count once.
- `run-sync-get-dataset-items` waits at most 300 s. For long lists (many URLs on one site with a large `Crawl-delay`)
  start a normal run and read its dataset when it finishes; the run keeps converting (and charging) after a sync call
  gives up.
- Use `markdown` when `status == "ok"`. Otherwise read `message`: `needs_browser` means use a browser-based scraper,
  `not_html` means the URL is a file (PDF, image, ...). Only `ok` items are charged.
- Field names are stable and every item has every field (unused ones are `null`, `0`, `false` or `[]`).

### Input

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `urls` | array of strings | required | Absolute http(s) URLs, max 200. A bare domain gets `https://`. `{"url": "..."}` objects also work. Duplicates are processed once, also when spelled differently (host case, default port, fragment, missing scheme). |
| `includeLinks` | boolean | `true` | `false` keeps link text but drops the URLs (fewer tokens). Images always stay as `![alt](url)`. |
| `maxChars` | integer ≥ 500 | none | Cap the Markdown per page. Cuts at the last heading that fits in the second half of the limit, else at the last paragraph break. Sets `truncated: true`. |
| `chunk` | boolean | `false` | Add `chunks`: 800 to 1,200 token pieces along heading boundaries, each with its `headingPath`. |
| `maxConcurrency` | integer 1 to 20 | `5` | Pages fetched in parallel across all domains. |
| `minDelayPerDomainSecs` | number 0.2 to 30 | `1` | Minimum gap between requests to the same domain. A larger robots.txt `Crawl-delay` (up to 10 s) wins. |

Example input:

```json
{
  "urls": ["https://docs.python.org/3/library/asyncio-task.html", "https://example.com/blog/post"],
  "includeLinks": true,
  "maxChars": 40000,
  "chunk": true
}
```

### Output

One dataset item per URL. Example (shortened):

```json
{
  "url": "https://news.example.com/uk/2026/10/flood-barrier-approved?utm_source=x",
  "finalUrl": "https://news.example.com/uk/2026/10/flood-barrier-approved?utm_source=x",
  "status": "ok",
  "message": null,
  "httpStatus": 200,
  "contentType": "text/html",
  "redirectChain": [],
  "title": "City council approves £48m riverside flood barrier",
  "canonicalUrl": "https://news.example.com/uk/2026/10/flood-barrier-approved",
  "language": "en-GB",
  "publishedAt": "2026-10-27T08:15:00+00:00",
  "description": "The £48m scheme will protect 3,200 homes along the Wend.",
  "markdown": "# City council approves £48m riverside flood barrier\n\nCouncillors voted 41 to 9 ...\n\n## What happens next\n\n...",
  "wordCount": 270,
  "tokenEstimate": 442,
  "tokenizer": "chars/4",
  "truncated": false,
  "chunks": [
    {"index": 0, "headingPath": ["City council approves £48m riverside flood barrier"], "markdown": "# City council ...", "tokenEstimate": 441}
  ],
  "fetchedAt": "2026-10-29T13:20:00+00:00"
}
```

The Actor's dataset schema describes every field. `truncated` is also set when a page is cut to fit the platform's
9 MB limit per dataset item (possible with chunk mode on a very long page). `tokenizer` is normally `cl100k_base`; the example shows the
`chars/4` fallback, used only if that encoding cannot load.

#### Status values

| `status` | Meaning | Charged |
| --- | --- | --- |
| `ok` | Converted. `markdown` is set. | ✅ yes |
| `needs_browser` | JavaScript-rendered page (SPA shell) with almost no server-side text. Use a browser-based scraper. | no |
| `not_html` | PDF, image, JSON or another non-HTML file, also one the server mislabelled as HTML. `contentType` says which. | no |
| `http_error` | The server answered 4xx/5xx, or a 3xx that is not a followable redirect such as 300 or 304 (see `httpStatus`). | no |
| `blocked_by_robots` | robots.txt disallows the URL (or robots.txt itself answered 5xx / was unreachable). | no |
| `blocked_address` | The URL, a redirect, or the DNS answer for its host points to a private, loopback, link local or cloud metadata address. Never fetched. | no |
| `timeout` | No complete response within 15 s for one request, no result within 60 s for the URL as a whole (redirects and retries included, waits for the per domain delay not counted), or the run reached its time limit first. | no |
| `fetch_error` | DNS/TLS/connection error, redirect loop, unsupported `Content-Encoding`, body over 10 MB, page with more than 400,000 HTML tags, or a result that could not be stored. | no |
| `invalid_url` | Not an http(s) URL, or it contains credentials (`user:password@`). `message` says which. | no |
| `empty_content` | HTML loaded but had no readable main content. | no |
| `skipped` | Not processed because the run reached the max charge you set. | no |

### What the Markdown looks like

- Main content only. Navigation, headers, footers, sidebars, cookie and consent banners, ads, share buttons,
  newsletter boxes, comments, hidden elements, forms and scripts are removed.
- Starts with `# Title` (added if the content has no H1). Heading levels are kept.
- Tables become Markdown tables (pipes escaped, ragged rows padded, colspans expanded, captions in bold).
- Links become `[text](absolute URL)`; images become `![alt text](absolute URL)` (lazy-loaded `data-src` resolved,
  tracking pixels and base64 placeholders dropped). Spaces, brackets and parentheses in URLs are percent-encoded so a
  page cannot break out of a link target; images that are not `http(s)` keep only their alt text.
- Code blocks are fenced and keep their language when the page declares it (`language-python`, Sphinx `highlight-*`, ...).

### How it fetches

Plain HTTP (httpx), no browser. Redirects are followed manually (max 10) and recorded in `redirectChain`. robots.txt
is checked for every URL and every redirect hop (RFC 9309 rules: 4xx = no rules, 5xx/unreachable = disallow). Requests
to the same domain are spaced by `minDelayPerDomainSecs`. Each request has a 15 s timeout; 429/502/503/504 and network
errors are retried twice (honouring `Retry-After` up to 10 s), and one URL gets at most 60 s of fetching (time spent
waiting for its per domain slot is not counted), so a bad page never stalls the run. Non-HTML responses are detected
from the headers (and file signatures) and their bodies are not downloaded; neither are error pages. Only gzip and
deflate are accepted as `Content-Encoding`, inflated under the same caps: HTML bodies over 10 MB are refused and
robots.txt is read up to 500 KiB. Pages with more than 400,000 HTML tags are not converted, and very long pages are
converted one at a time in bounded slices, so one huge or hostile page cannot exhaust the run's memory. The run stops
30 s before its platform timeout and reports unfinished URLs as `timeout`, so every URL has an item.

#### Safety

Only `http` and `https` URLs are fetched, and never ones with credentials in them. Before every request (the input
URL, each redirect hop and robots.txt) the host is resolved and the request is refused with `blocked_address` if any
address is private, loopback, link local, carrier grade NAT, multicast or reserved, including cloud metadata
(`169.254.169.254`, `fd00:ec2::254`), IPv4 mapped, compatible and translated IPv6 and numeric spellings such as
`http://2130706433/`. IPv6 is fetched only from global unicast addresses (`2000::/3`). The
connection layer resolves the name again, checks again and connects to the checked IP, so DNS rebinding cannot slip
past the first check. Proxy settings from the environment are ignored.

### Pricing (pay per event)

| Event | Price | When |
| --- | --- | --- |
| `apify-actor-start` | 0.00005 USD | Apify's standard start event: once per run at up to 1 GB of memory, once more per extra GB. |
| `page-converted` | 0.002 USD | Once per item with `status: "ok"`. |

Every other status has no page charge, and there is no per-item dataset charge. If you set a maximum cost per run, the
Actor stops converting once it is reached and reports the remaining URLs as `skipped`.

# Actor input Schema

## `urls` (type: `array`):

Required. Page URLs, one per line, such as https://example.com/article (a bare domain like example.com gets https://). At most 200 per run; duplicates (also spellings that differ only in host case, default port or fragment) are processed once. Objects of the form {"url": "..."} are also accepted.

## `includeLinks` (type: `boolean`):

Default true: links stay as \[text]\(absolute URL). False keeps only the link text, which saves tokens. Images are always kept as !\[alt text]\(absolute URL).

## `maxChars` (type: `integer`):

Optional cap on the Markdown length per page, for example 20000 (at least 500). The text is cut at the last heading that fits in the second half of the limit (else at the last paragraph break), and the item gets truncated=true. Empty: no limit.

## `chunk` (type: `boolean`):

Default false. True adds a chunks array: the Markdown split into pieces of 800 to 1,200 tokens (the last one may be smaller) along heading boundaries, each with its heading path. Ready for embedding or RAG.

## `maxConcurrency` (type: `integer`):

Pages fetched in parallel across all domains: default 5, from 1 to 20. Requests to the same domain are always spaced out (see minDelayPerDomainSecs).

## `minDelayPerDomainSecs` (type: `number`):

Minimum seconds between two requests to the same domain: default 1, from 0.2 to 30. A larger Crawl-delay in the site's robots.txt (up to 10 s) takes precedence.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown"
  ],
  "includeLinks": true,
  "chunk": false,
  "maxConcurrency": 5,
  "minDelayPerDomainSecs": 1
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Markdown"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("edgef-4/url-to-markdown-for-ai-agents").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://en.wikipedia.org/wiki/Markdown"] }

# Run the Actor and wait for it to finish
run = client.actor("edgef-4/url-to-markdown-for-ai-agents").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown"
  ]
}' |
apify call edgef-4/url-to-markdown-for-ai-agents --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,edgef-4/url-to-markdown-for-ai-agents"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ypZSbh6kUNIBzP4MA/builds/umxRLocf6fmrRob4B/openapi.json