# Agent Fetch: URL to Clean Markdown (`omargnagy/agent-fetch-markdown`) Actor

Turn any URL into clean, LLM-ready markdown: main content only, plus title, canonical URL, language, word count and links. Optional headless rendering for JavaScript pages, and deterministic JSON Schema field extraction with no LLM and no API key. robots.txt respected, failures come back labelled.

- **URL**: https://apify.com/omargnagy/agent-fetch-markdown.md
- **Developed by:** [Omar Nagy](https://apify.com/omargnagy) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page fetcheds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Agent Fetch: URL to Clean Markdown

Give it a URL and get back the page as clean markdown an LLM can read straight away. No navigation, no cookie bar, no sidebar, no footer, no script tags. Just the main content, plus the title, the canonical URL, the language, a word count and the links that live inside the text.

Up to 25 URLs per run. Plain HTTP by default, a headless browser when a page needs one, and an optional JSON Schema that pulls named fields out of the page without calling any LLM.

### Who this is for

**AI agents that need to read a web page mid-task.** An agent holds a URL and needs the text, not the HTML. Feeding it raw HTML burns thousands of tokens on markup and menus before it reaches a sentence worth reading. This Actor is the step in between: one call, one flat record, markdown that drops straight into a prompt.

It is also for the plumbing around that:

- **RAG ingestion.** Turn a list of documentation pages into markdown chunks with titles, canonical URLs and word counts already attached.
- **Research pipelines.** Pull ten sources into one dataset and summarise them in a single pass.
- **Link and content monitoring.** Track what a page says and which pages it points at over time.
- **Structured scraping without an LLM bill.** Hand it a JSON Schema and get author, published date and description back from the page's own JSON-LD, deterministically.

### What you get per URL

| Field | Type | Always present | Notes |
|---|---|---|---|
| `url` | string | yes | exactly the URL you asked for |
| `finalUrl` | string | yes | after redirects |
| `status` | string | yes | see the status table below |
| `httpStatus` | integer | yes | `0` when no response was received |
| `method` | string | yes | `http` or `browser` |
| `fetchedAt` | string | yes | ISO 8601 |
| `title` | string | yes | from the readability pass, falling back to the page title |
| `canonicalUrl` | string | yes | `rel=canonical`, falling back to `og:url` |
| `language` | string | yes | from `html lang`, falling back to `og:locale` |
| `markdown` | string | yes | the main content |
| `wordCount` | integer | yes | words in the markdown |
| `charCount` | integer | yes | characters in the markdown, after any truncation |
| `linkCount` | integer | yes | links inside the extracted content |
| `truncated` | boolean | yes | true when `maxMarkdownChars` cut the text |
| `durationMs` | integer | yes | time spent on this URL |
| `reason` | string | on failures only | why this URL did not produce content |
| `links` | array | when `includeLinks` | `{ url, text, external }` |
| `article` | object | when `includeArticleJson` | excerpt, byline, site name, dates, image, JSON-LD |
| `extracted` | object | when a schema found something | your fields, coerced to your declared types |
| `extractedSources` | object | with `extracted` | where each field came from |
| `extractedFieldCount` | integer | when a schema was given | how many fields were found |

Optional keys are left out rather than set to null, so the dataset never grows a column that is empty in every row.

#### Status values

| `status` | Meaning | Charged |
|---|---|---|
| `ok` | content extracted | yes |
| `blocked_by_robots` | robots.txt disallows this path | no |
| `http_error` | server answered 4xx or 5xx | no |
| `timeout` | no response inside `timeoutSecs` | no |
| `fetch_error` | DNS, TLS or connection failure | no |
| `invalid_url` | not http or https, or a private address | no |
| `unsupported_content_type` | a PDF, image, archive or other binary | no |
| `render_unavailable` | `render` was on and the browser would not start | no |
| `empty_content` | fetched fine, held no readable text | no |

A bad URL is a record, never a crashed run. One dead link out of twenty-five costs you nothing and does not touch the other twenty-four.

### Input

Minimum:

```json
{
  "urls": ["https://en.wikipedia.org/wiki/Markdown"]
}
```

Everything:

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://docs.python.org/3/tutorial/introduction.html"
  ],
  "render": false,
  "respectRobotsTxt": true,
  "timeoutSecs": 30,
  "includeLinks": true,
  "includeArticleJson": true,
  "maxMarkdownChars": 120000,
  "schema": {
    "properties": {
      "author": { "type": "string" },
      "datePublished": { "type": "string", "format": "date" },
      "siteName": { "type": "string" }
    }
  }
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `urls` | array of strings | | **required**, 1 to 25, http or https only |
| `render` | boolean | `false` | load every URL in a headless Chromium instead of a plain fetch |
| `respectRobotsTxt` | boolean | `true` | a disallowed path returns `blocked_by_robots` and is not charged |
| `timeoutSecs` | integer 5 to 120 | `30` | per URL |
| `userAgent` | string | `AgentFetchBot/1.0` | also used to match robots.txt groups |
| `includeLinks` | boolean | `true` | adds the `links` array |
| `includeArticleJson` | boolean | `false` | adds the `article` object |
| `maxMarkdownChars` | integer 500 to 500000 | `120000` | cut at a paragraph boundary |
| `schema` | object | | JSON Schema with a `properties` map |

Conditional behaviour, all of it stated on the input form too:

- `render: true` charges a **render** event per page instead of a **fetch** event, and runs pages one at a time instead of four at a time.
- `schema` is only acted on when it is an object with a `properties` map. Give it one and a page that yields at least one field also charges an **extract** event.
- `includeLinks` and `includeArticleJson` only add keys to records whose `status` is `ok`.
- `urls` over 25, a non-http scheme, or a private or loopback address is rejected as a caller mistake before any request goes out.

### Output

Real records from a run on 5 September 2026.

A documentation page. The `markdown` value below is the verbatim first 559 characters of an 18183 character field:

```json
{
  "url": "https://docs.python.org/3/tutorial/introduction.html",
  "finalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "status": "ok",
  "httpStatus": 200,
  "method": "http",
  "fetchedAt": "2026-09-04T22:41:35.655Z",
  "title": "3. An Informal Introduction to Python",
  "canonicalUrl": "https://docs.python.org/3/tutorial/introduction.html",
  "language": "en",
  "wordCount": 2834,
  "charCount": 18183,
  "linkCount": 19,
  "truncated": false,
  "durationMs": 1121,
  "markdown": "In the following examples, input and output are distinguished by the presence or absence of prompts ([\\>>>](https://docs.python.org/3/glossary.html#term-0) and [\u2026](https://docs.python.org/3/glossary.html#term-...)): to repeat the example, you must type everything after the prompt, when the prompt appears; lines that do not begin with a prompt are output from the interpreter. Note that a secondary prompt on a line by itself in an example means you must type a blank line; this is used to end a multi-line command.\n\nYou can use the \u201cCopy\u201d button (it appears",
  "links": [
    {
      "url": "https://docs.python.org/3/glossary.html#term-0",
      "text": ">>>",
      "external": false
    },
    {
      "url": "https://docs.python.org/3/library/functions.html#int",
      "text": "int",
      "external": false
    }
  ]
}
```

A Wikipedia article from the same run, metadata only. Its `markdown` field holds 40740 characters of converted article text, including the infobox rendered as a table:

```json
{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "status": "ok",
  "httpStatus": 200,
  "method": "http",
  "fetchedAt": "2026-09-04T22:41:35.638Z",
  "title": "Markdown",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "language": "en",
  "wordCount": 5334,
  "charCount": 40740,
  "linkCount": 204,
  "truncated": false,
  "durationMs": 1793
}
```

A URL that robots.txt disallows:

```json
{
  "url": "https://docs.python.org/2.7/index.html",
  "status": "blocked_by_robots",
  "httpStatus": 0,
  "method": "http",
  "markdown": "",
  "wordCount": 0,
  "reason": "robots.txt at https://docs.python.org/robots.txt disallows /2.7/index.html for this user agent."
}
```

With a schema, on a news article that publishes JSON-LD:

```json
{
  "extractedFieldCount": 6,
  "extracted": {
    "headline": "Toxic Gaslighting: How 3M Executives Convinced a Scientist the Forever Chemicals She Found in Human Blood Were Safe",
    "author": "Sharon Lerner, Haruka Sakaguchi",
    "datePublished": "2024-05-20T10:00:00+00:00",
    "siteName": "ProPublica",
    "wordCount": 8169
  },
  "extractedSources": {
    "headline": "json-ld",
    "author": "json-ld",
    "datePublished": "json-ld",
    "siteName": "json-ld",
    "wordCount": "json-ld"
  }
}
```

`extractedSources` is there so you can tell a value the page published from a value a heuristic guessed at. `json-ld` and `meta` are the page's own machine-readable claims. `page:pairs`, `text:label` and `text:pattern` come from parsing what a human would read.

The run also writes a summary to the key-value store under `OUTPUT`:

```json
{
  "ok": true,
  "requested": 3,
  "returned": 3,
  "succeeded": 3,
  "byStatus": { "ok": 3 },
  "charged": { "fetch": 3, "render": 0, "extract": 0 },
  "totalWords": 16535,
  "tookMs": 1799
}
```

The `examples/` folder of the source holds two full sample datasets: every failure mode in one run, and the same JavaScript-rendered page with `render` off and on side by side. Full markdown output is on the dataset of any run you start.

### How the extraction works

1. Fetch the page, over plain HTTP or in Chromium.
2. Harvest the metadata the page publishes: JSON-LD, meta and microdata tags, definition lists, two-column table rows, and the visible text. This happens **before** step 3, because the readability pass rewrites the document.
3. Run a readability pass to find the main content and drop the furniture. If it declines the page, fall back to the body with navigation, headers, footers, sidebars and forms removed.
4. Convert that HTML to markdown, with GitHub-flavoured tables, fenced code blocks and inline links.
5. If you supplied a schema, match each property against the harvest in this order: JSON-LD, then meta and microdata, then structural label and value pairs, then labelled lines of text, then a format-driven pattern. Coerce to the declared type, check any `enum`, and drop anything that does not fit.

JSON-LD `@id` references are resolved against the rest of the page's graph, so `"author": {"@id": "..."}` comes back as the person's name rather than an internal URL.

### Pay per event

**Pricing is not set yet.** This Actor is published with pay-per-event billing wired up and no rates configured, so nothing is being charged today. When rates are published they will appear on the Pricing tab, and these are the events they will apply to:

| Event | Fires when |
|---|---|
| `fetch` | one page was retrieved over plain HTTP and converted to markdown |
| `render` | one page was loaded in the headless browser and converted to markdown |
| `extract` | one page produced at least one field from your schema |

A page charges `fetch` **or** `render`, never both. `extract` is on top, and only when the schema actually found something.

Nothing is charged for a URL that was blocked by robots.txt, timed out, answered with an error status, was refused by the allowlist, or held no readable text. You pay for content, not for attempts.

### Limits and things worth knowing

- **25 URLs per run.** Above that the run stops with a message telling you to split the list. Chain runs, or schedule them.
- **Datacenter egress only, no proxy.** This Actor never asks for a residential IP. A site that blocks datacenter traffic will answer 403 and you get an `http_error` record. If you need residential IPs, this is not the tool.
- **robots.txt is fetched once per host and cached for the run.** Missing, empty or unreachable means allowed, which is what the standard says. The user agent token matched against robots.txt groups is `AgentFetchBot`, matched exactly, not by substring.
- **Redirects are re-checked.** If a URL redirects to a different origin, that origin's robots.txt is consulted before anything is extracted or charged.
- **Private and loopback addresses are refused.** `localhost`, `127.x`, `10.x`, `192.168.x`, `172.16-31.x`, `169.254.x`, IPv6 loopback and unique-local, `.local` and `.internal` all return `invalid_url` without a request being made.
- **12MB page ceiling.** Bigger responses return `unsupported_content_type`.
- **`links` covers the extracted content, not the whole page.** Site navigation and footer link farms are deliberately excluded, and anchors pointing back into the same page are not counted. The list is capped at 500 per page.
- **`render` is slower and heavier.** Rendered pages run one at a time and load with images, fonts and media blocked. Leave it off unless a plain fetch comes back nearly empty.
- **Markdown is not a pixel-perfect copy.** Tables, code blocks, headings, lists and links survive. Complex multi-column layouts, canvas, and content behind a login do not.
- **Public pages only.** No login, no cookies, no session. What you get is what an anonymous visitor gets.

### What it does not do

It does not use an LLM, does not need any API key, does not log in to anything, does not run in Standby mode, does not use residential proxies, does not crawl links it finds, does not read PDFs or images, and does not extract fields the page never stated.

### FAQ

**Why is my markdown nearly empty?**
The page probably builds its content with JavaScript. Set `render: true` and run it again.

**Why did I get `blocked_by_robots` on a page I can open in a browser?**
robots.txt speaks to automated clients, not to browsers. If you own the site or are licensed to crawl it, set `respectRobotsTxt: false`.

**Does the schema use an LLM?**
No. It reads what the page publishes about itself and parses its structure. It is deterministic, it is free of API keys, and it returns nothing when the page does not say it. For fields a page never states, use an LLM step after this one.

**Can I get the raw HTML back?**
No. This Actor returns markdown and metadata. If you need the HTML, that is a different tool.

**Can I run it on a schedule?**
Yes. Use Apify's scheduler, or call it from your own job through the API or the MCP server.

**Why is it not a Standby HTTP API?**
Deliberate. Batch runs with pay-per-event billing and limited permissions are the shape agentic-payment rails expect, and Standby would break that shape.

***

Built by Omar Nagy. Part of an agent-native data-tool series on Apify.

# Actor input Schema

## `urls` (type: `array`):

The pages to convert, up to 25 per run. Only http:// and https:// are accepted; anything else is rejected before a single request is made. Duplicates are removed and the fragment (#section) is dropped, so the same page is never fetched or charged twice.

## `render` (type: `boolean`):

Off by default. Leave it off for normal pages: a plain HTTP fetch is faster and cheaper. Turn it on only when a page builds its content with JavaScript and the plain fetch comes back nearly empty. When on, every URL in the run is loaded in the Chromium that ships inside this Actor's image, and each page is charged as a render event instead of a fetch event. No proxy is used in either path.

## `respectRobotsTxt` (type: `boolean`):

On by default. The robots.txt of each host is fetched once per run and cached. A path that is disallowed for this Actor's user agent comes back as a record with status blocked\_by\_robots and is not charged. If robots.txt is missing, empty or unreachable, the URL is treated as allowed, which is what the standard says. Switch this off only for hosts you own or are licensed to crawl.

## `timeoutSecs` (type: `integer`):

How long a single page may take before it is given up on. A URL that runs out of time becomes a record with status timeout and is not charged; the rest of the run continues.

## `userAgent` (type: `string`):

Sent with every request and used when matching robots.txt groups. Leave it as it is unless a site has asked you to identify differently.

## `includeLinks` (type: `boolean`):

On by default. Adds a links array to each successful record: the absolute URL, the anchor text and whether the link points off the page's own host. linkCount is always returned even when this is off.

## `includeArticleJson` (type: `boolean`):

Off by default. Adds an article object holding the page metadata that the readability pass and the page head expose: excerpt, byline, site name, published and modified time, description, lead image, and any JSON-LD blocks found in the page. Only the keys that actually have a value are included.

## `maxMarkdownChars` (type: `integer`):

Markdown longer than this is cut at a paragraph boundary and a truncation marker is appended. charCount always reports the length after truncation. Raise it for long reference pages, lower it to keep records inside a small context window.

## `schema` (type: `object`):

Optional. A JSON Schema object with a properties map, for example {"properties":{"author":{"type":"string"},"datePublished":{"type":"string","format":"date"}}}. Each property is looked for in the page's JSON-LD, its meta and microdata tags, its definition lists and tables, and finally in labelled lines of the page text, then coerced to the declared type. This is deterministic parsing, not an LLM: no model is called, no API key is needed, and a field that cannot be found honestly is left out rather than guessed. When a schema is given and at least one field is found, the record gains an extracted object and an extract event is charged. Leave it empty to skip extraction entirely.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown"
  ],
  "render": false,
  "respectRobotsTxt": true,
  "timeoutSecs": 30,
  "userAgent": "Mozilla/5.0 (compatible; AgentFetchBot/1.0; +https://apify.com/omargnagy/agent-fetch-markdown)",
  "includeLinks": true,
  "includeArticleJson": false,
  "maxMarkdownChars": 120000,
  "schema": {
    "properties": {
      "author": {
        "type": "string"
      },
      "datePublished": {
        "type": "string",
        "format": "date"
      },
      "description": {
        "type": "string"
      }
    }
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://docs.python.org/3/tutorial/introduction.html",
        "https://www.propublica.org/article/3m-forever-chemicals-pfas-pfos-inside-story"
    ],
    "render": false,
    "respectRobotsTxt": true,
    "timeoutSecs": 30,
    "userAgent": "Mozilla/5.0 (compatible; AgentFetchBot/1.0; +https://apify.com/omargnagy/agent-fetch-markdown)",
    "includeLinks": true,
    "includeArticleJson": false,
    "maxMarkdownChars": 120000
};

// Run the Actor and wait for it to finish
const run = await client.actor("omargnagy/agent-fetch-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://docs.python.org/3/tutorial/introduction.html",
        "https://www.propublica.org/article/3m-forever-chemicals-pfas-pfos-inside-story",
    ],
    "render": False,
    "respectRobotsTxt": True,
    "timeoutSecs": 30,
    "userAgent": "Mozilla/5.0 (compatible; AgentFetchBot/1.0; +https://apify.com/omargnagy/agent-fetch-markdown)",
    "includeLinks": True,
    "includeArticleJson": False,
    "maxMarkdownChars": 120000,
}

# Run the Actor and wait for it to finish
run = client.actor("omargnagy/agent-fetch-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Markdown",
    "https://docs.python.org/3/tutorial/introduction.html",
    "https://www.propublica.org/article/3m-forever-chemicals-pfas-pfos-inside-story"
  ],
  "render": false,
  "respectRobotsTxt": true,
  "timeoutSecs": 30,
  "userAgent": "Mozilla/5.0 (compatible; AgentFetchBot/1.0; +https://apify.com/omargnagy/agent-fetch-markdown)",
  "includeLinks": true,
  "includeArticleJson": false,
  "maxMarkdownChars": 120000
}' |
apify call omargnagy/agent-fetch-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,omargnagy/agent-fetch-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e3VAta4eeA3gYI3TA/builds/k2BOhj5TCqp37OSBy/openapi.json