# URL to Markdown Reader - $0.25/1K Pages | Johto Data (`johto.data/llm-web-reader`) Actor

Read many URLs at once as Markdown cut to your token budget, with token counts and a content hash. Pass last hashes: unchanged pages are free. $0.25 per 1,000 pages; failed pages free.

- **URL**: https://apify.com/johto.data/llm-web-reader.md
- **Developed by:** [Jo](https://apify.com/johto.data) (community)
- **Categories:** AI, Agents, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Token-Budget Web Reader: Bulk URLs to Markdown, Pay per Change | Johto Data

### For AI agents

**Call this tool when you have one or many page URLs and need their content as Markdown that fits your context budget.** It is also for re-checking pages you already read, when you only want to pay if they changed. It does not search the web and does not render JavaScript. Find URLs with your own search tool first.

- **Read with a token budget:** `{"query": "https://docs.apify.com/platform/actors", "maxTokens": 2000}`. Each page is cut to about 2,000 tokens. `stats.tokensEstimateFull` tells you how big the whole page is, so you can decide whether to read more.
- **Read in bulk:** `{"urls": ["https://a.com/x", "https://b.com/y", "https://c.com/z"], "maxTokens": 1000}` reads the URLs in parallel in one call. One row comes back per URL, in input order (`searchResult.rank`).
- **Pay only for changes:** pass the hashes you got last time, as in `{"urls": ["https://a.com/pricing"], "knownHashes": {"https://a.com/pricing": "ecb64355722797f8"}}`. If the page's content is the same, you get `unchanged: true` with no Markdown, and that page is not charged.
- **Output per URL:**
  - `crawl.requestStatus` (`handled` | `failed`) and `crawl.httpStatusCode`
  - `unchanged`
  - `metadata.title`
  - `markdown`
  - `stats.tokensEstimate`, `stats.tokensEstimateFull`, `stats.truncated`
  - `stats.contentHash` (a hash of the full page content; store it for the next call)
- **Price:** $0.00025 per page returned with content, plus $0.00005 per run. Failed, unchanged and robots-blocked pages are free.

### Why agents use it

- **Fits the context window:** `maxTokens` caps every page, and the full-size estimate comes with it.
- **Cheap monitoring:** re-reading 100 watched pages where 5 changed costs 5 × $0.00025 + $0.00005 = $0.0013.
- **One call for many URLs:** there is one run-start fee per batch, not per page.
- **Clean content:** navigation, footers, cookie banners and scripts are removed, and the main content is extracted (trafilatura, with a full-body fallback). `.md` and `llms.txt` responses pass through untouched.

### Input example

```json
{
  "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation", "https://docs.apify.com/platform/actors"],
  "maxTokens": 1500,
  "knownHashes": {"https://docs.apify.com/platform/actors": "2dba0e54a6685ca0"}
}
```

### Output example

```json
[
  {"crawl": {"httpStatusCode": 200, "requestStatus": "handled", "loadedUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation", "loadedAt": "2026-10-22T08:42:49Z"},
   "searchResult": {"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation", "title": "Retrieval-augmented generation - Wikipedia", "resultType": "DIRECT_URL", "rank": 1},
   "metadata": {"title": "Retrieval-augmented generation - Wikipedia", "languageCode": "en"},
   "unchanged": false,
   "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique ...",
   "stats": {"tokensEstimate": 1498, "tokensEstimateFull": 6564, "truncated": true, "contentHash": "f3009901c423b24f", "tokenEstimateMethod": "chars/4"}},
  {"crawl": {"httpStatusCode": 200, "requestStatus": "handled"}, "searchResult": {"url": "https://docs.apify.com/platform/actors", "rank": 2},
   "unchanged": true, "markdown": "", "stats": {"contentHash": "2dba0e54a6685ca0", "tokensEstimateFull": 744}}
]
```

### Pricing

| Event | Price |
|---|---|
| Page returned with content (`fetch`) | $0.00025 ($0.25 per 1,000) |
| Run start (once per batch) | $0.00005 |
| Failed page, unchanged page, or page blocked by robots.txt | free |

### Use from MCP, LangChain, n8n

- **MCP:** `https://mcp.apify.com/?tools=johto.data/llm-web-reader`. Agents can pay with x402 or Skyfire through Apify.
- **LangChain:** `ApifyActorsTool("johto.data/llm-web-reader")`
- **API:** `POST https://api.apify.com/v2/acts/johto.data~llm-web-reader/run-sync-get-dataset-items`

### Compatibility note

The input fields `query`, `outputFormats`, `requestTimeoutSecs`, `removeElementsCssSelector`, `htmlTransformer` and `maxResults`, and the output blocks `crawl`, `searchResult`, `metadata` and `markdown`, use the same names as Apify's RAG Web Browser URL mode. Existing parsers keep working. For Google search or JavaScript rendering, use that Actor.

### Limits and good use

- **robots.txt is respected.** Each URL (and each redirect hop) is checked against the site's robots.txt for `johtodata-reader`. A disallowed URL returns a free row with `status: "robots_disallowed"` and is not fetched. A missing or unreachable robots.txt allows the fetch; a 5xx blocks it. You are responsible for the target site's terms.
- **Address safety.** http(s) only, ports 80, 443, 8080 and 8443. Every connection resolves the host once, refuses it unless every address is public (private, loopback, link-local and cloud metadata addresses are refused), then connects only to that checked address (DNS pinning). Every redirect hop is re-resolved and re-checked the same way. A refused URL returns a free failed row and is not fetched or charged. Bodies are capped at 6 MB; only text, HTML, XML and JSON pages are read.
- Raw HTTP only: no JavaScript rendering, logins or CAPTCHA solving. Token counts are estimates (characters / 4).
- It sends an honest user agent (`johtodata-reader/1.0`). Read only pages you may access, and follow each site's terms.

Made by [Johto Data](https://johtodata.com).

# Actor input Schema

## `query` (type: `string`):

The web page URL to read, e.g. `https://docs.apify.com/platform/actors`. Same field name as apify/rag-web-browser. Several URLs separated by spaces are fetched in parallel; commas are kept as part of the URL. Keyword search is NOT supported: search with your own tool, then pass the URLs here. Use when: you have one page URL (or a few, space-separated). Don't use when: you only have keywords (search first) or the page needs JavaScript to render.

## `urls` (type: `array`):

Optional list of page URLs to read in the same call (batch). Each item is one URL, used as given. Each successful page is one `fetch` event. Use when: you read a batch of URLs in one run (one start fee per batch).

## `maxTokens` (type: `integer`):

Return at most about this many tokens of Markdown per page (chars/4 estimate), e.g. `2000`. Overrides maxContentChars. `stats.tokensEstimateFull` still reports the full page size, so you can decide whether to read more. Use when: the page must fit a context window. Leave empty for the full page (up to maxContentChars).

## `knownHashes` (type: `object`):

Map of URL to the `stats.contentHash` you got last time, e.g. `{"https://example.com/pricing": "ecb64355722797f8"}`. If a page's content is the same, you get a row with `unchanged: true` and no Markdown, and that page is NOT charged. Use when: you re-check pages you read before and only want to pay for changes.

## `outputFormats` (type: `array`):

Formats to return for each page. `markdown` (default) is best for LLMs. Same values as apify/rag-web-browser.

## `maxContentChars` (type: `integer`):

Cut each page's content to this many characters to keep the payload small for your context window (about 5,000 tokens at the default). `stats.tokensEstimateFull` still reports the full size. 0 = no limit.

## `htmlTransformer` (type: `string`):

`readableText` (default) keeps the main article/body content and drops menus and boilerplate. `none` converts the whole page body after removing `removeElementsCssSelector`.

## `removeElementsCssSelector` (type: `string`):

CSS selector for elements to delete before conversion (used when htmlTransformer is `none`, and for the `text`/`html` formats).

## `includeLinks` (type: `boolean`):

Also return the page's absolute links (up to 500) in `links`, for agents that browse onward. Use when: your agent browses onward from this page.

## `requestTimeoutSecs` (type: `integer`):

Per-page timeout in seconds. Same name and default as apify/rag-web-browser.

## `maxRequestRetries` (type: `integer`):

Retries per page on network error or HTTP 5xx.

## `desiredConcurrency` (type: `integer`):

Pages fetched in parallel when several URLs are given.

## `maxUrls` (type: `integer`):

Safety cap on how many URLs one run reads. Use it to cap spend: each page with content costs $0.00025 ($0.25 per 1,000).

## `scrapingTool` (type: `string`):

Accepted for apify/rag-web-browser compatibility. This Actor always uses fast raw HTTP (no JavaScript rendering).

## `maxResults` (type: `integer`):

Accepted for apify/rag-web-browser compatibility; ignored because there is no search step.

## `proxyConfiguration` (type: `object`):

Optional. Off by default (direct, fastest). Use when: a site blocks direct requests. Leave off otherwise.

## `debugMode` (type: `boolean`):

Keep Open Graph and JSON-LD blocks in `metadata`.

## Actor input object example

```json
{
  "query": "https://docs.apify.com/platform/actors",
  "urls": [
    "https://example.com"
  ],
  "outputFormats": [
    "markdown"
  ],
  "maxContentChars": 20000,
  "htmlTransformer": "readableText",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, form, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], .cookie, #cookie, [class*=\"cookie-banner\"], [id*=\"cookie-banner\"]",
  "includeLinks": false,
  "requestTimeoutSecs": 40,
  "maxRequestRetries": 1,
  "desiredConcurrency": 5,
  "maxUrls": 100,
  "scrapingTool": "raw-http",
  "maxResults": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "debugMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "https://docs.apify.com/platform/actors"
};

// Run the Actor and wait for it to finish
const run = await client.actor("johto.data/llm-web-reader").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "https://docs.apify.com/platform/actors" }

# Run the Actor and wait for it to finish
run = client.actor("johto.data/llm-web-reader").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "https://docs.apify.com/platform/actors"
}' |
apify call johto.data/llm-web-reader --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,johto.data/llm-web-reader"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iANzoHVdTB04GcxWh/builds/uzfnwVPlGPdYUHLlK/openapi.json