Token-Budget Web Reader: Bulk URLs to Markdown, Pay per Change | Johto Data

For AI agents

Call this tool when you have one or many page URLs and need their content as Markdown that fits your context budget. It is also for re-checking pages you already read, when you only want to pay if they changed. It does not search the web and does not render JavaScript. Find URLs with your own search tool first.

• Read with a token budget: {"query": "https://docs.apify.com/platform/actors", "maxTokens": 2000}. Each page is cut to about 2,000 tokens. stats.tokensEstimateFull tells you how big the whole page is, so you can decide whether to read more.
• Read in bulk: {"urls": ["https://a.com/x", "https://b.com/y", "https://c.com/z"], "maxTokens": 1000} reads the URLs in parallel in one call. One row comes back per URL, in input order (searchResult.rank).
• Pay only for changes: pass the hashes you got last time, as in {"urls": ["https://a.com/pricing"], "knownHashes": {"https://a.com/pricing": "ecb64355722797f8"}}. If the page's content is the same, you get unchanged: true with no Markdown, and that page is not charged.
• Output per URL:
  • crawl.requestStatus (handled | failed) and crawl.httpStatusCode
  • unchanged
  • metadata.title
  • markdown
  • stats.tokensEstimate, stats.tokensEstimateFull, stats.truncated
  • stats.contentHash (a hash of the full page content; store it for the next call)
• Price: $0.00025 per page returned with content, plus $0.00005 per run. Failed, unchanged and robots-blocked pages are free.

Why agents use it

• Fits the context window: maxTokens caps every page, and the full-size estimate comes with it.
• Cheap monitoring: re-reading 100 watched pages where 5 changed costs 5 × $0.00025 + $0.00005 = $0.0013.
• One call for many URLs: there is one run-start fee per batch, not per page.
• Clean content: navigation, footers, cookie banners and scripts are removed, and the main content is extracted (trafilatura, with a full-body fallback). .md and llms.txt responses pass through untouched.

Input example

{
  "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation", "https://docs.apify.com/platform/actors"],
  "maxTokens": 1500,
  "knownHashes": {"https://docs.apify.com/platform/actors": "2dba0e54a6685ca0"}
}

Output example

[
  {"crawl": {"httpStatusCode": 200, "requestStatus": "handled", "loadedUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation", "loadedAt": "2026-10-22T08:42:49Z"},
   "searchResult": {"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation", "title": "Retrieval-augmented generation - Wikipedia", "resultType": "DIRECT_URL", "rank": 1},
   "metadata": {"title": "Retrieval-augmented generation - Wikipedia", "languageCode": "en"},
   "unchanged": false,
   "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique ...",
   "stats": {"tokensEstimate": 1498, "tokensEstimateFull": 6564, "truncated": true, "contentHash": "f3009901c423b24f", "tokenEstimateMethod": "chars/4"}},
  {"crawl": {"httpStatusCode": 200, "requestStatus": "handled"}, "searchResult": {"url": "https://docs.apify.com/platform/actors", "rank": 2},
   "unchanged": true, "markdown": "", "stats": {"contentHash": "2dba0e54a6685ca0", "tokensEstimateFull": 744}}
]

Pricing

Event	Price
Page returned with content (fetch)	$0.00025 ($0.25 per 1,000)
Run start (once per batch)	$0.00005
Failed page, unchanged page, or page blocked by robots.txt	free

Use from MCP, LangChain, n8n

• MCP: https://mcp.apify.com/?tools=johto.data/llm-web-reader. Agents can pay with x402 or Skyfire through Apify.
• LangChain: ApifyActorsTool("johto.data/llm-web-reader")
• API: POST https://api.apify.com/v2/acts/johto.data~llm-web-reader/run-sync-get-dataset-items

Compatibility note

The input fields query, outputFormats, requestTimeoutSecs, removeElementsCssSelector, htmlTransformer and maxResults, and the output blocks crawl, searchResult, metadata and markdown, use the same names as Apify's RAG Web Browser URL mode. Existing parsers keep working. For Google search or JavaScript rendering, use that Actor.

Limits and good use

• robots.txt is respected. Each URL (and each redirect hop) is checked against the site's robots.txt for johtodata-reader. A disallowed URL returns a free row with status: "robots_disallowed" and is not fetched. A missing or unreachable robots.txt allows the fetch; a 5xx blocks it. You are responsible for the target site's terms.
• Raw HTTP only: no JavaScript rendering, logins or CAPTCHA solving. Token counts are estimates (characters / 4).
• It sends an honest user agent (johtodata-reader/1.0). Read only pages you may access, and follow each site's terms.

Made by Johto Data.