Agent Web Reader: Search + URL to Markdown for LLMs

Agent Web Reader: web search + URL to clean markdown for AI agents

Why agents choose this

Tested Oct 25, 2026 on 192 live URLs: 0.92 s median per page vs 3.2 s for Jina Reader, 1,474 median tokens per page vs 5,163, and the best main-text score (F1 0.61 vs 0.52). Search: the top result was on-topic for 85% of 20 queries vs 75% for rag-web-browser. We read 80% of URLs, Jina 91%: a headless-browser fallback now rescues some JavaScript pages, but JS-only apps are still our weak spot (10/30 vs 26/30) because many block cloud IPs.

Test (Oct 25, 2026, same URLs)	This Actor (Apify cloud)	Jina Reader (free)	apify/rag-web-browser	apify/website-content-crawler
Pages read, 192 live URLs	80.2% (was 78.1% before the browser fallback)	90.6%	not run	not run
Pages read, 49-URL subset	83.7%	87.8%	75.5%	77.6%
PDFs read (6 in subset)	5/6	5/6	0/6	0/6
JavaScript apps (30 in full set)	10/30 (was 9/30)	26/30	n/a	n/a
Main-text F1 vs Wikipedia article text (49 pages)	0.61	0.52	0.46 (13-page subset)	0.60 (13-page subset)
Median latency per page	0.92 s (p95 16 s when the browser fallback kicks in)	3.2 s	1.2 s	batch run, 15.7 min for 50 URLs
Median tokens per page	1,474	5,163	4,755 (subset)	2,421 (subset)
Search + read top result, median (20 queries, one at a time)	2.9 s (before Google ranking; see benchmark)	n/a	8.0 s	n/a
Top result matches the query words (20 queries)	85% (was 40%; Google ranking)	n/a	75%	n/a
Dead links (404)	not_found row, not charged	5 of 8 returned the 404 page as success

Full method and scripts: benchmark of 192 URLs (news, docs, JavaScript apps, paywalls, PDFs, Wikipedia, misc), with 8 dead links excluded for every engine.

• Headless-browser fallback: pages that come back empty, blocked or under 200 characters are re-rendered in Chrome (renderJs: auto/always/never, renderBudgetSecs caps browser time per call). Needs 1 GB+ memory (the default). Only pages that end up with content are charged.
• $0.003 per search, $0.002 per page read. 404s, timeouts, blocks and empty pages come back as rows with errorCode and are never charged.
• Every row says what happened: outcome is ok, no_content, blocked, not_found, timeout, ... plus retryable true/false. Search with no hits (no_results) is told apart from a blocked engine (search_blocked).
• Exact token counts: each ok row has tokens (cl100k), and optional chunkSize returns chunks with charStart/charEnd, headingPath and tokens.
• One flat row per page with url, title, markdown, status. queryMatch (0-1) flags off-topic search hits.

Cost per call

Event	Price	When
search	$0.003	once per query that returned results
page	$0.002	per page or PDF read successfully
actor-start	$0.001	once per batch run (not charged in Standby/HTTP mode)

Typical call: query + 3 results = $0.001 + $0.003 + 3 x $0.002 = $0.010. Reading one URL = $0.003. Errors (404, timeout, blocked, empty, scanned PDF) are never charged. Set maxTotalChargeUsd on the run to cap spend; when the cap is reached remaining URLs return SPENDING_LIMIT rows instead of being fetched.

Input (only query or urls is required)

{ "query": "what is the model context protocol", "maxResults": 3, "maxCharsPerResult": 4000 }

{ "urls": ["https://arxiv.org/pdf/1706.03762", "https://en.wikipedia.org/wiki/Model_Context_Protocol"], "outputMode": "summary" }

Field	Default	Notes
query	–	Web search; top maxResults results are opened.
urls	–	Up to 50 URLs (HTML or PDF). Combine with query if you like.
maxResults	3	1-10 search results to read.
maxCharsPerResult	6000	Token budget per result (~4 chars per token). Cut on a paragraph boundary.
outputMode	markdown	summary = lead paragraphs + section outline, max 1,500 chars.
includeLinks	false	Keep [text](url) links.
fullPage	false	Whole page body instead of main-article extraction (list/index pages).
country	–	2-letter code for localized search.
maxPdfPages	50	Pages read per PDF.
timeoutSecs	15	Per-URL timeout; slow URLs become error rows.

Output (one flat row per page; same fields every time)

{
  "status": "ok",
  "url": "https://en.wikipedia.org/wiki/Model_Context_Protocol",
  "finalUrl": "https://en.wikipedia.org/wiki/Model_Context_Protocol",
  "title": "Model Context Protocol",
  "markdown": "The Model Context Protocol (MCP) is an open standard ...",
  "chars": 4000, "totalChars": 16749, "truncated": true,
  "source": "search", "searchRank": 1, "query": "what is the model context protocol",
  "contentType": "text/html", "fetchedAt": "2026-10-25T05:01:12.000Z", "ms": 384,
  "errorCode": null, "errorMessage": null
}

{ "status": "error", "url": "https://example.com/missing", "markdown": null, "errorCode": "HTTP_404", "errorMessage": "The server answered 404." }

Error codes: INVALID_URL, DNS_NOT_FOUND, TIMEOUT, HTTP_403/HTTP_429 (site blocked bots), HTTP_404, NO_TEXT_LAYER (scanned PDF), EMPTY_CONTENT, UNSUPPORTED_TYPE, TOO_LARGE, PARSE_FAILED, SEARCH_FAILED, SPENDING_LIMIT. One bad URL never fails the run. The OUTPUT key-value record holds all rows plus searchEngine, okCount, errorCount and chargedEvents.

Fast calls: Standby (HTTP) mode

The Actor runs in Apify Standby mode, so it answers like an API with no cold start per call:

curl "https://catalyst-labs--agent-web-reader.apify.actor/?query=apify%20mcp&maxResults=2&token=$APIFY_TOKEN"
curl -X POST "https://catalyst-labs--agent-web-reader.apify.actor/?token=$APIFY_TOKEN" -H 'content-type: application/json' -d '{"urls":["https://example.com"]}'

Response: { "status": "ok", "query": ..., "searchEngine": ..., "results": [ ...rows... ] }.

Use from MCP (Claude, Cursor, ChatGPT, any MCP client)

https://mcp.apify.com/?tools=catalyst-labs/agent-web-reader (Apify token as Bearer auth). The tool appears as catalyst-labs--agent-web-reader.

How it works

Search: Bing, with DuckDuckGo and Google (Apify SERP proxy) fallbacks; searchEngine tells you which answered. Pages are fetched directly with a datacenter-proxy retry when a site blocks; main content is extracted (Mozilla Readability) and converted to GitHub-flavored markdown (tables kept); scripts, nav, footers and images are dropped (image alt text kept). PDFs are converted page by page with ## Page N headings.

Same input gives the same rows in the same order (search ranks can change over time as the web does). Respect the sites you read; this tool does not bypass logins or paywalls.

Made by Catalyst Labs. Also see: YouTube Transcript API, Document to Markdown, Google Trends API, Website Screenshot & PDF API.