AgentReader: Clean Web Content for AI Agents

Turn any URL into clean, token-efficient, cited content in a single call. AgentReader fetches a web page or a PDF, strips the navigation, ads, and boilerplate, and returns the main content as tidy Markdown plus structured metadata. It is built for AI agents and LLM pipelines, and it works as an MCP tool out of the box.

Why AgentReader

Most scrapers hand an agent a wall of raw HTML: menus, cookie banners, footers, and scripts that waste tokens and bury the answer. AgentReader is tuned for how agents actually consume the web:

• Token efficient. Aggressive main-content extraction, plus an optional hard maxTokens budget so an agent can cap its own context cost. Every result reports its exact token count.
• Reliable. A fast HTTP path first, with an automatic headless-browser fallback for JavaScript-heavy pages and pages that soft-block a plain request. The agent gets content instead of an empty response.
• One shape for HTML and PDF. Pages and PDFs return the same structured result, so your calling code never has to branch on the source type.

What it does

Give AgentReader one or more URLs. For each one it returns:

• content: clean Markdown (or plain text) of the main body
• title and metadata (author, date, description, site name, language)
• tokenCount and whether the content was truncated to your budget
• sourceType (html or pdf) and renderedWith (http or browser)

Failed reads come back with a clear error code (blocked, timeout, not_extractable, too_large, not_found) and are never billed.

Inputs

Field	Type	Default	Description
urls	array of strings	required	The URLs to read. HTML pages and PDFs are both auto-detected.
maxTokens	integer	0	Hard truncation budget per URL. 0 means no cap.
includeLinks	boolean	true	Keep hyperlinks in the Markdown.
format	string	markdown	markdown or text.
renderJs	string	auto	auto tries a fast fetch and falls back to a browser when a page is empty or blocked. never disables the browser. always forces it.

Example

Input:

{
  "urls": ["https://en.wikipedia.org/wiki/Model_Context_Protocol"],
  "maxTokens": 4000
}

Each dataset item contains the clean Markdown, the metadata, and the token count, ready to feed straight into an agent or a RAG index.

Use as an MCP tool

AgentReader is available as an MCP tool, so assistants and agent frameworks that speak the Model Context Protocol can call it directly to read the web on demand. Point your MCP client at the Apify MCP endpoint with your Apify token and the read tool appears automatically.

Pricing

AgentReader uses pay-per-event pricing. You are charged only for a successful read. Reads that need the headless-browser fallback are billed at a separate rate because they use more compute. Failed reads are free.