# AgentReader: Clean Web Content for AI Agents (`f0rty7even/agent-reader`) Actor Turn any URL (HTML page or PDF) into clean, token-efficient Markdown with metadata, ready for an AI agent or LLM. Auto-detects PDFs and renders JavaScript-heavy pages when needed. - **URL**: https://apify.com/f0rty7even/agent-reader.md - **Developed by:** [Michael Yousrie](https://apify.com/f0rty7even) (community) - **Categories:** AI, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $4.00 / 1,000 read (fast path)s This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AgentReader: Clean Web Content for AI Agents Turn any URL into clean, token-efficient, cited content in a single call. AgentReader fetches a web page or a PDF, strips the navigation, ads, and boilerplate, and returns the main content as tidy Markdown plus structured metadata. It is built for AI agents and LLM pipelines, and it works as an MCP tool out of the box. ### Why AgentReader Most scrapers hand an agent a wall of raw HTML: menus, cookie banners, footers, and scripts that waste tokens and bury the answer. AgentReader is tuned for how agents actually consume the web: - **Token efficient.** Aggressive main-content extraction, plus an optional hard `maxTokens` budget so an agent can cap its own context cost. Every result reports its exact token count. - **Reliable.** A fast HTTP path first, with an automatic headless-browser fallback for JavaScript-heavy pages and pages that soft-block a plain request. The agent gets content instead of an empty response. - **One shape for HTML and PDF.** Pages and PDFs return the same structured result, so your calling code never has to branch on the source type. ### What it does Give AgentReader one or more URLs. For each one it returns: - `content`: clean Markdown (or plain text) of the main body - `title` and `metadata` (author, date, description, site name, language) - `tokenCount` and whether the content was `truncated` to your budget - `sourceType` (`html` or `pdf`) and `renderedWith` (`http` or `browser`) Failed reads come back with a clear `error` code (`blocked`, `timeout`, `not_extractable`, `too_large`, `not_found`) and are never billed. ### Inputs | Field | Type | Default | Description | |-------|------|---------|-------------| | `urls` | array of strings | required | The URLs to read. HTML pages and PDFs are both auto-detected. | | `maxTokens` | integer | `0` | Hard truncation budget per URL. `0` means no cap. | | `includeLinks` | boolean | `true` | Keep hyperlinks in the Markdown. | | `format` | string | `markdown` | `markdown` or `text`. | | `renderJs` | string | `auto` | `auto` tries a fast fetch and falls back to a browser when a page is empty or blocked. `never` disables the browser. `always` forces it. | ### Example Input: ```json { "urls": ["https://en.wikipedia.org/wiki/Model_Context_Protocol"], "maxTokens": 4000 } ```` Each dataset item contains the clean Markdown, the metadata, and the token count, ready to feed straight into an agent or a RAG index. ### Use as an MCP tool AgentReader is available as an MCP tool, so assistants and agent frameworks that speak the Model Context Protocol can call it directly to read the web on demand. Point your MCP client at the Apify MCP endpoint with your Apify token and the `read` tool appears automatically. ### Pricing AgentReader uses pay-per-event pricing. You are charged only for a successful read. Reads that need the headless-browser fallback are billed at a separate rate because they use more compute. Failed reads are free. # Actor input Schema ## `urls` (type: `array`): One or more URLs to read. HTML pages and PDFs are both supported and auto-detected. ## `maxTokens` (type: `integer`): Hard truncation budget per URL. 0 means no cap. Lets an agent bound its context cost. ## `includeLinks` (type: `boolean`): Preserve hyperlinks in the extracted Markdown. ## `format` (type: `string`): Content format returned per URL. ## `renderJs` (type: `string`): auto tries a fast HTTP fetch and falls back to a browser only when the result is empty or blocked. never disables the browser. always forces it. ## `proxyConfiguration` (type: `object`): Proxies to route requests through. Apify Proxy is recommended, as it reduces blocks on sites that reject datacenter or repeated requests. ## Actor input object example ```json { "urls": [ "https://docs.apify.com/platform" ], "maxTokens": 0, "includeLinks": true, "format": "markdown", "renderJs": "auto", "proxyConfiguration": { "useApifyProxy": true } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://docs.apify.com/platform" ] }; // Run the Actor and wait for it to finish const run = await client.actor("f0rty7even/agent-reader").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["https://docs.apify.com/platform"] } # Run the Actor and wait for it to finish run = client.actor("f0rty7even/agent-reader").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://docs.apify.com/platform" ] }' | apify call f0rty7even/agent-reader --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=f0rty7even/agent-reader", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AgentReader: Clean Web Content for AI Agents", "description": "Turn any URL (HTML page or PDF) into clean, token-efficient Markdown with metadata, ready for an AI agent or LLM. Auto-detects PDFs and renders JavaScript-heavy pages when needed.", "version": "0.1", "x-build-id": "rlmZsShau2n5E0bce" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/f0rty7even~agent-reader/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-f0rty7even-agent-reader", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/f0rty7even~agent-reader/runs": { "post": { "operationId": "runs-sync-f0rty7even-agent-reader", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/f0rty7even~agent-reader/run-sync": { "post": { "operationId": "run-sync-f0rty7even-agent-reader", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "URLs", "type": "array", "description": "One or more URLs to read. HTML pages and PDFs are both supported and auto-detected.", "items": { "type": "string" } }, "maxTokens": { "title": "Max tokens per URL", "minimum": 0, "maximum": 200000, "type": "integer", "description": "Hard truncation budget per URL. 0 means no cap. Lets an agent bound its context cost.", "default": 0 }, "includeLinks": { "title": "Keep links", "type": "boolean", "description": "Preserve hyperlinks in the extracted Markdown.", "default": true }, "format": { "title": "Output format", "enum": [ "markdown", "text" ], "type": "string", "description": "Content format returned per URL.", "default": "markdown" }, "renderJs": { "title": "JavaScript rendering", "enum": [ "auto", "never", "always" ], "type": "string", "description": "auto tries a fast HTTP fetch and falls back to a browser only when the result is empty or blocked. never disables the browser. always forces it.", "default": "auto" }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxies to route requests through. Apify Proxy is recommended, as it reduces blocks on sites that reject datacenter or repeated requests.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```