Web Fetch for AI Agents and MCP, URL to Markdown

Give your agent a web fetch tool that returns clean Markdown for $0.001 a page, and costs nothing when a page fails. It tries a cheap HTTP request first and opens a headless browser only when a page is blocked, needs JavaScript or comes back empty.

Plug it into Claude, Cursor, VS Code or any MCP client through the Apify MCP server, or call it from the API with up to 100 URLs per run. Every row says how the page was fetched, and every failed row says why, so an agent can tell a dead link from a bot wall and move on.

Add it to your agent through MCP

The Apify MCP server at https://mcp.apify.com turns this Actor into a tool. Pin it with the tools parameter, and the agent sees the Actor's input fields as the tool's arguments.

{
  "mcpServers": {
    "web-fetch": {
      "url": "https://mcp.apify.com?tools=zovo-one/web-fetch-fallback"
    }
  }
}

Your browser opens on the first connection so you can sign in to Apify (OAuth). If your client can't do that, send your Apify token in a header instead.

{
  "mcpServers": {
    "web-fetch": {
      "url": "https://mcp.apify.com?tools=zovo-one/web-fetch-fallback",
      "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }
    }
  }
}

In Claude Code it's one command.

claude mcp add --transport http web-fetch "https://mcp.apify.com?tools=zovo-one/web-fetch-fallback" \
  --header "Authorization: Bearer YOUR_APIFY_TOKEN"

You can list more tools in the same parameter, comma-separated, for example ?tools=actors,zovo-one/web-fetch-fallback to keep Apify's Actor search next to it. The Actor runs with limited permissions, which the MCP server requires.

A few settings help agents. maxCharacters caps how much text lands in the context window (the default is 200,000 characters). fetchMode: "http" is the fastest choice for docs, blogs and news pages that don't need JavaScript. And outputFormat: "text" drops link markup when the agent only needs the words.

What you get

One dataset row per URL.

Field	Example	Notes
url	https://en.wikipedia.org/wiki/Markdown	The URL from the input, normalised. A line that isn't a usable URL keeps the raw input, cut at 300 characters
status	success	success or failed. Only success rows are charged
content	**Markdown**[^10] is a [lightweight markup language](...)	The page as Markdown, plain text or cleaned HTML. null on failure
title	Markdown	Page title
description	null	Meta description, when the page has one
author	Wikipedia	When the page states one
publishedAt	2005-08-09T19:56:00Z	Publication date as the page states it
language	en	Page language
fetchedWith	http	The step that produced the content: http, http-browser-headers or browser
attempts	["http: ok"]	Every step tried and how it ended
extraction	main-content	How the content was picked (see below)
format	markdown	Format of content
characters	33329	Length of content
wordCount	2640	Words in the whole extracted text, before any cut
truncated	false	true when content was cut at maxCharacters
httpStatus	200	Status of the last answer
finalUrl	https://en.wikipedia.org/wiki/Markdown	URL after redirects
contentType	text/html; charset=UTF-8	Content-Type of the page
errorCode, error	null	Why the URL failed, as a code and as a sentence
elapsedMs, fetchedAt	1779, 2026-11-05T14:57:30.107Z	Timing. Not present on INVALID_URL and BLOCKED_URL rows for unusable input lines, or on INTERNAL_ERROR rows

Links and image paths in the content are absolute.

extraction	Meaning
main-content	Defuddle found the main block of the page.
full-page	extractMainContent was off, or the main block held almost nothing, so the whole body was used.
large-page	The page has more than 3,000 HTML elements, or Defuddle ran past the conversion time limit. The <main> or <article> element (or the body) was converted directly, without Defuddle's clutter scoring. It's less precise but several times faster on big pages.
raw-text	The URL returned plain text, Markdown, JSON, XML or CSV, which is passed through as it is.

The key-value store record OUTPUT holds the run summary. It has counts per step and per error code, the number of charged pages, and the first 100 URLs skipped at the maximum charge or the run timeout.

Example output

These two rows come from a run on Apify's cloud on 2026-11-05 with the default input. Only the Wikipedia content is shortened here.

[
  {
    "url": "https://en.wikipedia.org/wiki/Markdown",
    "status": "success",
    "errorCode": null,
    "error": null,
    "httpStatus": 200,
    "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
    "fetchedWith": "http",
    "attempts": ["http: ok"],
    "contentType": "text/html; charset=UTF-8",
    "format": "markdown",
    "extraction": "main-content",
    "title": "Markdown",
    "description": null,
    "author": "Wikipedia",
    "publishedAt": "2005-08-09T19:56:00Z",
    "language": "en",
    "content": "**Markdown**[^10] is a [lightweight markup language](https://en.wikipedia.org/wiki/Lightweight_markup_language \"Lightweight markup language\") for creating [formatted text](https://en.wikipedia.org/wiki/Formatted_text \"Formatted text\") using a ...",
    "characters": 33329,
    "wordCount": 2640,
    "truncated": false,
    "elapsedMs": 1779,
    "fetchedAt": "2026-11-05T14:57:30.107Z"
  },
  {
    "url": "https://example.com/",
    "status": "success",
    "errorCode": null,
    "error": null,
    "httpStatus": 200,
    "finalUrl": "https://example.com/",
    "fetchedWith": "http",
    "attempts": ["http: ok"],
    "contentType": "text/html; charset=utf-8",
    "format": "markdown",
    "extraction": "main-content",
    "title": "Example Domain",
    "description": null,
    "author": null,
    "publishedAt": null,
    "language": "en",
    "content": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
    "characters": 156,
    "wordCount": 25,
    "truncated": false,
    "elapsedMs": 1829,
    "fetchedAt": "2026-11-05T14:57:30.161Z"
  }
]

A page that needed all three steps shows them in order, for example ["http: HTTP 403 cloudflare", "http-browser-headers: HTTP 403 cloudflare", "browser: ok"], with fetchedWith: "browser".

Pricing

It's pay per event. You pay only for rows with status: "success".

Event	Price	When it's charged
page	$0.001	Once per URL that came back with content. Truncated pages count.
Failed URL	$0	Refused URLs, timeouts, HTTP errors, blocked pages, empty pages and unsupported files, whichever step they reached.

A worked example. An agent that reads 20 pages per task and runs 50 tasks a day fetches 1,000 pages. If 950 come back with content and 50 fail, the day costs 950 × $0.001 = $0.95. A page that needed the browser costs the same $0.001 as one fetched over plain HTTP.

Pages with content	Cost
100	$0.10
1,000	$1.00
10,000	$10.00

If you set a maximum charge for a run, the Actor stops starting new URLs once it's reached. The URLs it didn't start aren't charged and are listed in OUTPUT under skippedUrls.

How the fallback chain works

Each URL goes through up to three steps. A step only runs when the one before it ended in a way the next one can fix.

Step	What it does	When the next step runs
http	A plain GET with an honest user agent. Follows up to 10 redirects and keeps cookies across them.	HTTP 403, 406, 429, 500, 502, 503, 504 or 520 to 524, a bot-check page, or a reset connection
http-browser-headers	The same request with Chrome-like headers (user agent, Accept-Language, Sec-Fetch-*, client hints).	HTTP 403, 406, 429 or 503, a bot-check page, or a reset connection
browser	Headless Chromium loads the page, waits for the network to settle, then reads the rendered HTML.	Last step

Some answers skip straight to the browser. A page that's only an empty app shell (a <div id="root"> plus scripts, or a "please enable JavaScript" notice), an empty page, a TLS error or a redirect loop goes to browser without the header retry. A 404, a DNS failure, a timeout, a PDF or a refused address stops at once, because a browser wouldn't change the answer.

Input

Field	Type	Default	What it does
urls	array of strings	two example URLs	Pages to fetch, 1 to 100. example.com becomes https://example.com/. Exact duplicates are fetched once.
outputFormat	markdown, text, html	markdown	Format of content. html is the cleaned HTML of the extracted part, not the raw page source.
extractMainContent	boolean	true	Keep the article or main block and drop menus, headers, footers and sidebars. Off converts the whole body.
fetchMode	auto, http, browser	auto	http never starts a browser. browser always uses it.
timeoutSecs	integer, 5 to 120	30	Limit for each step. A URL that needs all three steps can take three times this.
maxCharacters	integer, 500 to 1,000,000	200000	content is cut here and the row gets truncated: true.
maxConcurrency	integer, 1 to 10	5	URLs fetched in parallel. Browser pages run 1 at a time below 2 GB of memory and 2 at a time from 2 GB.

A run with an empty input fetches the two example URLs, so you can try it with one click.

{
  "urls": ["https://en.wikipedia.org/wiki/Markdown", "news.ycombinator.com"],
  "outputFormat": "markdown",
  "fetchMode": "auto"
}

Call it from the API

The synchronous endpoint returns the dataset rows in one HTTP call. Put your token in APIFY_TOKEN.

curl -s -X POST \
  "https://api.apify.com/v2/acts/zovo-one~web-fetch-fallback/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/"], "outputFormat": "markdown"}'

The synchronous call waits at most 300 seconds for the run. If the run takes longer, the call ends without the rows, but the run keeps going. A few URLs over HTTP finish well within that. For long lists, or pages that need the browser, start a normal run and read the dataset when it ends.

Python

$pip install apify-client

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("zovo-one/web-fetch-fallback").call(run_input={
    "urls": ["https://en.wikipedia.org/wiki/Markdown", "https://example.com/"],
    "outputFormat": "markdown",
    "maxCharacters": 50000,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["status"] == "success":
        print(row["url"], row["wordCount"], row["content"][:200])
    else:
        print(row["url"], row["errorCode"], row["error"])

Node.js

$npm install apify-client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('zovo-one/web-fetch-fallback').call({
  urls: ['https://en.wikipedia.org/wiki/Markdown', 'https://example.com/'],
  outputFormat: 'markdown',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const row of items) {
  console.log(row.url, row.status, row.fetchedWith, row.errorCode);
}

Error codes

A failed row has status: "failed", content: null, an errorCode and a readable error. Failed rows aren't charged.

errorCode	Meaning
INVALID_URL	The line isn't a usable http or https URL (other schemes, spaces, embedded user and password, longer than 2,048 characters).
BLOCKED_URL	The URL or a redirect points at a private, local or reserved address, or a redirect goes to another scheme (such as ftp:) or to a URL with a user and password.
DNS_FAILED	The hostname doesn't resolve.
CONNECTION_FAILED	The server refused or reset the connection, or sent a corrupt compressed body.
TLS_ERROR	The certificate is invalid or the TLS handshake failed.
TIMEOUT	No complete answer within timeoutSecs, or converting the page ran past its time limit (timeoutSecs, but at least 15 and at most 60 seconds).
TOO_MANY_REDIRECTS	More than 10 redirects.
HTTP_ERROR	An error status such as 404 or 500, or a redirect status without a Location header. See httpStatus.
BLOCKED_BY_SITE	The last step it tried got a 403, 406, 429 or 503, or a bot-check page (Cloudflare, DataDome, PerimeterX, Imperva, Akamai and others). The code comes from the last step, so 403, 403 and then a browser timeout ends as TIMEOUT. attempts shows every step.
JS_REQUIRED	The page needs JavaScript and fetchMode is http.
UNSUPPORTED_CONTENT_TYPE	A PDF, image, archive or other non-text file, a body that turns out to be binary data, or a compression other than gzip, deflate or Brotli.
CONTENT_TOO_LARGE	The page is larger than 5 MB after decompression, or it's too large to convert within the run's memory. At 1 GB of memory, HTML pages of a few MB can hit this. Give the run more memory to convert them.
EMPTY_CONTENT	The page loaded but has no readable text, or its HTML is nested too deeply to parse.
BROWSER_UNAVAILABLE	The page needed the browser and Chromium couldn't start.
BROWSER_ERROR	Chromium failed while loading the page.
INTERNAL_ERROR	A bug on our side. Please report it with the URL.

Invalid input fails the run (for example more than 100 URLs or an unknown outputFormat). Apart from that, a run where every URL fails still succeeds, with one failed row per URL, so an agent always gets a reason per URL. Pages are converted in a separate thread with its own memory cap, so a page that's too big to convert fails its own row, not the run.

Limits

• 100 URLs per run, 5 MB per page (after decompression), 10 redirects per step.
• Requests go out directly from Apify's servers, with no proxy. Sites that block datacenter addresses end as BLOCKED_BY_SITE, and those rows are free.
• It doesn't log in, fill forms, solve captchas or scroll for lazy content. The browser waits up to 5 seconds for the network to go quiet and up to 8 seconds for a bot check to clear itself, within timeoutSecs.
• It doesn't read PDFs. They fail with UNSUPPORTED_CONTENT_TYPE and cost nothing.
• It doesn't read robots.txt. You're responsible for respecting each site's terms and for what you do with the content.

HTTPS certificates are verified, so a site with an expired or self-signed certificate fails with TLS_ERROR. HTML is converted in one background thread (two from 2 GB of memory), which gets about 35% of the run's memory. Pages wait their turn for it, so on a run with many big pages a small page can take longer than its download time.

Security

The Actor refuses anything that isn't a public web address. That covers localhost, .local and .internal names, cloud metadata hosts, private and reserved IPv4 and IPv6 ranges (including IPv4-mapped, NAT64, 6to4 and Teredo forms), and URLs with embedded credentials. That matters when an agent picks the URLs, because a prompt injection can't point it at internal or metadata addresses.

In the HTTP steps the check runs before each request and each redirect hop, and again inside the socket's own DNS lookup, so a hostname that switches to a private address between the two lookups is still refused. In the browser step every request the page makes is checked, including subresources, redirects and WebSockets. Service workers are blocked and WebRTC is removed. One gap remains there. Chromium resolves hostnames itself, so a hostname with a zero TTL that answers our check with a public address and Chromium a moment later with a private one isn't caught. The browser only ever returns page text, never response headers or files.

Extraction runs on jsdom with scripts off and no subresource loading. Defuddle's site-specific API calls are switched off, so extraction never makes a network request.

FAQ

Does a failed page cost anything?

No. Only rows with status: "success" are charged. A 404, a timeout, a bot wall or a PDF costs $0, and the row tells you which one it was.

Will it get past Cloudflare and other bot checks?

Sometimes. The header retry and the headless browser get through some checks, and the browser waits up to 8 seconds for a check to clear itself. It doesn't use proxies or solve captchas, so a site that blocks datacenter addresses ends as BLOCKED_BY_SITE, free of charge.

How do I keep pages small enough for my model's context?

Set maxCharacters. The content is cut there and the row gets truncated: true, while wordCount still counts the words of the whole extracted text, before the cut. With extractMainContent on, that's the main block, not the whole page. Leaving extractMainContent on also drops menus, footers and sidebars before the cut.

Which fetch mode should my agent use?

Leave it on auto unless you know the sites. http is faster and never starts a browser, but JavaScript-only pages fail with JS_REQUIRED. browser is for sites you know need it every time.

Why do the rows come back in a different order than my URLs?

URLs are fetched in parallel, so rows land as they finish. Match them on url.

Can I run it on a schedule?

Yes. Use an Apify schedule or task with a fixed URL list. Each run is priced the same way.

Open-source parts

Main content detection is Defuddle (MIT). The DOM is jsdom (MIT), Markdown comes from turndown (MIT), and HTTP from undici (MIT). Mozilla Readability isn't used. This Actor's own code is MIT licensed.

Support Zovo

Zovo builds and maintains this Actor. If it saves you time, you can support the work on GitHub Sponsors. Sponsorship is optional, and the tool works the same without it.

Zovo is made by theluckystrike on GitHub.

More tools by Zovo

• Website Screenshot API. Saves full-page screenshots and PDFs of public web pages.
• YouTube Transcript Scraper. Gets YouTube transcripts and captions from videos, playlists and channels.
• Google News Scraper. Gets Google News results with the real publisher URLs.
• Substack and Medium Scraper. Gets Medium and Substack articles by author, publication, tag or search.