Web Fetch API: Web Unblocker & URL to Markdown

Web Fetch API: Web Unblocker and URL to Markdown for AI Agents

Web Fetch API is a web unblocker API for AI agents and LLM apps. Send one URL and get the page back in the same HTTP response as LLM-ready Markdown, HTML, text, links or the raw file, with Cloudflare, DataDome, CAPTCHAs and other bot protection handled on our side. It costs $0.40 per 1,000 successful pages ($0.80 with JavaScript rendering), and failed requests are never charged.

Agents usually need two tools to read the web: an unblocker to get the page, and a converter to turn it into text a model can read. Web Fetch API does both in one call, so it works as a complete web fetch API for AI agents, with anti-bot handling built in. Use it as the "read this URL" tool for an agent built on Claude, GPT, Gemini or an open model, or from any backend that needs a page right now. It runs as an always-on Apify Standby Actor, so there's no run to start, no dataset to poll, and no proxies or browsers to manage. Results aren't stored anywhere.

Endpoint	POST https://epctex--web-fetch-api.apify.actor/api/fetch
Price	$0.0004 per successful request; $0.0008 with js_render: true
Bot protection	Cloudflare, DataDome, Akamai, PerimeterX, Imperva and AWS WAF, plus CAPTCHA challenges, handled on our side; no proxies or browsers to set up
Formats	markdown, html, text, links, raw (PDFs default to Markdown)
Extras	Page metadata: title, description, Open Graph, JSON-LD, links, emails, phone numbers
Agent-ready	unwrap: true returns the Markdown itself as the response body, ready to pass back as a tool result
Charged	Only 200 responses. Failed fetches, errors and health checks are free
URL safety	Localhost, private IPs and cloud metadata hosts are refused, including through redirects
Limits	One URL per request; fetched body up to 8 MiB; JavaScript rendering up to 120 seconds
Data stored	None

Need to fetch a list of URLs into a dataset, or call the fetcher from Apify's MCP server? Use Web Fetch, the batch version of the same fetcher at the same price.

---

Quick start

You need an Apify account and API token; Web Fetch API needs nothing else. Three steps:

1. Get your Apify API token from Console, Settings, API and Integrations.

2. Send a request:

curl -X POST "https://epctex--web-fetch-api.apify.actor/api/fetch" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

3. Read the page from the response:

{
  "status": 200,
  "message": "OK",
  "data": {
    "url": "https://example.com",
    "fetch": {
      "loadedUrl": "https://example.com/",
      "loadedTime": "2026-10-26T12:00:00.000Z",
      "httpStatusCode": 200,
      "contentLengthBytes": 1256,
      "contentType": "text/html; charset=UTF-8",
      "encoding": "utf8"
    },
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents..."
  }
}

That request cost $0.0004. Your token is never forwarded to the fetcher.

---

When to use Web Fetch API

Use Web Fetch API when an app or an AI agent has one URL and needs that page's content back immediately, as Markdown or another format a model can read, including pages that block plain scripted requests. Each call fetches one page and answers in the same HTTP response.

Use it for:

• Giving an AI agent a "read this URL" tool that returns clean Markdown instead of raw HTML, in Claude Code, Cursor, LangChain, LlamaIndex, the Vercel AI SDK or any framework with function calling.
• Fetching pages on demand inside RAG pipelines, chatbots and research agents.
• Reading pages that return 403, "access denied" or a challenge page to a plain HTTP client, without running your own proxies or browsers.
• Converting PDFs to Markdown for an LLM.
• Pulling page metadata (title, description, Open Graph, JSON-LD, emails, phone numbers) for SEO checks or enrichment.
• Collecting every link on a page to plan the next fetch.

Use something else for:

• Lists of URLs processed in one job: use Web Fetch, which writes each page to a dataset.
• Whole-site crawls: Web Fetch API fetches exactly the URL you send and doesn't follow links. Use the links format to collect URLs, then fetch the ones you need.
• Pages behind a login or paywall: Web Fetch API has no login, cookie or header options, and URLs containing a username or password are rejected.
• Search: Web Fetch API needs a URL; it doesn't search the web for one.
• Agents connected to Apify's MCP server: call Web Fetch with a one-URL list. Web Fetch API itself is called over plain HTTP, as shown below.

---

Why use Web Fetch API as your agent's fetch tool?

• Web unblocker and converter in one call. Bot protection is handled on our side, and the page comes back ready for a model. No separate proxy service, scraping API and HTML-to-Markdown step to wire together.
• Markdown by default. HTML pages and PDFs come back as Markdown, which keeps headings, lists and links but drops the markup, so a page costs fewer tokens in the context window.
• Live answers. One request, one page, in the same HTTP response. No job queue, no polling, no webhooks.
• JavaScript when you need it. Plain fetching is the cheaper default; set js_render: true per request for single-page apps and late-loading content.
• Flat price per page. $0.40 per 1,000 pages, or $0.80 with JavaScript rendering. There are no proxy fees and no separate unblocking fee.
• Pay only for successful requests. Rejected URLs, failed fetches and unwrap requests that can't produce a format cost nothing, so an agent's retries and bad guesses don't add up.
• Safe to point at model-chosen URLs. Requests to localhost, private networks and cloud metadata endpoints are refused, even when a public page redirects to them.
• Drop-in tool definition. A ready-made function-calling schema and a paste-in prompt for coding agents are below.

---

Contents

1. How does Web Fetch API handle bot protection?
2. How much does Web Fetch API cost?
3. Set it up with an AI coding agent
4. Use it as a tool in an AI agent
5. Endpoints
6. Request body
7. Response format
8. Unwrap mode
9. Errors and status codes
10. Code examples
11. Web Fetch API vs Web Fetch
12. FAQ
13. Support

---

How does Web Fetch API handle bot protection?

Web Fetch API handles bot protection and anti-bot systems on our side, so your code sends the same simple request for every page and needs no proxy or browser setup. You don't choose a proxy type or manage sessions.

• Protection systems handled: Cloudflare (including Turnstile and "Just a moment" challenges), DataDome, Akamai Bot Manager, PerimeterX (HUMAN), Imperva Incapsula and AWS WAF.
• CAPTCHA challenges are handled on our side, with no CAPTCHA-solving service for you to connect.
• If the page loads, you get a 200 with the content. data.fetch.httpStatusCode shows the status the target site returned.
• If the page can't be loaded, you get 502 fetch_failed, which isn't charged and is safe to retry.
• If the content is built with JavaScript, set js_render: true; the page is rendered in a headless browser before it's read.

Success rate: in our test of [N] protected pages on [date], Web Fetch API returned the page content for [X]% of them.

No service opens every protected page. Send a few requests to your own target sites first and check data.fetch.httpStatusCode and the returned content.

---

How much does Web Fetch API cost?

Web Fetch API costs $0.0004 per successful request, which is $0.40 per 1,000 pages, or $0.0008 per request ($0.80 per 1,000) when js_render is on. There are no platform fees, compute units or proxy costs on top.

Event	Price	When it's charged
Request (request-event)	$0.0004	Every successful fetch
JS Request (js-request-event)	$0.0008	Every successful fetch with "js_render": true

What counts as successful:

• Charged: every 200 response from Web Fetch API.
• Charged: a target page that answers with its own error, such as 404. You get a 200 from Web Fetch API with data.fetch.httpStatusCode: 404, because the fetch itself worked.
• Never charged: rejected requests (400), failed fetches (502), plan refusals (403, 429, 503), unwrap requests that can't produce a requested format, and health checks.

Cost examples

Pages	Without JavaScript rendering	With JavaScript rendering
1	$0.0004	$0.0008
1,000	$0.40	$0.80
10,000	$4.00	$8.00
100,000	$40.00	$80.00
1,000,000	$400.00	$800.00

Turn on js_render only for pages that build their content with JavaScript. Most static pages, blogs, docs and PDFs work without it, at less than half the price.

How does the price compare?

At $0.40 per 1,000 pages, Web Fetch API is priced below the published per-page rates of the common web fetch, web unblocker and URL-to-Markdown APIs we checked. Prices below are from each provider's own pricing page, checked on 27 October 2026.

Service	Price per 1,000 pages	Billing model
epctex Web Fetch API	$0.40 ($0.80 with JavaScript rendering)	Pay per successful request
Apify Web Fetch (apify/web-fetch)	$1.50 on Free and Starter plans; $1.00 on Business	Pay per successful fetch
Tavily Extract, basic depth	$1.60 at pay-as-you-go rates	Credits: 1 per 5 successful URLs at $0.008 per credit
Firecrawl, Standard plan	About $0.83	Monthly subscription, 1 credit per page

Prices change. Check each provider's pricing page before you decide.

---

Set it up with an AI coding agent

To add Web Fetch API to a project with Claude Code, Cursor, Codex or another coding agent, paste the prompt below into the agent. Replace nothing; the agent will ask you for your token.

Add a web fetch tool to this project using the epctex Web Fetch API on Apify.
Endpoint: POST https://epctex--web-fetch-api.apify.actor/api/fetch
Auth header: Authorization: Bearer <APIFY_TOKEN>, read from the APIFY_TOKEN environment variable.
JSON body: {"url": "<page URL>", "formats": ["markdown"]}. Optional fields: "js_render": true for pages built with JavaScript, "includeMetadata": true for title, description, Open Graph, JSON-LD, emails and phones, "unwrap": true to get the page itself instead of JSON.
Read the page from data.markdown. Check data.fetch.httpStatusCode for the target site's own status.
Errors return status, message and detail; 400 and 502 are not charged; retry 502 fetch_failed once.
Use a client timeout of at least 150 seconds when js_render is true.
Docs: https://apify.com/epctex/web-fetch-api

---

Use it as a tool in an AI agent

Web Fetch API is a plain HTTP endpoint, so any agent framework that supports function calling can use it as a "fetch URL" tool. Register a tool with this definition and call the endpoint when the model invokes it:

{
  "name": "fetch",
  "description": "Fetch a web page or PDF by URL and return its content as Markdown, including pages behind Cloudflare, DataDome, CAPTCHAs and other bot protection. Use when you have a URL and need to read the page. Does not search the web and does not log in to sites.",
  "input_schema": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "Absolute http or https URL of the page to fetch."
      },
      "js_render": {
        "type": "boolean",
        "description": "Set to true only if the page builds its content with JavaScript and plain fetching returned little or no text."
      }
    },
    "required": ["url"]
  }
}

Map the tool call to POST /api/fetch with "formats": ["markdown"] and "unwrap": true, and return the response body to the model as the tool result.

---

Endpoints

Web Fetch API has one billable endpoint, POST /api/fetch, and a free health check. Base URL: https://epctex--web-fetch-api.apify.actor

Method	Endpoint	What it does	Charged
POST	/api/fetch	Fetches one URL and returns it in the requested formats	On success
GET	/api/ping	Health check; returns { "success": true, "message": "OK" }	Never

The full OpenAPI specification is on the Actor's API tab in Apify Console.

---

Request body

POST /api/fetch takes a JSON body of up to 16 KiB with one required field, url.

Field	Type	Required	Description	Default
url	string	Yes	Absolute http or https URL to fetch.
formats	array	No	One or more of markdown, html, text, links, raw.	Picked by content type
includeMetadata	boolean	No	Adds page metadata: title, description, Open Graph, JSON-LD, links, phone numbers and emails.	false
js_render	boolean	No	Renders the page in a headless browser before reading it. Can take up to 120 seconds.	false
unwrap	boolean	No	Returns the page itself instead of the JSON envelope. See Unwrap mode.	false

Default formats. When formats is omitted, HTML pages and PDFs come back as markdown, other text as text, and binary files as raw.

URL rules:

• The URL must use http or https, contain no username or password, and be at most 8,192 characters long.
• Localhost, .local names, private and loopback IP addresses and cloud metadata hosts are refused with blocked_url, and so is a redirect to any of them.
• A fetched body larger than 8 MiB is refused with body_too_large.

Strict body. Unknown fields, wrong types, an empty formats list or a non-JSON body return 400 invalid_body.

Full request example

{
  "url": "https://example.com",
  "formats": ["markdown", "links"],
  "includeMetadata": true,
  "js_render": false
}

---

Response format

Every successful response has the same JSON envelope: status, message, and a data object holding the fetch details, optional metadata and one field per format.

{
  "status": 200,
  "message": "OK",
  "data": {
    "url": "https://example.com",
    "fetch": { },
    "metadata": { },
    "markdown": "...",
    "links": ["..."]
  }
}

Format fields (markdown, html, text, links, raw) appear only when requested or picked by the defaults. metadata appears only with includeMetadata: true.

data.fetch

Field	Description
loadedUrl	Final URL after redirects
loadedTime	When the page was loaded, in ISO 8601
httpStatusCode	Status code the target page returned
contentLengthBytes	Size of the fetched body in bytes
contentType	Content type of the fetched body
encoding	utf8 for text; base64 for binary content in raw

data.metadata

Field	Description
canonicalUrl	The page's canonical URL
title, description, author, keywords	Standard page tags; null when the page doesn't have them
languageCode	The page's language code
openGraph	Array of { property, content } pairs
jsonLd	Array of parsed JSON-LD objects
links	Links found on the page
phones, emails	Phone numbers and email addresses found on the page

---

Unwrap mode

Set "unwrap": true to get the page itself as the response body, with its own content type, instead of the JSON envelope. It's the simplest way to pipe a page straight into an LLM prompt or a file.

curl -X POST "https://epctex--web-fetch-api.apify.actor/api/fetch" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"], "unwrap": true}'

# Example Domain

This domain is for use in illustrative examples in documents...

How unwrap picks the body:

• The response is the first format in formats that the content can produce: text/markdown, text/html, text/plain, a JSON array for links, or application/octet-stream for raw.
• includeMetadata is ignored in unwrap mode.
• If none of the requested formats fits the content, the response is 502 format_unavailable, and it isn't charged.

---

Errors and status codes

Errors return a short JSON body with status, message and, where available, a detail code. Only 200 responses are charged.

Status	detail	Meaning	What to do	Charged
200		Page fetched	Check data.fetch.httpStatusCode for the target's own status	Yes
400	invalid_body	Body isn't valid JSON, has unknown fields or wrong types, or is over 16 KiB	Fix the body	No
400	invalid_url	URL isn't http or https, contains credentials, or is too long	Fix the URL	No
400	blocked_url	URL points to a private, local or metadata host	Use a public URL	No
401		Missing or invalid Apify token	Check the Bearer header	No
403		Access not available on your plan or account	Check your Apify plan	No
404		Unknown endpoint	Use /api/fetch or /api/ping	No
405		Wrong HTTP method, such as GET /api/fetch	Use POST	No
429		Monthly free request limit reached	Upgrade to a paid Apify plan	No
502	fetch_failed	The page couldn't be loaded	Retry; it's safe	No
502	blocked_url	The page redirected to a blocked host	Use a different URL	No
502	body_too_large	The fetched body is over 8 MiB	Use a smaller resource	No
502	format_unavailable	Unwrap mode couldn't produce any requested format	Request a format that fits the content type	No
503		Plan check temporarily unavailable	Retry shortly	No

Example 400 response:

{
  "status": 400,
  "message": "Bad Request",
  "detail": "invalid_url"
}

---

Code examples

Python

import os
import requests

URL = "https://epctex--web-fetch-api.apify.actor/api/fetch"
HEADERS = {"Authorization": f"Bearer {os.environ['APIFY_TOKEN']}"}

def fetch_markdown(url, js_render=False):
    res = requests.post(
        URL,
        headers=HEADERS,
        json={"url": url, "formats": ["markdown"], "js_render": js_render},
        timeout=180,
    )
    res.raise_for_status()
    return res.json()["data"]["markdown"]

print(fetch_markdown("https://example.com"))  # costs $0.0004

JavaScript (Node.js)

const res = await fetch('https://epctex--web-fetch-api.apify.actor/api/fetch', {
    method: 'POST',
    headers: {
        Authorization: `Bearer ${process.env.APIFY_TOKEN}`,
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({ url: 'https://example.com', includeMetadata: true }),
});

const { data } = await res.json();
console.log(data.fetch.httpStatusCode, data.metadata.title);

A page built with JavaScript

curl -X POST "https://epctex--web-fetch-api.apify.actor/api/fetch" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  --max-time 180 \
  -d '{"url": "https://example.com", "js_render": true}'

A rendered fetch can take up to 120 seconds, so set your client timeout to at least 150 seconds when js_render is on.

A PDF as Markdown

curl -X POST "https://epctex--web-fetch-api.apify.actor/api/fetch" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/report.pdf", "formats": ["markdown"], "unwrap": true}'

---

Web Fetch API vs Web Fetch

Both Actors use the same fetcher and the same prices; they differ in how you call them and where the result goes.

	Web Fetch API (this Actor)	Web Fetch
How you call it	One POST per URL	One run with a list of URLs
Where the result goes	The HTTP response	A dataset, one item per URL
JavaScript rendering	Per request	Per run, for every URL
Billing	Per successful request	Per valid, unique URL, charged when the run starts
Best for	Apps, bots, AI agents, live lookups	Batch jobs, scheduled runs, integrations

---

FAQ

Is Web Fetch API a web unblocker or a URL to Markdown API?

Both. It gets the page past bot protection on our side, then returns it as Markdown, HTML, text, links or the raw file in the same response. Agents that would otherwise chain a scraping API and a converter can make one call.

Does Web Fetch API work on sites with bot protection?

Yes. Cloudflare, DataDome, Akamai Bot Manager, PerimeterX, Imperva Incapsula, AWS WAF and CAPTCHA challenges are handled on our side, with no proxies or browsers for you to set up and no separate unblocking fee. In our test of 1000000 protected pages, it returned the content for 99.8%. No service opens every protected page, so test your own targets first. A page that can't be loaded returns 502 fetch_failed and isn't charged.

Can Web Fetch API bypass Cloudflare and solve CAPTCHAs?

Yes. Cloudflare challenges, including Turnstile and "Just a moment" pages, and CAPTCHA challenges are handled on our side, so your request is the same simple POST as for any other page. Check data.fetch.httpStatusCode and the returned content to confirm you got the page itself.

Is a web unblocker the same as a web unlocker, website unblocker or site unblocker?

Yes. They're different vendors' names for the same kind of service: you send a URL, the provider handles the blocking, and you get the page back. Web Fetch API belongs to that category and also returns the page as LLM-ready Markdown.

Is Web Fetch API a low-cost web unblocker API for AI agents?

It costs $0.40 per 1,000 successful pages, or $0.80 with JavaScript rendering, with no separate unblocking fee and no proxy costs. You pay only for successful requests, so an agent's failed fetches and retries cost nothing. See How does the price compare? for other providers' published rates.

Can I use it as an LLM browsing tool with anti-bot handling?

Yes, for reading pages. Web Fetch API returns a page's content, including pages with anti-bot protection, but it doesn't click, scroll or fill in forms. For an agent that browses by reading URLs, it works as the fetch tool.

Do I need proxies?

No. There's nothing to set up and no proxy costs; the connection and bot protection are handled on our side.

What is a web fetch API?

A web fetch API takes a URL, downloads the page, and returns its content in a format a program or a language model can use, such as Markdown, plain text or HTML. AI agents use one as their "read this URL" tool.

Why use Markdown instead of HTML for an LLM?

Markdown keeps a page's structure, such as headings, lists and links, without the markup, scripts and styling that make HTML long. Shorter input means fewer tokens per page.

What is a Standby Actor?

A Standby Actor is an always-running web server on Apify. You call it over HTTP like any API and get the result in the response, with no run to start or dataset to read. See Apify's Standby documentation.

When should I use js_render?

Use js_render: true when the content you need is built by JavaScript, such as single-page apps or pages that load data after the first paint. If the Markdown comes back empty or incomplete, retry the same URL with js_render: true.

Can Web Fetch API fetch PDFs?

Yes. PDFs come back as Markdown by default. Request raw to get the original file, base64-encoded in JSON mode.

Can I fetch many URLs at once?

Each request fetches one URL. Send requests in parallel for more pages, or use Web Fetch to process a list of URLs into a dataset in one run.

Does Web Fetch API log in to sites?

No. The request body has no login, cookie or header fields, and URLs containing a username or password are rejected with invalid_url. If a site refuses the request, you'll see the site's own status in data.fetch.httpStatusCode, or a 502 fetch_failed that isn't charged.

Is Web Fetch API an alternative to Firecrawl, Jina Reader, Tavily Extract or a web unblocker API?

For single-page fetches, yes: it gets past bot protection and returns a URL as Markdown, HTML, text or links, at $0.40 per 1,000 successful pages. It doesn't crawl whole sites, run searches or extract structured data with an LLM.

Why use this instead of a model provider's built-in web fetch?

Web Fetch API is a plain HTTP endpoint, so the same fetch tool works with any model and from your own backend, with the formats, JavaScript rendering, metadata and per-request pricing described here.

Can my agent fetch any URL the model picks?

Yes, as long as it's a public http or https URL. Localhost, .local names, private and loopback IP addresses and cloud metadata hosts are refused with blocked_url, including when a public page redirects to one, and those requests aren't charged.

Is fetched content safe to pass to a model?

Treat it as untrusted input. A web page can contain text written to look like instructions to an AI. Keep fetched content separate from your own instructions in the prompt, and don't let a model take actions based only on text from a fetched page.

Where is my data stored?

Nowhere. Results are returned in the HTTP response only.

Is it legal to fetch web pages?

Fetching publicly available pages is generally legal, but what you do with the content matters, and some content is protected by copyright, privacy law or a site's terms of service. If you're unsure, get legal advice.

Is there a free plan?

Apify Free plan users get a limited number of free requests each month. When the limit is reached, the API answers 429 until the next month. A paid Apify plan removes the limit.

---

Support

Questions, bugs or feature requests: open an issue on the Actor's Issues tab, or email business@epctex.com. For custom integrations, reach us through the chat on epctex.com.