# Web Fetch: Bulk Web Unblocker & URL to Markdown (`epctex/web-fetch`) Actor

Bulk web unblocker: a list of URLs to LLM-ready Markdown, HTML, text or links in one run, for RAG. $0.40 per 1,000 pages. Handles Cloudflare, DataDome and CAPTCHAs; PDFs to Markdown; JavaScript rendering $0.80 per 1,000. Optional page metadata. Export JSON, CSV or Excel. Doesn't crawl.

- **URL**: https://apify.com/epctex/web-fetch.md
- **Developed by:** [epctex](https://apify.com/epctex) (community)
- **Features:** Limited permissions
- **Stats:** 6 total users, 5 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.37 / 1,000 fetches

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Fetch: Bulk Web Unblocker and URL to Markdown for RAG and AI Agents

**Web Fetch is a bulk web unblocker that turns a list of URLs into LLM-ready Markdown, HTML, text, links or raw files in one run, one dataset item per page. Cloudflare, DataDome, CAPTCHAs and other bot protection are handled on our side, JavaScript is rendered on request, and PDFs come back as Markdown. It costs $0.40 per 1,000 pages ($0.80 with JavaScript rendering), with no proxy or dataset fees.**

Paste in URLs, run it once or on a schedule, and export the results as JSON, CSV, Excel or XML, or read them through the Apify API. There are no proxies or browsers to manage, and AI agents can call it through Apify's MCP server. It's built for RAG and LLM ingestion, content monitoring, SEO checks and any job that turns many pages, including protected ones, into clean, model-ready text.

| | |
| --- | --- |
| **Input** | `startUrls`: a list of `http` or `https` URLs |
| **Price** | $0.0004 per URL; $0.0008 per URL with `jsRender: true` |
| **Bot protection** | Cloudflare, DataDome, Akamai, PerimeterX, Imperva and AWS WAF, plus CAPTCHA challenges, handled on our side; no proxies or browsers to set up |
| **Formats** | `markdown`, `html`, `text`, `links`, `raw` (PDFs default to Markdown) |
| **Extras** | Page metadata: title, description, Open Graph, JSON-LD, links, emails, phone numbers |
| **Output** | One dataset item per URL, exportable as JSON, CSV, Excel or XML |
| **AI agents** | Callable from Claude, Cursor, VS Code and other MCP clients via `https://mcp.apify.com/?tools=epctex/web-fetch` |
| **Clean input** | Duplicate URLs are fetched once; invalid URLs are skipped with a warning and not charged |

Need one page at a time, live, over HTTP? Use **[Web Fetch API](https://apify.com/epctex/web-fetch-api?fpr=yhdrb)**, the always-on version of the same fetcher at the same price.

***

### Quick start

**One required field, `startUrls`. Everything else is optional.**

```json
{
  "startUrls": ["https://example.com", "https://docs.apify.com/platform/actors"],
  "formats": ["markdown"]
}
```

Each URL comes back as one dataset item:

```json
{
  "url": "https://example.com",
  "fetch": { "loadedUrl": "https://example.com/", "httpStatusCode": 200, "contentType": "text/html; charset=UTF-8" },
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents..."
}
```

Those two pages cost $0.0008. Full input and output reference is below.

***

### When to use Web Fetch

**Use Web Fetch when you have a list of URLs, including pages that block plain scripted requests, and want every page saved as clean text in one dataset.** Each run processes the whole list and writes one item per page.

**Use it for:**

- **RAG and LLM ingestion:** turn documentation, blog posts and PDFs into Markdown before chunking and embedding.
- **Protected pages at scale:** read sites that return `403`, "access denied" or a challenge page to plain HTTP clients, without running your own proxies or browsers.
- **Content monitoring:** fetch the same pages on a schedule and compare the results.
- **SEO checks:** read titles, descriptions, canonical URLs, Open Graph tags and JSON-LD in bulk with `includeMetadata`.
- **Lead research:** collect public emails and phone numbers from company websites with `includeMetadata`.
- **Link discovery:** get every link on a set of pages to plan the next run.

**Use something else for:**

- **One page at a time, inside an app or agent:** use [Web Fetch API](https://apify.com/epctex/web-fetch-api?fpr=yhdrb), which answers in the HTTP response.
- **Whole-site crawls:** Web Fetch fetches exactly the URLs you give it and doesn't follow links. Run it once with the `links` format, then again on the URLs you need.
- **Pages behind a login or paywall:** Web Fetch has no login or cookie options, and URLs containing a username or password are skipped.

***

### Why use Web Fetch for RAG and LLM ingestion?

- **Web unblocker and converter in one run.** Bot protection and anti-bot systems are handled on our side, and every page lands in the dataset ready for a model. No separate proxy service, scraping API and HTML-to-Markdown step.
- **Markdown by default.** HTML pages and PDFs come back as Markdown, which keeps headings, lists and links but drops the markup, so each page costs fewer tokens to embed or prompt with.
- **One flat price per page.** $0.40 per 1,000 pages, with no proxy or dataset item fees and no separate unblocking fee.
- **JavaScript only where needed.** Plain fetching is the cheaper default; turn on `jsRender` for single-page apps and late-loading content.
- **Metadata in the same pass.** `includeMetadata` adds titles, descriptions, canonical URLs, Open Graph, JSON-LD, emails and phone numbers to each item.
- **Agent and pipeline friendly.** Call it from an MCP client, the Apify API, Python or JavaScript, or no-code tools like Make, Zapier and n8n, and load the dataset into LangChain, LlamaIndex or a vector database.

***

### Contents

1. [How does Web Fetch handle bot protection?](#how-does-web-fetch-handle-bot-protection)
2. [How much does Web Fetch cost?](#how-much-does-web-fetch-cost)
3. [How to use Web Fetch](#how-to-use-web-fetch)
4. [Input](#input)
5. [Output](#output)
6. [Use Web Fetch with AI agents and integrations](#use-web-fetch-with-ai-agents-and-integrations)
7. [Web Fetch vs Web Fetch API](#web-fetch-vs-web-fetch-api)
8. [FAQ](#faq)
9. [Support](#support)

***

### How does Web Fetch handle bot protection?

**Web Fetch handles bot protection and anti-bot systems on our side, so every URL in the list goes through the same simple input with no proxy or browser setup.** You don't choose a proxy type or manage sessions.

- **Protection systems handled:** Cloudflare (including Turnstile and "Just a moment" challenges), DataDome, Akamai Bot Manager, PerimeterX (HUMAN), Imperva Incapsula and AWS WAF.
- **CAPTCHA challenges** are handled on our side, with no CAPTCHA-solving service for you to connect.
- **If a page loads,** its item holds the content, and `fetch.httpStatusCode` shows the status the target site returned.
- **If a page can't be loaded,** it's logged as an error and has no item in the dataset.
- **If the content is built with JavaScript,** turn on `jsRender`; every page in the run is rendered in a headless browser first.

**Success rate:** in our test of 1000000 protected pages on 2026-10-25, Web Fetch returned the page content for 99.8% of them.

URLs are charged when the run starts, so a page that can't be loaded is still charged. No service opens every protected page: run a short list of your target sites first, then the full list.

***

### How much does Web Fetch cost?

**Web Fetch costs $0.0004 per URL, which is $0.40 per 1,000 pages, or $0.0008 per URL ($0.80 per 1,000) when `jsRender` is on.** There are no dataset item fees and no proxy costs on top.

| Event | Price | When it's charged |
| --- | --- | --- |
| Request | $0.0004 per URL | Each URL, when `jsRender` is off |
| JS Request | $0.0008 per URL | Each URL, when `jsRender` is on |

**How billing works:**

- Each valid, unique URL is charged once, when the run starts.
- Duplicate URLs are fetched once and charged once.
- Invalid URLs are skipped with a warning and aren't charged.
- `jsRender` applies to the whole run, so every URL in a run uses the same price.

Because URLs are charged when the run starts, a URL that later fails to load is still charged. Test unfamiliar sites with a short list first.

#### Cost examples

| Scenario | Calculation | Total |
| --- | --- | --- |
| 10 pages | 10 × $0.0004 | $0.004 |
| 1,000 pages | 1,000 × $0.0004 | $0.40 |
| 1,000 pages with JavaScript rendering | 1,000 × $0.0008 | $0.80 |
| 10,000 pages | 10,000 × $0.0004 | $4.00 |
| 100,000 pages | 100,000 × $0.0004 | $40.00 |

Turn on `jsRender` only for pages that need it. Most static pages, blogs, docs and PDFs work without it, at less than half the price. If a list mixes both kinds, run them as two separate runs.

***

### How to use Web Fetch

**You need an Apify account; Web Fetch needs no proxy setup or browser configuration.** Four steps:

1. **Open** Web Fetch in Apify Console and paste your URLs into **Start URLs**.
2. **Choose** the formats you want, or leave them empty for the defaults.
3. **Turn on** `includeMetadata` or `jsRender` if you need them.
4. **Click Start**, then download the dataset from the **Output** tab.

To run it on a schedule, save the input as a task and add a schedule in Apify Console.

***

### Input

**Web Fetch has one required field, `startUrls`, and three optional settings.**

| Field | Type | Required | Description | Default |
| --- | --- | --- | --- | --- |
| `startUrls` | array | Yes | List of `http` or `https` URLs to fetch, for example `["https://example.com"]`. | |
| `formats` | array | No | Formats to return: `markdown`, `html`, `text`, `links`, `raw`. Leave empty for the defaults. | Picked by content type |
| `includeMetadata` | boolean | No | Adds page metadata: title, description, Open Graph, JSON-LD, links, phone numbers and emails. | `false` |
| `jsRender` | boolean | No | Renders each page in a headless browser before reading it. A rendered fetch can take up to 120 seconds. | `false` |

**Default formats.** With no formats selected, HTML pages and PDFs come back as `markdown`, other text as `text`, and binary files as `raw`.

**URL rules.** URLs must use `http` or `https`, must not contain a username or password, and can be up to 8,192 characters long.

#### Input example

```json
{
  "startUrls": [
    "https://example.com",
    "https://docs.apify.com/platform/actors"
  ],
  "formats": ["markdown", "links"],
  "includeMetadata": true,
  "jsRender": false
}
```

***

### Output

**Each URL becomes one item in the dataset, with the fetch details, optional metadata and one field per format.** Format fields appear only when requested or picked by the defaults; `metadata` appears only when `includeMetadata` is on.

#### Output example

```json
{
  "url": "https://example.com",
  "fetch": {
    "loadedUrl": "https://example.com/",
    "loadedTime": "2026-10-26T12:00:00.000Z",
    "httpStatusCode": 200,
    "contentLengthBytes": 1256,
    "contentType": "text/html; charset=UTF-8"
  },
  "metadata": {
    "canonicalUrl": null,
    "title": "Example Domain",
    "description": null,
    "author": null,
    "keywords": null,
    "languageCode": "en",
    "openGraph": [],
    "jsonLd": [],
    "links": ["https://www.iana.org/domains/example"],
    "phones": [],
    "emails": []
  },
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents...",
  "links": ["https://www.iana.org/domains/example"]
}
```

#### Output fields

| Field | Description |
| --- | --- |
| `url` | The URL you provided |
| `fetch.loadedUrl` | The final URL after redirects |
| `fetch.loadedTime` | When the page was loaded, in ISO 8601 |
| `fetch.httpStatusCode` | The status code the page returned, such as `200` or `404` |
| `fetch.contentLengthBytes` | Size of the fetched body in bytes |
| `fetch.contentType` | Content type of the fetched body |
| `metadata` | Page metadata, when `includeMetadata` is on |
| `markdown`, `html`, `text` | The page in that format |
| `links` | Every link found on the page |
| `raw` | The original body |

You can download the dataset as JSON, CSV, Excel or XML, or read it through the [Apify API](https://docs.apify.com/api/v2?fpr=yhdrb).

***

### Use Web Fetch with AI agents and integrations

**Web Fetch runs like any other Apify Actor, so it works with the Apify API, Apify's MCP server and Apify's integrations.**

- **AI agents (MCP):** add `https://mcp.apify.com/?tools=epctex/web-fetch` as a remote MCP server in Claude, Cursor, VS Code or another MCP client. `startUrls` is the only required field, and no site login or credentials are needed.
- **Run length with agents:** for an answer inside one agent turn, send a few URLs at a time. Larger runs take longer, and the agent can read the dataset when the run finishes.
- **Code:** call it with the Apify client for Python or JavaScript, and read the run's dataset.
- **No-code tools:** connect it to Make, Zapier, n8n and other tools through [Apify integrations](https://docs.apify.com/platform/integrations?fpr=yhdrb).
- **LLM frameworks:** load the dataset into LangChain, LlamaIndex or a vector database for RAG.

#### Python example

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("epctex/web-fetch").call(run_input={
    "startUrls": ["https://example.com", "https://docs.apify.com/platform/actors"],
    "formats": ["markdown"],
})

for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["fetch"]["httpStatusCode"], len(page.get("markdown") or ""))
```

***

### Web Fetch vs Web Fetch API

**Both Actors use the same fetcher and the same prices; they differ in how you call them and where the result goes.**

| | Web Fetch (this Actor) | [Web Fetch API](https://apify.com/epctex/web-fetch-api?fpr=yhdrb) |
| --- | --- | --- |
| How you call it | One run with a list of URLs | One `POST` per URL |
| Where the result goes | A dataset, one item per URL | The HTTP response |
| JavaScript rendering | Per run, for every URL | Per request |
| Billing | Per valid, unique URL, charged when the run starts | Per successful request |
| Best for | Batch jobs, scheduled runs, integrations | Apps, bots, AI agents, live lookups |

***

### FAQ

#### Is Web Fetch a web unblocker or a URL to Markdown converter?

Both. It gets each page past bot protection on our side, then saves it as Markdown, HTML, text, links or the raw file. A list of URLs goes in; one clean item per page comes out.

#### Does Web Fetch work on sites with bot protection?

Yes. Cloudflare, DataDome, Akamai Bot Manager, PerimeterX, Imperva Incapsula, AWS WAF and CAPTCHA challenges are handled on our side, with no proxies or browsers for you to set up and no separate unblocking fee. In our test of 1000000 protected pages, it returned the content for 99.8%. No service opens every protected page, so test a short list of your targets first. Pages that can't be loaded are logged as errors and are still charged, because URLs are charged when the run starts.

#### Is a web unblocker the same as a web unlocker, website unblocker or site unblocker?

Yes. They're different vendors' names for the same kind of service: you send a URL, the provider handles the blocking, and you get the page back. Web Fetch does this for a whole list of URLs in one run and saves each page as LLM-ready Markdown.

#### Can Web Fetch bypass Cloudflare and solve CAPTCHAs?

Yes. Cloudflare challenges, including Turnstile and "Just a moment" pages, and CAPTCHA challenges are handled on our side for every URL in the run. Check `fetch.httpStatusCode` and the saved content to confirm each page is the page itself.

#### Is Web Fetch a low-cost bulk web unblocker?

It costs $0.40 per 1,000 pages, or $0.80 with JavaScript rendering, with no separate unblocking fee, no proxy costs and no dataset fees. Every valid, unique URL is charged when the run starts, including pages that later fail to load.

#### When should I turn on `jsRender`?

Turn on `jsRender` when the content you need isn't in the page source, such as single-page apps or pages that load data after the first paint. If the Markdown comes back empty or incomplete, run those URLs again with `jsRender` on.

#### What happens if a page returns 404 or 500?

The page is still saved, and `fetch.httpStatusCode` shows the status. Check it before you use the content.

#### Why is one of my URLs missing from the results?

Invalid URLs, meaning not `http` or `https`, containing a username or password, or longer than 8,192 characters, are skipped and logged as a warning. Duplicates are fetched once. A URL that can't be loaded at all is logged as an error.

#### Can Web Fetch fetch PDFs?

Yes. PDFs come back as Markdown by default.

#### Do I need proxies?

No. There's nothing to set up and no proxy costs; the connection and bot protection are handled on our side.

#### Does Web Fetch log in to sites?

No. The input has no login or cookie fields, and URLs containing a username or password are skipped with a warning. It fetches public pages.

#### Is Web Fetch an alternative to Firecrawl, Jina Reader or a web unblocker API?

For fetching a known list of pages, including protected ones, and converting them to Markdown, yes, at $0.40 per 1,000 pages. It doesn't crawl whole sites or extract structured data with an LLM.

#### Can I use Web Fetch from Claude, Cursor or another MCP client?

Yes. Add `https://mcp.apify.com/?tools=epctex/web-fetch` as a remote MCP server. `startUrls` is the only required field. Send a few URLs at a time when the agent needs the answer in the same turn.

#### Can Web Fetch crawl a whole website?

No. It fetches exactly the URLs you give it and doesn't follow links. To work through a site, run it once with the `links` format, then run it again on the URLs you need.

#### How do I convert a long list of URLs to Markdown?

Paste the whole list into `startUrls` and leave `formats` empty or set it to `["markdown"]`. Duplicates are removed automatically, and every page lands in one dataset you can export or read through the API.

#### Is fetched content safe to pass to a model?

Treat it as untrusted input. A web page can contain text written to look like instructions to an AI. Keep fetched content separate from your own instructions in the prompt, and don't let a model take actions based only on text from a fetched page.

#### Is it legal to fetch web pages?

Fetching publicly available pages is generally legal, but what you do with the content matters, and some content is protected by copyright, privacy law or a site's terms of service. Collecting emails and phone numbers can be subject to data protection law such as the GDPR. If you're unsure, get legal advice.

#### Is there a free plan?

Apify Free plan users can try Web Fetch with limited usage. [A paid Apify plan](https://apify.com/pricing?fpr=yhdrb) removes the limit.

***

### Support

Questions, bugs or feature requests: open an issue on the Actor's **Issues** tab, or email <business@epctex.com>. For custom integrations, reach us through the chat on [epctex.com](https://epctex.com).

# Actor input Schema

## `startUrls` (type: `array`):

List of http or https URLs to fetch.

## `jsRender` (type: `boolean`):

Run the page in a headless browser before reading it. A rendered fetch can take up to 120 seconds.

## `includeMetadata` (type: `boolean`):

Include page metadata such as title, description, links, phones, and emails.

## `formats` (type: `array`):

Representations to return. When omitted, HTML and PDF come back as markdown, other text as text, and binary content as raw.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "jsRender": true,
  "includeMetadata": true,
  "formats": [
    "markdown",
    "links"
  ]
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://example.com"
    ],
    "formats": [
        "markdown"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("epctex/web-fetch").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://example.com"],
    "formats": ["markdown"],
}

# Run the Actor and wait for it to finish
run = client.actor("epctex/web-fetch").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://example.com"
  ],
  "formats": [
    "markdown"
  ]
}' |
apify call epctex/web-fetch --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,epctex/web-fetch"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MyxRKaOCbfJ6m5WKX/builds/KvCrm0vU7mfWLIo8F/openapi.json