Web Fetch: Bulk Web Unblocker & URL to Markdown

Web Fetch: Bulk Web Unblocker and URL to Markdown for RAG and AI Agents

Web Fetch is a bulk web unblocker that turns a list of URLs into LLM-ready Markdown, HTML, text, links or raw files in one run, one dataset item per page. Cloudflare, DataDome, CAPTCHAs and other bot protection are handled on our side, JavaScript is rendered on request, and PDFs come back as Markdown. It costs $0.40 per 1,000 pages ($0.80 with JavaScript rendering), with no proxy or dataset fees.

Paste in URLs, run it once or on a schedule, and export the results as JSON, CSV, Excel or XML, or read them through the Apify API. There are no proxies or browsers to manage, and AI agents can call it through Apify's MCP server. It's built for RAG and LLM ingestion, content monitoring, SEO checks and any job that turns many pages, including protected ones, into clean, model-ready text.

Input	startUrls: a list of http or https URLs
Price	$0.0004 per URL; $0.0008 per URL with jsRender: true
Bot protection	Cloudflare, DataDome, Akamai, PerimeterX, Imperva and AWS WAF, plus CAPTCHA challenges, handled on our side; no proxies or browsers to set up
Formats	markdown, html, text, links, raw (PDFs default to Markdown)
Extras	Page metadata: title, description, Open Graph, JSON-LD, links, emails, phone numbers
Output	One dataset item per URL, exportable as JSON, CSV, Excel or XML
AI agents	Callable from Claude, Cursor, VS Code and other MCP clients via https://mcp.apify.com/?tools=epctex/web-fetch
Clean input	Duplicate URLs are fetched once; invalid URLs are skipped with a warning and not charged

Need one page at a time, live, over HTTP? Use Web Fetch API, the always-on version of the same fetcher at the same price.

---

Quick start

One required field, startUrls. Everything else is optional.

{
  "startUrls": ["https://example.com", "https://docs.apify.com/platform/actors"],
  "formats": ["markdown"]
}

Each URL comes back as one dataset item:

{
  "url": "https://example.com",
  "fetch": { "loadedUrl": "https://example.com/", "httpStatusCode": 200, "contentType": "text/html; charset=UTF-8" },
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents..."
}

Those two pages cost $0.0008. Full input and output reference is below.

---

When to use Web Fetch

Use Web Fetch when you have a list of URLs, including pages that block plain scripted requests, and want every page saved as clean text in one dataset. Each run processes the whole list and writes one item per page.

Use it for:

• RAG and LLM ingestion: turn documentation, blog posts and PDFs into Markdown before chunking and embedding.
• Protected pages at scale: read sites that return 403, "access denied" or a challenge page to plain HTTP clients, without running your own proxies or browsers.
• Content monitoring: fetch the same pages on a schedule and compare the results.
• SEO checks: read titles, descriptions, canonical URLs, Open Graph tags and JSON-LD in bulk with includeMetadata.
• Lead research: collect public emails and phone numbers from company websites with includeMetadata.
• Link discovery: get every link on a set of pages to plan the next run.

Use something else for:

• One page at a time, inside an app or agent: use Web Fetch API, which answers in the HTTP response.
• Whole-site crawls: Web Fetch fetches exactly the URLs you give it and doesn't follow links. Run it once with the links format, then again on the URLs you need.
• Pages behind a login or paywall: Web Fetch has no login or cookie options, and URLs containing a username or password are skipped.

---

Why use Web Fetch for RAG and LLM ingestion?

• Web unblocker and converter in one run. Bot protection and anti-bot systems are handled on our side, and every page lands in the dataset ready for a model. No separate proxy service, scraping API and HTML-to-Markdown step.
• Markdown by default. HTML pages and PDFs come back as Markdown, which keeps headings, lists and links but drops the markup, so each page costs fewer tokens to embed or prompt with.
• One flat price per page. $0.40 per 1,000 pages, with no proxy or dataset item fees and no separate unblocking fee.
• JavaScript only where needed. Plain fetching is the cheaper default; turn on jsRender for single-page apps and late-loading content.
• Metadata in the same pass. includeMetadata adds titles, descriptions, canonical URLs, Open Graph, JSON-LD, emails and phone numbers to each item.
• Agent and pipeline friendly. Call it from an MCP client, the Apify API, Python or JavaScript, or no-code tools like Make, Zapier and n8n, and load the dataset into LangChain, LlamaIndex or a vector database.

---

Contents

1. How does Web Fetch handle bot protection?
2. How much does Web Fetch cost?
3. How to use Web Fetch
4. Input
5. Output
6. Use Web Fetch with AI agents and integrations
7. Web Fetch vs Web Fetch API
8. FAQ
9. Support

---

How does Web Fetch handle bot protection?

Web Fetch handles bot protection and anti-bot systems on our side, so every URL in the list goes through the same simple input with no proxy or browser setup. You don't choose a proxy type or manage sessions.

• Protection systems handled: Cloudflare (including Turnstile and "Just a moment" challenges), DataDome, Akamai Bot Manager, PerimeterX (HUMAN), Imperva Incapsula and AWS WAF.
• CAPTCHA challenges are handled on our side, with no CAPTCHA-solving service for you to connect.
• If a page loads, its item holds the content, and fetch.httpStatusCode shows the status the target site returned.
• If a page can't be loaded, it's logged as an error and has no item in the dataset.
• If the content is built with JavaScript, turn on jsRender; every page in the run is rendered in a headless browser first.

Success rate: in our test of 1000000 protected pages on 2026-10-25, Web Fetch returned the page content for 99.8% of them.

URLs are charged when the run starts, so a page that can't be loaded is still charged. No service opens every protected page: run a short list of your target sites first, then the full list.

---

How much does Web Fetch cost?

Web Fetch costs $0.0004 per URL, which is $0.40 per 1,000 pages, or $0.0008 per URL ($0.80 per 1,000) when jsRender is on. There are no dataset item fees and no proxy costs on top.

Event	Price	When it's charged
Request	$0.0004 per URL	Each URL, when jsRender is off
JS Request	$0.0008 per URL	Each URL, when jsRender is on

How billing works:

• Each valid, unique URL is charged once, when the run starts.
• Duplicate URLs are fetched once and charged once.
• Invalid URLs are skipped with a warning and aren't charged.
• jsRender applies to the whole run, so every URL in a run uses the same price.

Because URLs are charged when the run starts, a URL that later fails to load is still charged. Test unfamiliar sites with a short list first.

Cost examples

Scenario	Calculation	Total
10 pages	10 × $0.0004	$0.004
1,000 pages	1,000 × $0.0004	$0.40
1,000 pages with JavaScript rendering	1,000 × $0.0008	$0.80
10,000 pages	10,000 × $0.0004	$4.00
100,000 pages	100,000 × $0.0004	$40.00

Turn on jsRender only for pages that need it. Most static pages, blogs, docs and PDFs work without it, at less than half the price. If a list mixes both kinds, run them as two separate runs.

---

How to use Web Fetch

You need an Apify account; Web Fetch needs no proxy setup or browser configuration. Four steps:

1. Open Web Fetch in Apify Console and paste your URLs into Start URLs.
2. Choose the formats you want, or leave them empty for the defaults.
3. Turn on includeMetadata or jsRender if you need them.
4. Click Start, then download the dataset from the Output tab.

To run it on a schedule, save the input as a task and add a schedule in Apify Console.

---

Input

Web Fetch has one required field, startUrls, and three optional settings.

Field	Type	Required	Description	Default
startUrls	array	Yes	List of http or https URLs to fetch, for example ["https://example.com"].
formats	array	No	Formats to return: markdown, html, text, links, raw. Leave empty for the defaults.	Picked by content type
includeMetadata	boolean	No	Adds page metadata: title, description, Open Graph, JSON-LD, links, phone numbers and emails.	false
jsRender	boolean	No	Renders each page in a headless browser before reading it. A rendered fetch can take up to 120 seconds.	false

Default formats. With no formats selected, HTML pages and PDFs come back as markdown, other text as text, and binary files as raw.

URL rules. URLs must use http or https, must not contain a username or password, and can be up to 8,192 characters long.

Input example

{
  "startUrls": [
    "https://example.com",
    "https://docs.apify.com/platform/actors"
  ],
  "formats": ["markdown", "links"],
  "includeMetadata": true,
  "jsRender": false
}

---

Output

Each URL becomes one item in the dataset, with the fetch details, optional metadata and one field per format. Format fields appear only when requested or picked by the defaults; metadata appears only when includeMetadata is on.

Output example

{
  "url": "https://example.com",
  "fetch": {
    "loadedUrl": "https://example.com/",
    "loadedTime": "2026-10-26T12:00:00.000Z",
    "httpStatusCode": 200,
    "contentLengthBytes": 1256,
    "contentType": "text/html; charset=UTF-8"
  },
  "metadata": {
    "canonicalUrl": null,
    "title": "Example Domain",
    "description": null,
    "author": null,
    "keywords": null,
    "languageCode": "en",
    "openGraph": [],
    "jsonLd": [],
    "links": ["https://www.iana.org/domains/example"],
    "phones": [],
    "emails": []
  },
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents...",
  "links": ["https://www.iana.org/domains/example"]
}

Output fields

Field	Description
url	The URL you provided
fetch.loadedUrl	The final URL after redirects
fetch.loadedTime	When the page was loaded, in ISO 8601
fetch.httpStatusCode	The status code the page returned, such as 200 or 404
fetch.contentLengthBytes	Size of the fetched body in bytes
fetch.contentType	Content type of the fetched body
metadata	Page metadata, when includeMetadata is on
markdown, html, text	The page in that format
links	Every link found on the page
raw	The original body

You can download the dataset as JSON, CSV, Excel or XML, or read it through the Apify API.

---

Use Web Fetch with AI agents and integrations

Web Fetch runs like any other Apify Actor, so it works with the Apify API, Apify's MCP server and Apify's integrations.

• AI agents (MCP): add https://mcp.apify.com/?tools=epctex/web-fetch as a remote MCP server in Claude, Cursor, VS Code or another MCP client. startUrls is the only required field, and no site login or credentials are needed.
• Run length with agents: for an answer inside one agent turn, send a few URLs at a time. Larger runs take longer, and the agent can read the dataset when the run finishes.
• Code: call it with the Apify client for Python or JavaScript, and read the run's dataset.
• No-code tools: connect it to Make, Zapier, n8n and other tools through Apify integrations.
• LLM frameworks: load the dataset into LangChain, LlamaIndex or a vector database for RAG.

Python example

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("epctex/web-fetch").call(run_input={
    "startUrls": ["https://example.com", "https://docs.apify.com/platform/actors"],
    "formats": ["markdown"],
})

for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["fetch"]["httpStatusCode"], len(page.get("markdown") or ""))

---

Web Fetch vs Web Fetch API

Both Actors use the same fetcher and the same prices; they differ in how you call them and where the result goes.

	Web Fetch (this Actor)	Web Fetch API
How you call it	One run with a list of URLs	One POST per URL
Where the result goes	A dataset, one item per URL	The HTTP response
JavaScript rendering	Per run, for every URL	Per request
Billing	Per valid, unique URL, charged when the run starts	Per successful request
Best for	Batch jobs, scheduled runs, integrations	Apps, bots, AI agents, live lookups

---

FAQ

Is Web Fetch a web unblocker or a URL to Markdown converter?

Both. It gets each page past bot protection on our side, then saves it as Markdown, HTML, text, links or the raw file. A list of URLs goes in; one clean item per page comes out.

Does Web Fetch work on sites with bot protection?

Yes. Cloudflare, DataDome, Akamai Bot Manager, PerimeterX, Imperva Incapsula, AWS WAF and CAPTCHA challenges are handled on our side, with no proxies or browsers for you to set up and no separate unblocking fee. In our test of 1000000 protected pages, it returned the content for 99.8%. No service opens every protected page, so test a short list of your targets first. Pages that can't be loaded are logged as errors and are still charged, because URLs are charged when the run starts.

Is a web unblocker the same as a web unlocker, website unblocker or site unblocker?

Yes. They're different vendors' names for the same kind of service: you send a URL, the provider handles the blocking, and you get the page back. Web Fetch does this for a whole list of URLs in one run and saves each page as LLM-ready Markdown.

Can Web Fetch bypass Cloudflare and solve CAPTCHAs?

Yes. Cloudflare challenges, including Turnstile and "Just a moment" pages, and CAPTCHA challenges are handled on our side for every URL in the run. Check fetch.httpStatusCode and the saved content to confirm each page is the page itself.

Is Web Fetch a low-cost bulk web unblocker?

It costs $0.40 per 1,000 pages, or $0.80 with JavaScript rendering, with no separate unblocking fee, no proxy costs and no dataset fees. Every valid, unique URL is charged when the run starts, including pages that later fail to load.

When should I turn on jsRender?

Turn on jsRender when the content you need isn't in the page source, such as single-page apps or pages that load data after the first paint. If the Markdown comes back empty or incomplete, run those URLs again with jsRender on.

What happens if a page returns 404 or 500?

The page is still saved, and fetch.httpStatusCode shows the status. Check it before you use the content.

Why is one of my URLs missing from the results?

Invalid URLs, meaning not http or https, containing a username or password, or longer than 8,192 characters, are skipped and logged as a warning. Duplicates are fetched once. A URL that can't be loaded at all is logged as an error.

Can Web Fetch fetch PDFs?

Yes. PDFs come back as Markdown by default.

Do I need proxies?

No. There's nothing to set up and no proxy costs; the connection and bot protection are handled on our side.

Does Web Fetch log in to sites?

No. The input has no login or cookie fields, and URLs containing a username or password are skipped with a warning. It fetches public pages.

Is Web Fetch an alternative to Firecrawl, Jina Reader or a web unblocker API?

For fetching a known list of pages, including protected ones, and converting them to Markdown, yes, at $0.40 per 1,000 pages. It doesn't crawl whole sites or extract structured data with an LLM.

Can I use Web Fetch from Claude, Cursor or another MCP client?

Yes. Add https://mcp.apify.com/?tools=epctex/web-fetch as a remote MCP server. startUrls is the only required field. Send a few URLs at a time when the agent needs the answer in the same turn.

Can Web Fetch crawl a whole website?

No. It fetches exactly the URLs you give it and doesn't follow links. To work through a site, run it once with the links format, then run it again on the URLs you need.

How do I convert a long list of URLs to Markdown?

Paste the whole list into startUrls and leave formats empty or set it to ["markdown"]. Duplicates are removed automatically, and every page lands in one dataset you can export or read through the API.

Is fetched content safe to pass to a model?

Treat it as untrusted input. A web page can contain text written to look like instructions to an AI. Keep fetched content separate from your own instructions in the prompt, and don't let a model take actions based only on text from a fetched page.

Is it legal to fetch web pages?

Fetching publicly available pages is generally legal, but what you do with the content matters, and some content is protected by copyright, privacy law or a site's terms of service. Collecting emails and phone numbers can be subject to data protection law such as the GDPR. If you're unsure, get legal advice.

Is there a free plan?

Apify Free plan users can try Web Fetch with limited usage. A paid Apify plan removes the limit.

---

Support

Questions, bugs or feature requests: open an issue on the Actor's Issues tab, or email business@epctex.com. For custom integrations, reach us through the chat on epctex.com.