# Web Fetch (`apify/web-fetch`) Actor Real-time web fetch API that turns any URL into clean Markdown, HTML, or links, with automatic JavaScript rendering and anti-bot protection. - **URL**: https://apify.com/apify/web-fetch.md - **Developed by:** [Apify](https://apify.com/apify) (Apify) - **Categories:** AI, Automation, Developer tools - **Stats:** 8 total users, 3 monthly users, 99.6% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.50 / 1,000 fetches This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README **Turn any URL into clean, LLM-ready Markdown, HTML, or links** with a single API call. Web Fetch automatically **bypasses anti-bot protection, rate limits, and browser-based challenges**, then strips out navigation, ads, and boilerplate so you get clean content back - no browser automation infrastructure, no proxy management, no CAPTCHA-solving code to maintain yourself. Web Fetch is the unblocker your AI agent needs to reach the web. Because it runs as an always-on [Standby Actor](https://docs.apify.com/platform/actors/development/programming-interface/standby) on the Apify platform, there's no run to start and no result to poll for - you call one HTTP endpoint and get your answer back in the response, like calling any other API. ### Why use Web Fetch to scrape websites? - **Feed LLMs and RAG pipelines clean content.** Markdown is far cheaper (in tokens) and easier for a model to parse than raw HTML, making it the ideal input for retrieval-augmented generation, AI agents, and fine-tuning datasets. - **Get past anti-bot walls without building your own bypass logic.** Web Fetch automatically handles common blocking mechanisms, including browser-based challenges, so you don't need to reverse-engineer them yourself. - **No infrastructure to run or scale.** Because it's a Standby Actor, there's no cold start for a new run and no server to provision - just call the endpoint and get a response. - **Automate anywhere.** Call it directly via HTTP, through the [Apify API](https://docs.apify.com/api/v2) client libraries for Python and JavaScript, or wire it into [Apify integrations](https://docs.apify.com/integrations) like Make, Zapier, and n8n. ### How to use Web Fetch 1. Authenticate with your [Apify API token](https://console.apify.com/settings/integrations), either as a `token` query parameter or as an `Authorization: Bearer ` header (the header is more secure since URLs can end up in logs or browser history) - the platform uses it to identify and bill the calling user for each successful fetch. 2. Send a GET or POST request to the Actor's Standby URL, `https://web-fetch.apify.actor`, with the URL you want to convert - both behave identically: ```bash curl 'https://web-fetch.apify.actor/?url=https://apify.com&formats=markdown,links&token=***' ``` ```bash curl -X POST 'https://web-fetch.apify.actor/' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer ***' \ -d '{ "url": "https://apify.com", "formats": ["markdown", "links"] }' ``` 3. Read the `text` / `markdown` / `html` / `raw` / `links` / `fetch` / `metadata` fields straight out of the JSON response - no polling, no separate results endpoint. Prefer a regular Actor run instead? Fill in the same fields on the [Input tab](https://console.apify.com/actors) and hit **Start** - Web Fetch performs the single fetch, saves the result to the run's dataset, and exits, just like any other Actor. Running the Actor in standard mode gives you access to all of Apify's native integrations. It also lets you schedule Web Fetch to run on a schedule. [Scheduling your Actor](https://www.youtube.com/watch?v=1jI7WcVQmwM) #### Input The fields below are exactly what the Standby URL accepts, either as GET query params or as a POST JSON body; the same fields (minus `unwrap`, which is Standby/MCP only) also appear on the [Input tab](https://console.apify.com/actors) for a regular Actor run. | Field | Description | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | The URL of a web page or resource to fetch (required). | | `formats` | Which outputs to return, case-insensitive: `text`, `markdown`, `html`, `raw`, `links`. If omitted, a single best-effort format is picked based on the content type (see below). | | `unwrap` | Standby/MCP only - not available (and not needed) for a regular Actor run, which always returns the JSON envelope. Return the first requested format that can actually be produced (or the best-effort default) as a plain HTTP body instead of the JSON envelope; `raw` is the only format that never fails, regardless of content type. Composes with `formats` - not mutually exclusive. Default: `false`. Accepts `true`/`false`, `1`/`0`, case-insensitive. | | `headers` | Extra HTTP headers to send with the request. | ```json { "url": "https://apify.com", "formats": ["markdown", "links"] } ``` The links format will return a list of all links found on the page, without any deduplication. This is a useful output format for building a crawl queue that fetches more content. #### Output Every request (other than `unwrap=true`) returns a JSON envelope with `fetch` (how the page was fetched), `metadata` (what's on the page), and one key per requested format. If `formats` is omitted, a single best-effort format is picked based on the fetched content type, so you never need to know it up front: HTML or PDF defaults to `markdown`, plain text (`text/plain`) defaults to `text`, and any other binary content (images, archives, etc.) defaults to `raw`. `raw` itself always has to be requested explicitly via `formats` - it's the heaviest payload of the bunch, so it's never included unless asked for: ```json { "url": "https://apify.com", "fetch": { "loadedUrl": "https://apify.com", "loadedTime": "2026-07-27T12:41:41.064Z", "httpStatusCode": 200, "contentLengthBytes": 45210, "contentType": "text/html; charset=utf-8" }, "metadata": { "canonicalUrl": "https://apify.com", "title": "Apify: Full-stack web scraping and data extraction platform", "description": "Cloud platform for web scraping, browser automation, AI agents, and data for AI.", "author": null, "keywords": null, "languageCode": "en", "openGraph": [ { "property": "og:title", "content": "Apify" }, { "property": "og:site_name", "content": "Apify" } ], "jsonLd": null, "headers": { "content-type": "text/html; charset=utf-8", "content-length": "45210" } }, "markdown": "# Apify: Full-stack web scraping and data extraction platform\n\n...", "links": ["https://apify.com/pricing", "https://apify.com/store"] } ``` Each requested format is best-effort: if it can't be produced for the fetched resource's content type (e.g. `markdown` for an image), it's returned as `null` rather than failing the whole request. `raw` can be produced for any content type, so requesting `formats: ["raw"]` always succeeds regardless of what the URL returns. Beyond that, if the fetch fails outright (bad input, unreachable site, none of the requested formats could be produced, or a timeout), you'll get a flat error with a stable machine-readable `code` and a matching HTTP status instead: ```json { "code": "UNSUPPORTED_CONTENT_TYPE", "error": "The URL returned a content type (image/jpeg) that cannot be converted to any of the requested formats: markdown. Add \"raw\" to formats (works for any content type) to fetch it as-is." } ``` #### Unwrapped response: `unwrap=true` Setting `unwrap=true` returns a single format as a plain HTTP body instead of the JSON envelope, with an appropriate `Content-Type` (`text/markdown`, `text/html`, `text/plain`, or - for `raw` - the upstream resource's own content type). It composes with `formats` - there's no mutual exclusion: - `unwrap=true` alone → the best-effort format as a plain body (same content-type-based defaulting as above: HTML/PDF → `markdown`, `text/plain` → `text`, other binary → `raw`). - `formats=["markdown"]&unwrap=true` → the Markdown as `text/markdown`. - `formats=["raw"]&unwrap=true` → the raw original body, verbatim (works for any content type, including images and archives). If multiple formats are requested with `unwrap=true`, they're tried in the given order and the first one that can actually be produced for the fetched content type is returned as the body - e.g. `formats=["markdown","raw"]` returns Markdown when possible, or the raw body when it's not. `raw` is treated like any other requested format in that order - it's just the one format that never fails, regardless of content type, so listing it gives you a guaranteed response if everything listed before it comes back empty. ```bash ## Returns the PDF's extracted text as markdown (the best-effort default for PDF) curl 'https://web-fetch.apify.actor/?url=https://example.com/file.pdf&unwrap=true&token=***' ## Returns the same thing, requested explicitly curl 'https://web-fetch.apify.actor/?url=https://example.com/file.pdf&unwrap=true&formats=markdown&token=***' ## Returns the original PDF file, verbatim curl 'https://web-fetch.apify.actor/?url=https://example.com/file.pdf&unwrap=true&formats=raw&token=***' -o file.pdf ``` You only get an `UNSUPPORTED_CONTENT_TYPE` JSON error once every requested format has failed for the fetched content type (e.g. `unwrap=true&formats=markdown` on an image, with no `raw` fallback requested) - add `raw` to `formats` to guarantee a response regardless of content type. Every successful fetch is also saved as an item in the run's dataset - `unwrap=true` requests get the same shape as any other request (`fetch`, `metadata`, and the produced format's field), just with only one format populated - which you can download in JSON, HTML, CSV, or Excel format from the **Output** tab. #### Response | Field | Type | Description | | -------------------------- | -------- | ----------------------------------------------------------------------------------------------- | | `url` | string | The URL you requested. | | `text` | string | Plain-text rendering of the cleaned article content (if requested). | | `markdown` | string | Page content converted to clean Markdown (if requested). | | `html` | string | Cleaned HTML - navigation, footers, and ads stripped, main article content only (if requested). | | `raw` | string | Original raw HTTP response body (if requested): as-is for HTML/text, base64-encoded for binary. | | `links` | string\[] | Absolute URLs of every link found on the page (if requested). | | `fetch.loadedUrl` | string | The final URL, after any redirects. | | `fetch.loadedTime` | string | When the fetch completed, in ISO 8601. | | `fetch.httpStatusCode` | number | HTTP status code returned by the target site. | | `fetch.contentLengthBytes` | number | Size of the response body, in bytes. | | `fetch.contentType` | string | The `Content-Type` returned by the target site. | | `metadata.canonicalUrl` | string | The page's `` URL, or the loaded URL if absent. | | `metadata.title` | string | Page ``. | | `metadata.description` | string | The `<meta name="description">` content, if present. | | `metadata.languageCode` | string | IETF language tag from the page's `<html lang>` attribute. | | `metadata.openGraph` | array | Open Graph, Twitter Card, and related meta tags as `{ property, content }` pairs. | | `metadata.jsonLd` | array | Parsed `<script type="application/ld+json">` blocks, if any. | | `metadata.headers` | object | All HTTP response headers from the target site. | ### Use Web Fetch via MCP Web Fetch also runs a [Model Context Protocol](https://modelcontextprotocol.io) server at `/mcp`, exposing a single `web-fetch` tool with the same parameters and JSON output as the main API. Add it to any MCP-compatible client (Claude Code, Cursor, etc.) pointed at the Actor's Standby URL: ```bash claude mcp add web-fetch https://web-fetch.apify.actor/mcp -t http ``` Or, since this Actor is published on Apify Store, get the tool automatically by adding `apify/web-fetch` through the [Apify MCP server](https://mcp.apify.com/) instead of connecting directly. ### How much does it cost to run Web Fetch? Web Fetch uses [pay-per-event pricing](https://docs.apify.com/platform/actors/publishing/monetize/pay-per-event): you're charged one `fetch` event for every successful request, and nothing for failed requests. Check the Actor's page on Apify Store for the current price per event. There's no separate compute or proxy bill to account for. ### Tips for getting the best results - Request only the `formats` you actually need (e.g. just `markdown`) to keep responses smaller and faster to parse. - For file types other than HTML and PDF (images, archives, etc.), only `formats: ["raw"]` will return anything - `text`/`markdown`/`html`/`links` come back `null` since there's no text or markup to extract; this is also why these content types default to `raw` when `formats` is omitted. - The target URL's response is capped at 10 MB and an overall fetch timeout of 60 seconds (covering both fetching and reading the response) - a response over either limit fails with `UPSTREAM_FETCH_ERROR`/`FETCH_TIMEOUT` instead of being partially processed. ### FAQ #### Does Web Fetch handle sites with anti-bot protection? Yes - it's built to automatically get past common anti-bot mechanisms, rate limits, and browser-based challenges without any extra configuration on your end. #### Does Web Fetch render JavaScript-heavy pages like a browser would? Yes. Web Fetch renders JavaScript, so content that is generated or loaded client-side can be included in the output. #### Does Web Fetch support PDFs? Yes - `text`, `markdown`, and `html` all work for PDF URLs, using extracted text (no layout, images, or tables); `raw` returns the original PDF bytes, base64-encoded. Link extraction (`links`) is not yet supported for PDFs. #### Is web scraping legal? Scraping publicly available, non-personal data is generally legal, but what you do with the data afterward matters, and some content is protected by copyright or a site's Terms of Service. If you're unsure, seek legal advice. Read more in [this blog post](https://blog.apify.com/is-web-scraping-legal/). #### Why not use Claude native web fetch? Native LLM web fetches often rely on limited fetch infrastructure that can't reliably get past anti-bot walls or JavaScript-heavy pages at scale. Web Fetch is built for that job. #### What is a web fetch tool? A web fetch tool retrieves content from a URL on behalf of an AI agent or LLM app, usually converting it to a format the model can parse like Markdown. #### Why convert web content to Markdown? Markdown is the perfect format to feed an LLM. It's lighter than HTML but preserves text structure like titles and headings. Using Markdown instead of HTML lowers your token usage and your AI costs. ### Something's not working, or I need a custom feature. Please open an issue on the Actor's **Issues** tab with the URL and formats you tried. Custom extraction pipelines (screenshots, structured/LLM extraction, table extraction from PDFs, and more) can be built as a bespoke solution - reach out via [Apify's custom solutions page](https://apify.com/custom-solutions). # Actor input Schema ## `url` (type: `string`): The URL of a web page or resource to fetch. ## `formats` (type: `array`): Which output formats to include in the response (case-insensitive): plain text, Markdown, HTML, the original raw HTTP response body (as-is for textual content, base64-encoded for binary content like PDFs or images), or a list of links found on the page. "raw" works for any content type; the others require textual or PDF content and are set to null otherwise. If omitted, a single best-effort format is picked based on the fetched content type (HTML/PDF -> markdown, plain text -> text, other binary -> raw). ## `headers` (type: `object`): Additional HTTP headers to send with the request, e.g. { "Accept-Language": "de-DE" }. ## Actor input object example ```json { "url": "https://apify.com", "formats": [ "markdown" ] } ``` # Actor output Schema ## `dataset` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '<YOUR_API_TOKEN>' with your token const client = new ApifyClient({ token: '<YOUR_API_TOKEN>', }); // Prepare Actor input const input = { "url": "https://apify.com", "formats": [ "markdown" ] }; // Run the Actor and wait for it to finish const run = await client.actor("apify/web-fetch").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '<YOUR_API_TOKEN>' with your token. client = ApifyClient("<YOUR_API_TOKEN>") # Prepare the Actor input run_input = { "url": "https://apify.com", "formats": ["markdown"], } # Run the Actor and wait for it to finish run = client.actor("apify/web-fetch").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "url": "https://apify.com", "formats": [ "markdown" ] }' | apify call apify/web-fetch --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,apify/web-fetch" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/a6Sqhlnt4touIHTub/builds/8bIR0pB3gC7tWU8ym/openapi.json