# Web Fetch API — Fetch Any URL as HTML, Text, Markdown or JSON (`highbrow_fame/web-fetch-api`) Actor

Bulk HTTP fetch for agents: returns status, headers and body (raw HTML, text, Markdown or parsed JSON), follows redirects, optional browser rendering. Unofficial Web Fetch API.

- **URL**: https://apify.com/highbrow_fame/web-fetch-api.md
- **Developed by:** [yestrue](https://apify.com/highbrow_fame) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.30 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web Fetch API — Fetch Any URL as HTML, Text, Markdown or JSON

Bulk HTTP fetch built for AI agents and pipelines: give it a list of URLs and get back, for each one, the HTTP status, the response headers and the body — raw HTML, plain text, Markdown or parsed JSON. Follows redirects, supports GET/POST/HEAD and custom headers, and can render JavaScript pages in a headless browser when you ask it to.

**Try it in 1 minute:**

1. Press **Start** — the prefilled input fetches example.com, a public JSON API and a Wikipedia article.
2. Wait a few seconds.
3. Open the **Output** tab: one record per URL with `httpStatus`, `headers`, `contentType`, `body` (the JSON answer comes back parsed, the HTML pages raw) plus title, language and word count for HTML pages.

Price: $0.30 per 1,000 pages. URLs that cannot be fetched (404, timeouts, refused requests, blocked by robots.txt) are free.

**Use it to:**

- Give an AI agent a reliable "fetch this URL" tool through the Apify MCP server or the API — one call, a clean JSON record back.
- Pull hundreds of pages or API endpoints in one batch instead of writing retry and timeout logic yourself.
- Convert HTML pages to Markdown or plain text on the fly (`format: markdown`) for LLM prompts and RAG.
- Check status codes, redirects and response headers of a URL list (monitoring, link audits, SEO checks).
- Read JSON APIs and feeds into a dataset you can export as CSV or Excel.

### How it works

Every URL is requested with browser-like default headers (override any of them), a per-request timeout and automatic redirect following. The answer is kept whole: status code, headers, final URL, size and duration. The body is delivered in the format you choose — `auto` parses JSON content types, keeps HTML raw and returns everything else as text; `markdown` and `text` run an HTML page through Mozilla Readability (main content only, unless you turn that off) and Turndown. A 5xx or 429 answer is retried once through the Apify proxy; a network error too. Binary content (images, PDFs, archives) is reported with its size and content type but not delivered as a body.

**Use a browser** renders GET requests in headless Chromium first, for pages that build their content with JavaScript. It is slower and costs more compute per page, so leave it off unless a plain fetch comes back without the content you need.

`robots.txt` is respected by default: a URL the site disallows for bots is skipped with a free `ROBOTS_DISALLOWED` record. Turn the check off for URLs you have the right to fetch, such as your own API.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` | — | Addresses to fetch, one per line (required) |
| `method` | `GET` | GET, POST, HEAD, PUT, PATCH, DELETE or OPTIONS |
| `headers` | `{}` | Extra request headers (JSON object); they replace defaults of the same name |
| `format` | `auto` | `auto`, `html`, `text`, `markdown` or `json` |
| `useBrowser` | `false` | Render GET requests in headless Chromium |
| `body` | — | Request body for POST/PUT/PATCH |
| `followRedirects` | `true` | Follow 3xx and report the final URL |
| `timeoutSecs` | `30` | Per-request timeout |
| `respectRobotsTxt` | `true` | Skip URLs the site's robots.txt disallows |
| `mainContentOnly` | `true` | `text`/`markdown`: Readability main content only |
| `includeHeaders` | `true` | Deliver response headers |
| `maxBodyLength` | `0` | Cut text bodies after N characters |
| `maxConcurrency`, `proxyConfiguration` | `5`, Apify proxy | Speed and retry proxy |

### Output

| Field | What it is |
|---|---|
| `url`, `finalUrl`, `redirected`, `method` | what was requested and where it ended |
| `httpStatus`, `headers`, `contentType`, `bytes`, `durationMs` | the answer |
| `format`, `body`, `formatError` | the body as html / text / markdown / parsed json / binary (null) |
| `title`, `description`, `language`, `canonicalUrl`, `wordCount`, `linksCount` | HTML pages only |
| `renderedWithBrowser`, `truncated` | how the record was made |
| `input`, `status`, `error`, `scrapedAt` | what you gave, `OK` or the reason, and when |

Status codes: `OK` (any 2xx or final 3xx answer), `NOT_FOUND` (404/410), `ROBOTS_DISALLOWED`, `INVALID_INPUT`, `FAILED` (network error, timeout, other 4xx, 5xx after retries). Only `OK` records are charged.

#### Example

```json
{
  "input": "https://jsonplaceholder.typicode.com/posts/1",
  "url": "https://jsonplaceholder.typicode.com/posts/1",
  "finalUrl": "https://jsonplaceholder.typicode.com/posts/1",
  "redirected": false,
  "method": "GET",
  "httpStatus": 200,
  "contentType": "application/json",
  "headers": { "content-type": "application/json; charset=utf-8", "cache-control": "max-age=43200" },
  "format": "json",
  "body": { "userId": 1, "id": 1, "title": "sunt aut facere repellat provident", "body": "quia et suscipit..." },
  "bytes": 292,
  "durationMs": 184,
  "renderedWithBrowser": false,
  "status": "OK",
  "scrapedAt": "2026-10-19T12:00:00.000Z"
}
```

### FAQ

**Is this an official API?** No, it's an unofficial API built on public pages.

**Does it log in, solve captchas or get around bot walls?** No. It fetches public URLs only. A 401 or 403 answer is reported as `FAILED` with the status, free of charge.

**How is `markdown` different from `html`?** `html` is the page as the server sent it. `markdown` keeps only the main content (Readability) and converts it to Markdown with headings, lists, links and tables — a fraction of the tokens for an LLM.

**Why did a page come back without the content I see in my browser?** The site renders it with JavaScript. Turn on **Use a browser**.

**Can I send POST requests with a JSON body?** Yes: set `method` to POST, put the JSON text in `body` and add `{"Content-Type": "application/json"}` to `headers`.

**Like it?** A short review on the Store page helps other people find this Actor. Something missing or broken? Tell us on the Issues tab — we read every one.

### Related Actors

- [Website Tech Stack Detector](https://apify.com/highbrow_fame/website-tech-stack) — CMS, frameworks and analytics of any site
- [Sitemap Extractor & Scraper](https://apify.com/highbrow_fame/sitemap-url-extractor) — every URL, date and image from a sitemap
- [Domain Checker](https://apify.com/highbrow_fame/domain-whois-dns-ssl) — WHOIS, DNS, SSL and expiry dates in bulk
- [URL to Markdown API](https://apify.com/highbrow_fame/url-to-markdown-api) — clean Markdown from any URL for LLMs
- [Markdown Crawler API](https://apify.com/highbrow_fame/markdown-site-crawler) — crawl a whole website to Markdown for RAG

# Actor input Schema

## `urls` (type: `array`):

Addresses to fetch, one per line: web pages, JSON APIs, feeds, text files. Each becomes one record with status, headers and body.

## `method` (type: `string`):

HTTP method for every request.

## `headers` (type: `object`):

Extra request headers as a JSON object, for example {"Accept": "application/json", "Authorization": "Bearer ..."}. They replace the defaults of the same name (a browser-like User-Agent and Accept are sent by default).

## `format` (type: `string`):

How to deliver the body. `auto`: JSON answers parsed, HTML kept raw, everything else as text. `markdown` and `text` turn an HTML page into its main content (Readability); `json` parses the body.

## `useBrowser` (type: `boolean`):

Render GET requests in headless Chromium before reading the page. Needed only for pages that build their content with JavaScript; slower and more expensive than the default plain fetch.

## `body` (type: `string`):

Body for POST, PUT and PATCH requests (a string; JSON text is sent as is — set a Content-Type header if the server needs one).

## `followRedirects` (type: `boolean`):

Follow 3xx redirects and report the final URL. Off: the redirect answer itself is delivered.

## `timeoutSecs` (type: `integer`):

Give up on a request after this long.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs the site's robots.txt disallows for bots (a free ROBOTS_DISALLOWED record). Turn off only for URLs you have the right to read, for example your own API.

## `waitUntil` (type: `string`):

With "Use a browser": when the page counts as loaded. `load` is the safe default; `networkidle` waits for background requests to stop.

## `mainContentOnly` (type: `boolean`):

For the `text` and `markdown` formats: keep the main content (Readability) and drop navigation, footer and sidebars. Off: the whole page.

## `includeHeaders` (type: `boolean`):

Deliver the response headers as an object in `headers`.

## `maxBodyLength` (type: `integer`):

Cut a text body after this many characters (`truncated: true` is set). 0: no limit (10 MB hard limit).

## `maxConcurrency` (type: `integer`):

How many requests to make at the same time.

## `proxyConfiguration` (type: `object`):

Used only to try again after a network error or a 5xx answer: the first try goes out directly.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://jsonplaceholder.typicode.com/posts/1",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "method": "GET",
  "headers": {},
  "format": "auto",
  "useBrowser": false,
  "followRedirects": true,
  "timeoutSecs": 30,
  "respectRobotsTxt": true,
  "waitUntil": "load",
  "mainContentOnly": true,
  "includeHeaders": true,
  "maxBodyLength": 0,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Every result as a JSON record, with a status for each input.

## `resultsCsv` (type: `string`):

The same records as CSV, for a spreadsheet.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com",
        "https://jsonplaceholder.typicode.com/posts/1",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ],
    "headers": {},
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("highbrow_fame/web-fetch-api").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://example.com",
        "https://jsonplaceholder.typicode.com/posts/1",
        "https://en.wikipedia.org/wiki/Web_scraping",
    ],
    "headers": {},
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("highbrow_fame/web-fetch-api").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com",
    "https://jsonplaceholder.typicode.com/posts/1",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "headers": {},
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call highbrow_fame/web-fetch-api --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,highbrow_fame/web-fetch-api"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/COs6zmIOuWDItQuP1/builds/opbVnqC9iCkUVCVFb/openapi.json