# Smart Fetch - URL to Markdown from US, Singapore, EU or Taiwan (`tidytools/smart-fetch`) Actor

Fetch any public URL as clean Markdown, HTML and JSON metadata. Pick the region it is fetched from (US, Singapore, EU, Taiwan residential) or let Auto switch region when a site blocks. JavaScript rendering, robots.txt respected, failed pages free. From $1/1,000 pages.

- **URL**: https://apify.com/tidytools/smart-fetch.md
- **Developed by:** [Yukai Lin](https://apify.com/tidytools) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page (http)s

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Smart Fetch do?

Give it URLs and get each page back as **clean Markdown**, optionally the **HTML**, and **JSON metadata** (title, description, language, canonical URL, content type, size, Last-Modified, ETag, X-Robots-Tag, links). PDF, Word and Excel files are converted to Markdown too.

**Input:** a list of URLs (full URLs or bare domains like `example.com`). **Output:** one row per URL with the Markdown, optional HTML, metadata, the region and mode that delivered it, and what was charged. **Price:** **$1.00 per 1,000 pages** fetched over HTTP from Cloudflare, **$3.00 per 1,000** for pages rendered in a browser, and **+$2.00 per 1,000** when the page is delivered from our US, Singapore, EU or Taiwan network (so $3.00 from a region you choose). No start fee; failed, blocked and invalid URLs are free.

What makes it different: **you choose where the request comes from.**

| Region | Network |
|---|---|
| `auto` (default) | Cloudflare HTTP first; a JavaScript shell goes straight to Browser Rendering. If the HTTP page is blocked, the US network is tried before a browser (Singapore, London and Taiwan HTTP are used when you choose them) |
| `cf` | Cloudflare's network only (cheapest, no region fee) |
| `us` | Phoenix, United States |
| `sg` | Singapore |
| `eu` | London, United Kingdom |
| `tw` | Taiwan, residential internet line |

Pages that need JavaScript are rendered in a real browser (Cloudflare Browser Rendering) when you choose **Page fetching: Auto** or **Browser**. In Auto, a short JavaScript shell returned by Cloudflare HTTP goes straight to the browser, before US HTTP. If Cloudflare's browsers are busy, return 429, or reach our 9-hour UTC monthly usage cap, our Chrome pool in Taiwan renders the page instead (same page-browser price; `browserVia: "chrome-pool"`). A page refused by Cloudflare's browser can also use the pool.

**Error pages are not pages.** A site can answer "200 OK" with a "page not found" notice, an access-denied or "not available in your region" message, a bot-check or "unsupported browser" screen, a cookie-consent wall or a redirect to its login page. Smart Fetch recognises these, tries another network, and if nothing better comes back reports the page as failed (`not_found`, `blocked`, `login_required`, `error_page`) without charging it. The Markdown is also cleaned of leftovers such as stray `<!DOCTYPE>` lines and inline JSON configuration blobs.

**Honest by design.** Every request says who it is: the user agent is always `Mozilla/5.0 (compatible; TidyToolsSmartFetch/1.0; +https://apify.com/tidytools/smart-fetch)` (in browser mode, the browser's own user agent plus `TidyToolsSmartFetch/1.0`). When a site blocks one network, Smart Fetch only tries another **region**; it never changes its identity, never pretends to be a regular browser, does not solve CAPTCHAs or bot checks, does not log in and sends no cookies. `robots.txt` is respected by default. Pages it cannot get are reported with a reason and **are not charged**.

### Use cases

- **AI agents and RAG**: turn a URL into Markdown an LLM can read, with the page's metadata.
- **Regional price and storefront checks**: see what a page returns from four fixed vantage points: Phoenix (US), Singapore, London (UK) and Taiwan. In our 3 October test of 50 news, shop and pricing pages, 20 (40%) showed region-specific content: Spotify Premium at US $12.99 / S$11.98 / £12.99 / NT$168, Slack, Notion and Grammarly in GBP only from London, AliExpress in S$, GBP and NT$, country stores on Kayak and Nike. News front pages mostly look the same everywhere. These are four fixed locations, not "any country".
- **Fallback fetcher**: when your own fetcher is refused from a data center, try other networks without proxies.
- **Monitoring and archiving**: Last-Modified, ETag and the Markdown text for change detection.

### How much does it cost?

| Event | Price | When |
|---|---|---|
| Page (HTTP) | **$1.00 / 1,000 pages** | a page fetched over plain HTTP and returned |
| Page (browser) | **$3.00 / 1,000 pages** | a page rendered in a real browser and returned (instead of the HTTP page) |
| Region fee | **+$2.00 / 1,000 pages** | added when the page was delivered from our US, Singapore, EU or Taiwan network instead of Cloudflare |

**Examples:** 1,000 pages fetched from Cloudflare = $1.00. 1,000 pages from the Taiwan residential line = $1.00 + $2.00 = $3.00. In Auto mode a page costs $1.00 per 1,000 when Cloudflare gets it, and at most $3.00 per 1,000 when another region or the browser is needed (never both: a browser page is never charged the region fee, also when our Taiwan Chrome pool rendered it).

**No start fee**, no proxy costs, no platform usage on top.

#### Control your cost

- **Charged:** only pages returned with `success: true` (`charged: true`; `chargedEvents` lists the events, `priceUsd` the amount).
- **Free** (`success: false`, `charged: false`, the `error` ends with "(not charged)"): invalid lines (`invalid_input`), URLs that are the same page (fetched and charged once, the other spellings are listed in `inputs`: letter case of scheme and domain, a missing `/` after the domain and a `#fragment` make no difference; a different path or query string is a different page), pages that are not web pages (JSON, images and other unsupported content types: `unsupported_content`), pages blocked on every network tried (`blocked`, `rate_limited`), pages that need a login (`login_required`), `not_found`, error pages served as "200 OK" (`error_page` and the types above), pages disallowed by robots.txt (`robots_disallowed`), empty pages (`empty_page`), timeouts and network errors.
- **Choose the cheap path:** region `cf` with Page fetching `HTTP` never costs more than $1.00 / 1,000. In Auto, a short static page (under about 400 visible characters by default, such as example.com) may also be treated like a JavaScript shell and re-rendered in a browser, costing $3.00 / 1,000 instead of $1.00. This threshold can be adjusted by the service.
- **Before it starts**, the run logs its worst-case cost (pages × $0.003) and compares it with your **maximum charge per run**. When the maximum is reached the run stops cleanly: `SUMMARY.status` is `LIMIT_REACHED` and `SUMMARY.notProcessed` lists what was not fetched.

### How to use it

1. Paste your URLs into **URLs**, one per line (full URLs or bare domains). A bad line never stops the run: it becomes a row with `errorType: "invalid_input"` (not charged). Private, local and reserved addresses (`localhost`, `192.168.x.x`, `.local`, `.internal`, ...) and URLs with a user name or password are refused the same way. The same goes for a wrong setting value (for example `region: "mars"`; letter case and spaces are ignored, so `"US"` works), an impossible combination (Browser with a region other than Auto or Cloudflare) or an empty list: one free error row naming the setting, the run ends normally and nothing is charged (an unknown setting is never silently replaced by Auto).
2. Pick **Fetch from (region)** and **Page fetching**, or keep Auto.
3. Click **Start**, then export the results as JSON, CSV or Excel, or call the Actor from the API.

#### Input example

This is the input the form is prefilled with; a run with it takes about a minute and costs only cents:

```json
{
    "urls": [
        "https://www.python.org/about/",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ]
}
```

#### Output example

Real result (2 October 2026) with region `cf` and Page fetching `Auto`: Cloudflare's plain request was refused (HTTP 403), so the page was rendered in the browser and charged as one browser page.

```json
{
    "input": "https://www.python.org/about/",
    "url": "https://www.python.org/about/",
    "finalUrl": "https://www.python.org/about/",
    "success": true,
    "httpStatus": 200,
    "title": "About Python™ | Python.org",
    "description": "The official home of the Python Programming Language",
    "lang": "en",
    "markdown": "---\ndescription: The official home of the Python Programming Language\ntitle: Welcome to Python.org\n...",
    "markdownChars": 4815,
    "region": "cf",
    "regionLocation": "Cloudflare edge (nearest data center)",
    "mode": "browser",
    "metadata": { "title": "About Python™ | Python.org", "contentType": "text/html; charset=utf-8", "bytes": 51593, "lastModified": null, "etag": null, "xRobotsTag": null },
    "robots": "unavailable",
    "attempts": ["cf/http: blocked (HTTP 403)", "cf/browser: ok (HTTP 200)"],
    "chargedEvents": ["page-browser"],
    "priceUsd": 0.003,
    "charged": true,
    "userAgent": "Mozilla/5.0 (compatible; TidyToolsSmartFetch/1.0; +https://apify.com/tidytools/smart-fetch)"
}
```

`attempts` shows every network and mode that was tried, in order, and why it did not deliver. A `SUMMARY` record has the run `status`, pages per region and mode, failures by `errorType` and the cost plan.

### Benchmark (real end-to-end run)

We froze a sample of 300 popular websites (a Tranco sample across 12 categories, social networks excluded, robots.txt allowing our user agent) and sent every home page once through the production fetch chain on 2-3 October 2026, with the same honest user agent throughout. Home pages are usually the most protected pages of a site.

| Auto, 300 home pages | Result |
|---|---|
| Pages returned | **235 / 300 (78.3%)** |
| Pages returned with at least 200 characters of text | 226 / 300 (75.3%) |
| Error pages returned as a success (manual review) | **0** (7 before the error-page check was added) |
| Time per returned page, median / 90th percentile | 2.1 s / 7.8 s |
| Delivered by | Cloudflare 198, US 18, browser 16, Chrome pool 2, Singapore 1 |

By site type: entertainment 94%, news 90%, education 88%, tech 84%, government 79%, finance 71%, ecommerce 68%, travel 54%, telecom 36%.

Where the run starts matters. In this run Cloudflare's network left from Singapore. From Cloudflare's US edge (where runs on Apify start) Cloudflare alone returned 182 instead of 201 home pages, and the whole chain at least 213 / 300 (71%) at a moment when Cloudflare's browsers were fully busy. A second full run on 3 October 2026 through the live chain returned 212 / 300 (70.7%), 193 with at least 200 characters of real text: 23 sites had started refusing all data-center networks in one day. **Expect roughly 70-78% on home pages, and expect the number to drift as sites change their blocking.** Deep pages do better: 59 of 66 article and document pages (89%) in our deep-page sample.

**Regions.** We fetched 100 of the sites from all five networks at the same time. 8 served a clearly regional version (for example PayPal, SHEIN, Samsung, Xbox and Uber: a different title, language, currency or domain) and 26 were reachable from some networks but refused from others. Most sites serve the same page everywhere: choose a region when you need to know what that region sees, and let Auto handle refusals. With an honest user agent a residential line is **not** a way around bot protection; sites that refuse every network stay refused and are reported as `blocked`, not charged.

**For RAG.** With **Main content only** on, link text fell from 16% to 11% of the text and menu-like lines from 9% to 6% (median, 66 deep pages), while the median page kept 2,500 characters of text.

### Use it from code

```bash
curl -X POST "https://api.apify.com/v2/acts/tidytools~smart-fetch/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://en.wikipedia.org/wiki/Web_scraping"],"region":"auto"}'
```

**With AI agents (MCP):** connect Apify's MCP server (`https://mcp.apify.com?tools=tidytools/smart-fetch`) and ask, for example, "Fetch this page from Singapore as Markdown".

### Advanced settings

- **Respect robots.txt** (default on): pages disallowed for `TidyToolsSmartFetch` (or `*`) are skipped and not charged. When robots.txt cannot be read, the page is fetched and `robots` says `unavailable`.
- **Try other regions when blocked**: on by default for Auto, off for a chosen region (so a regional check never silently comes from somewhere else).
- **Remove navigation, header and footer** (default on), **Main content only**, **Browser: wait until**, **Time limit per network** (default 30 s), **Parallel URLs** (default 5).

### Token-saving output

The Markdown can be shaped to cost the reader (an LLM or a RAG index) fewer tokens. These settings only shape what the Actor returns; the page is still fetched and charged once, exactly as before.

- **Markdown token limit** (`maxTokens`): return at most that many tokens (about 4 characters per token for Latin text, one per CJK character). The Markdown is cut at a blank-line boundary, never mid-block; when there is more, the row also carries `nextStartIndex`, the character offset to pass as `startIndex` in a later run to continue from where this one stopped.
- **Start at character** (`startIndex`): return the Markdown from that character offset onward (a `nextStartIndex` you got earlier). `0` is the start of the page.
- **Links in Markdown** (`links`): `inline` keeps `[text](url)`; `footnotes` moves the URLs to numbered `[^n]` notes at the end; `none` keeps the link text only.
- **Images in Markdown** (`images`): `all` keeps `![alt](url)`; `alt` keeps the alt text only; `none` drops the images.
- **Keep blocks matching a query** (`query`, with **Blocks kept for a query** `queryMaxBlocks`, default 5): ranks the Markdown blocks with BM25 against the query and keeps only the most relevant blocks, in the original page order. No AI model is used; when no block matches, the Markdown is empty.

### Limitations

- Public pages only: no logins, no cookies, no CAPTCHA or bot-check solving. Sites behind such checks are reported as `blocked` (not charged).
- Browser mode runs on Cloudflare (region Auto or Cloudflare; in Auto, with our Taiwan Chrome pool as the fallback when Cloudflare Browser Rendering is busy or reaches the 9-hour UTC monthly cap) or on our Taiwan residential Chrome pool (region Taiwan); regions `us`, `sg` and `eu` fetch over plain HTTP and do not support Browser mode. A browser page is one `page-browser` event on every route.
- The Taiwan line is a home internet connection with limited capacity: very large runs on `tw` can be slower, and a region that is temporarily unavailable is reported as `region_unavailable` (not charged).
- Up to 10 MB per page from regional networks, 15 MB from Cloudflare; HTML output up to 2,000,000 characters.

### FAQ

**Does the Actor collect usage data?** The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

**Is this a proxy or an unblocker?** No. It is a fetcher that runs from several networks you can choose. It always identifies itself and never tries to look like a person's browser. If a site blocks all of our networks, you get `blocked` and pay nothing.

**What is charged when Auto switches region?** The page plus the region fee ($0.003 in total) only if the page was delivered over HTTP from our US or Taiwan network in Auto (or from the region you chose, including Singapore and Europe); pages Cloudflare gets cost $0.001, browser pages $0.003.

**What happens when a site blocks Smart Fetch or answers 403 or 429?** In Auto, a blocked HTTP page tries the next configured HTTP network and then a browser; a Cloudflare HTTP JavaScript shell goes straight to the browser. Browser Rendering capacity 429 uses the Taiwan Chrome pool. If every route is refused you get one row with `errorType` `blocked` or `rate_limited`, the list of `attempts`, and no charge. It does not use anti-detection tricks.

**What if I enter a wrong URL or setting?** The run does not fail: you get a row with `errorType: "invalid_input"` that says what is wrong (for a wrong setting, which one and the allowed values), and you are not charged. If one setting is wrong, nothing is fetched at all, so fix it and start again.

**What if I list the same page twice?** It is fetched and charged once (`https://example.com`, `HTTPS://EXAMPLE.COM/` and `example.com/#top` are one page); `SUMMARY.duplicatesMerged` counts the merged lines and the row's `inputs` lists the spellings.

**Can I get only metadata?** Yes: turn off **Markdown**. The page is still charged as one page.

### Support

Open an issue in the **Issues** tab with the URL and the region. Issues are checked regularly.

# Actor input Schema

## `urls` (type: `array`):

Pages to fetch, one per line: full URLs or bare domains like example.com. Up to 50,000 per run. Invalid lines become a result row with errorType "invalid_input" (not charged) instead of failing the run.

## `region` (type: `string`):

Where the request leaves from. Auto (recommended): Cloudflare HTTP first; a JavaScript shell goes straight to a browser, while a blocked HTTP page tries our US server before a browser. A chosen region fetches only from there (turn on "Try other regions" to fall back). HTTP pages delivered from US, Singapore, EU or Taiwan add the region fee; browser pages do not.

## `mode` (type: `string`):

Auto: plain HTTP first; a short JavaScript shell goes straight to a browser. With region Auto, our Taiwan Chrome pool takes over when Cloudflare Browser Rendering is busy or at the 9-hour monthly cap. HTTP: never a browser. Browser: always render, from Cloudflare (Auto or Cloudflare) or our Taiwan residential Chrome (Taiwan); US, Singapore and Europe do not support Browser. A page is charged as a browser page only when the browser result is used.

## `includeMarkdown` (type: `boolean`):

Clean Markdown of the page (PDF, Word and Excel files too) with a YAML header (title, description, image).

## `includeHtml` (type: `boolean`):

The page's HTML as received (or as rendered in browser mode), up to 2,000,000 characters.

## `includeLinks` (type: `boolean`):

All absolute links on the page in metadata.links.

## `respectRobots` (type: `boolean`):

Skip pages that the site's robots.txt disallows for our user agent TidyToolsSmartFetch (errorType robots_disallowed, not charged).

## `tryOtherRegions` (type: `boolean`):

Leave unset for the default: on for Auto, off for a chosen region. On: when the chosen region is refused, Cloudflare, US and Taiwan are tried too.

## `removeBoilerplate` (type: `boolean`):

Leave site navigation, cookie banners, headers and footers out of the Markdown.

## `mainContent` (type: `boolean`):

Convert only the page's main element (article / main) when it is unambiguous. Falls back to the whole page when that is nearly empty.

## `waitUntil` (type: `string`):

When a browser page counts as loaded.

## `timeoutSecs` (type: `integer`):

How long one attempt may take before the next region is tried.

## `maxConcurrency` (type: `integer`):

How many URLs are fetched at the same time.

## `maxTokens` (type: `integer`):

Return at most this many tokens of Markdown (about 4 characters per token for Latin text, one per CJK character). The Markdown is cut at a blank-line boundary so a block is never split in half; the row's nextStartIndex is where to continue (pass it as startIndex in the next run). Leave empty for the whole page.

## `startIndex` (type: `integer`):

Return the Markdown from this character offset onward (a page you got earlier via nextStartIndex). 0 means the start of the page.

## `links` (type: `string`):

How links appear in the Markdown. Inline keeps [text](url). Footnotes moves the URLs to numbered \[^n] notes at the end. None keeps the link text only.

## `images` (type: `string`):

How images appear in the Markdown. Keep images leaves ![alt](url) as is; Alt text only keeps the image's alt text; None drops the images.

## `query` (type: `string`):

Rank the Markdown blocks with BM25 against this query and keep only the most relevant ones (up to Blocks kept for a query), in the original page order. No AI model is used. Leave empty to keep the whole page.

## `queryMaxBlocks` (type: `integer`):

How many blocks the query keeps (the most relevant ones, in page order).

## `startUrls` (type: `array`):

Upload or link .txt/.csv files of URLs (every http(s) URL in a file is read). Full URLs only here.

## Actor input object example

```json
{
  "urls": [
    "https://www.python.org/about/",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ],
  "region": "auto",
  "mode": "auto",
  "includeMarkdown": true,
  "includeHtml": false,
  "includeLinks": false,
  "respectRobots": true,
  "removeBoilerplate": true,
  "mainContent": false,
  "waitUntil": "load",
  "timeoutSecs": 30,
  "maxConcurrency": 5,
  "startIndex": 0,
  "links": "inline",
  "images": "all",
  "queryMaxBlocks": 5
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `full` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.python.org/about/",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidytools/smart-fetch").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://www.python.org/about/",
        "https://en.wikipedia.org/wiki/Web_scraping",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("tidytools/smart-fetch").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.python.org/about/",
    "https://en.wikipedia.org/wiki/Web_scraping"
  ]
}' |
apify call tidytools/smart-fetch --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidytools/smart-fetch"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ZwrMww9WkNBFqS9gl/builds/i7E3uPDgO4Q7iR0N4/openapi.json