Smart Fetch - URL to Markdown from US, Singapore, EU or Taiwan

What does Smart Fetch do?

Give it URLs and get each page back as clean Markdown, optionally the HTML, and JSON metadata (title, description, language, canonical URL, content type, size, Last-Modified, ETag, X-Robots-Tag, links). PDF, Word and Excel files are converted to Markdown too.

Input: a list of URLs (full URLs or bare domains like example.com). Output: one row per URL with the Markdown, optional HTML, metadata, the region and mode that delivered it, and what was charged. Price: $1.00 per 1,000 pages fetched over HTTP from Cloudflare, $3.00 per 1,000 for pages rendered in a browser, and +$2.00 per 1,000 when the page is delivered from our US, Singapore, EU or Taiwan network (so $3.00 from a region you choose). No start fee; failed, blocked and invalid URLs are free.

What makes it different: you choose where the request comes from.

Region	Network
auto (default)	Cloudflare HTTP first; a JavaScript shell goes straight to Browser Rendering. If the HTTP page is blocked, the US network is tried before a browser (Singapore, London and Taiwan HTTP are used when you choose them)
cf	Cloudflare's network only (cheapest, no region fee)
us	Phoenix, United States
sg	Singapore
eu	London, United Kingdom
tw	Taiwan, residential internet line

Pages that need JavaScript are rendered in a real browser (Cloudflare Browser Rendering) when you choose Page fetching: Auto or Browser. In Auto, a short JavaScript shell returned by Cloudflare HTTP goes straight to the browser, before US HTTP. If Cloudflare's browsers are busy, return 429, or reach our 9-hour UTC monthly usage cap, our Chrome pool in Taiwan renders the page instead (same page-browser price; browserVia: "chrome-pool"). A page refused by Cloudflare's browser can also use the pool.

Error pages are not pages. A site can answer "200 OK" with a "page not found" notice, an access-denied or "not available in your region" message, a bot-check or "unsupported browser" screen, a cookie-consent wall or a redirect to its login page. Smart Fetch recognises these, tries another network, and if nothing better comes back reports the page as failed (not_found, blocked, login_required, error_page) without charging it. The Markdown is also cleaned of leftovers such as stray <!DOCTYPE> lines and inline JSON configuration blobs.

Honest by design. Every request says who it is: the user agent is always Mozilla/5.0 (compatible; TidyToolsSmartFetch/1.0; +https://apify.com/tidytools/smart-fetch) (in browser mode, the browser's own user agent plus TidyToolsSmartFetch/1.0). When a site blocks one network, Smart Fetch only tries another region; it never changes its identity, never pretends to be a regular browser, does not solve CAPTCHAs or bot checks, does not log in and sends no cookies. robots.txt is respected by default. Pages it cannot get are reported with a reason and are not charged.

Use cases

• AI agents and RAG: turn a URL into Markdown an LLM can read, with the page's metadata.
• Regional price and storefront checks: see what a page returns from four fixed vantage points: Phoenix (US), Singapore, London (UK) and Taiwan. In our 3 October test of 50 news, shop and pricing pages, 20 (40%) showed region-specific content: Spotify Premium at US $12.99 / S$11.98 / £12.99 / NT$168, Slack, Notion and Grammarly in GBP only from London, AliExpress in S$, GBP and NT$, country stores on Kayak and Nike. News front pages mostly look the same everywhere. These are four fixed locations, not "any country".
• Fallback fetcher: when your own fetcher is refused from a data center, try other networks without proxies.
• Monitoring and archiving: Last-Modified, ETag and the Markdown text for change detection.

How much does it cost?

Event	Price	When
Page (HTTP)	$1.00 / 1,000 pages	a page fetched over plain HTTP and returned
Page (browser)	$3.00 / 1,000 pages	a page rendered in a real browser and returned (instead of the HTTP page)
Region fee	+$2.00 / 1,000 pages	added when the page was delivered from our US, Singapore, EU or Taiwan network instead of Cloudflare

Examples: 1,000 pages fetched from Cloudflare = $1.00. 1,000 pages from the Taiwan residential line = $1.00 + $2.00 = $3.00. In Auto mode a page costs $1.00 per 1,000 when Cloudflare gets it, and at most $3.00 per 1,000 when another region or the browser is needed (never both: a browser page is never charged the region fee, also when our Taiwan Chrome pool rendered it).

No start fee, no proxy costs, no platform usage on top.

Control your cost

• Charged: only pages returned with success: true (charged: true; chargedEvents lists the events, priceUsd the amount).
• Free (success: false, charged: false, the error ends with "(not charged)"): invalid lines (invalid_input), URLs that are the same page (fetched and charged once, the other spellings are listed in inputs: letter case of scheme and domain, a missing / after the domain and a #fragment make no difference; a different path or query string is a different page), pages that are not web pages (JSON, images and other unsupported content types: unsupported_content), pages blocked on every network tried (blocked, rate_limited), pages that need a login (login_required), not_found, error pages served as "200 OK" (error_page and the types above), pages disallowed by robots.txt (robots_disallowed), empty pages (empty_page), timeouts and network errors.
• Choose the cheap path: region cf with Page fetching HTTP never costs more than $1.00 / 1,000. In Auto, a short static page (under about 400 visible characters by default, such as example.com) may also be treated like a JavaScript shell and re-rendered in a browser, costing $3.00 / 1,000 instead of $1.00. This threshold can be adjusted by the service.
• Before it starts, the run logs its worst-case cost (pages × $0.003) and compares it with your maximum charge per run. When the maximum is reached the run stops cleanly: SUMMARY.status is LIMIT_REACHED and SUMMARY.notProcessed lists what was not fetched.

How to use it

1. Paste your URLs into URLs, one per line (full URLs or bare domains). A bad line never stops the run: it becomes a row with errorType: "invalid_input" (not charged). Private, local and reserved addresses (localhost, 192.168.x.x, .local, .internal, ...) and URLs with a user name or password are refused the same way. The same goes for a wrong setting value (for example region: "mars"; letter case and spaces are ignored, so "US" works), an impossible combination (Browser with a region other than Auto or Cloudflare) or an empty list: one free error row naming the setting, the run ends normally and nothing is charged (an unknown setting is never silently replaced by Auto).
2. Pick Fetch from (region) and Page fetching, or keep Auto.
3. Click Start, then export the results as JSON, CSV or Excel, or call the Actor from the API.

Input example

This is the input the form is prefilled with; a run with it takes about a minute and costs only cents:

{
    "urls": [
        "https://www.python.org/about/",
        "https://en.wikipedia.org/wiki/Web_scraping"
    ]
}

Output example

Real result (2 October 2026) with region cf and Page fetching Auto: Cloudflare's plain request was refused (HTTP 403), so the page was rendered in the browser and charged as one browser page.

{
    "input": "https://www.python.org/about/",
    "url": "https://www.python.org/about/",
    "finalUrl": "https://www.python.org/about/",
    "success": true,
    "httpStatus": 200,
    "title": "About Python™ | Python.org",
    "description": "The official home of the Python Programming Language",
    "lang": "en",
    "markdown": "---\ndescription: The official home of the Python Programming Language\ntitle: Welcome to Python.org\n...",
    "markdownChars": 4815,
    "region": "cf",
    "regionLocation": "Cloudflare edge (nearest data center)",
    "mode": "browser",
    "metadata": { "title": "About Python™ | Python.org", "contentType": "text/html; charset=utf-8", "bytes": 51593, "lastModified": null, "etag": null, "xRobotsTag": null },
    "robots": "unavailable",
    "attempts": ["cf/http: blocked (HTTP 403)", "cf/browser: ok (HTTP 200)"],
    "chargedEvents": ["page-browser"],
    "priceUsd": 0.003,
    "charged": true,
    "userAgent": "Mozilla/5.0 (compatible; TidyToolsSmartFetch/1.0; +https://apify.com/tidytools/smart-fetch)"
}

attempts shows every network and mode that was tried, in order, and why it did not deliver. A SUMMARY record has the run status, pages per region and mode, failures by errorType and the cost plan.

Benchmark (real end-to-end run)

We froze a sample of 300 popular websites (a Tranco sample across 12 categories, social networks excluded, robots.txt allowing our user agent) and sent every home page once through the production fetch chain on 2-3 October 2026, with the same honest user agent throughout. Home pages are usually the most protected pages of a site.

Auto, 300 home pages	Result
Pages returned	235 / 300 (78.3%)
Pages returned with at least 200 characters of text	226 / 300 (75.3%)
Error pages returned as a success (manual review)	0 (7 before the error-page check was added)
Time per returned page, median / 90th percentile	2.1 s / 7.8 s
Delivered by	Cloudflare 198, US 18, browser 16, Chrome pool 2, Singapore 1

By site type: entertainment 94%, news 90%, education 88%, tech 84%, government 79%, finance 71%, ecommerce 68%, travel 54%, telecom 36%.

Where the run starts matters. In this run Cloudflare's network left from Singapore. From Cloudflare's US edge (where runs on Apify start) Cloudflare alone returned 182 instead of 201 home pages, and the whole chain at least 213 / 300 (71%) at a moment when Cloudflare's browsers were fully busy. A second full run on 3 October 2026 through the live chain returned 212 / 300 (70.7%), 193 with at least 200 characters of real text: 23 sites had started refusing all data-center networks in one day. Expect roughly 70-78% on home pages, and expect the number to drift as sites change their blocking. Deep pages do better: 59 of 66 article and document pages (89%) in our deep-page sample.

Regions. We fetched 100 of the sites from all five networks at the same time. 8 served a clearly regional version (for example PayPal, SHEIN, Samsung, Xbox and Uber: a different title, language, currency or domain) and 26 were reachable from some networks but refused from others. Most sites serve the same page everywhere: choose a region when you need to know what that region sees, and let Auto handle refusals. With an honest user agent a residential line is not a way around bot protection; sites that refuse every network stay refused and are reported as blocked, not charged.

For RAG. With Main content only on, link text fell from 16% to 11% of the text and menu-like lines from 9% to 6% (median, 66 deep pages), while the median page kept 2,500 characters of text.

Use it from code

curl -X POST "https://api.apify.com/v2/acts/tidytools~smart-fetch/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://en.wikipedia.org/wiki/Web_scraping"],"region":"auto"}'

With AI agents (MCP): connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/smart-fetch) and ask, for example, "Fetch this page from Singapore as Markdown".

Advanced settings

• Respect robots.txt (default on): pages disallowed for TidyToolsSmartFetch (or *) are skipped and not charged. When robots.txt cannot be read, the page is fetched and robots says unavailable.
• Try other regions when blocked: on by default for Auto, off for a chosen region (so a regional check never silently comes from somewhere else).
• Remove navigation, header and footer (default on), Main content only, Browser: wait until, Time limit per network (default 30 s), Parallel URLs (default 5).

Token-saving output

The Markdown can be shaped to cost the reader (an LLM or a RAG index) fewer tokens. These settings only shape what the Actor returns; the page is still fetched and charged once, exactly as before.

• Markdown token limit (maxTokens): return at most that many tokens (about 4 characters per token for Latin text, one per CJK character). The Markdown is cut at a blank-line boundary, never mid-block; when there is more, the row also carries nextStartIndex, the character offset to pass as startIndex in a later run to continue from where this one stopped.
• Start at character (startIndex): return the Markdown from that character offset onward (a nextStartIndex you got earlier). 0 is the start of the page.
• Links in Markdown (links): inline keeps [text](url); footnotes moves the URLs to numbered [^n] notes at the end; none keeps the link text only.
• Images in Markdown (images): all keeps ![alt](url); alt keeps the alt text only; none drops the images.
• Keep blocks matching a query (query, with Blocks kept for a query queryMaxBlocks, default 5): ranks the Markdown blocks with BM25 against the query and keeps only the most relevant blocks, in the original page order. No AI model is used; when no block matches, the Markdown is empty.

Limitations

• Public pages only: no logins, no cookies, no CAPTCHA or bot-check solving. Sites behind such checks are reported as blocked (not charged).
• Browser mode runs on Cloudflare (region Auto or Cloudflare; in Auto, with our Taiwan Chrome pool as the fallback when Cloudflare Browser Rendering is busy or reaches the 9-hour UTC monthly cap) or on our Taiwan residential Chrome pool (region Taiwan); regions us, sg and eu fetch over plain HTTP and do not support Browser mode. A browser page is one page-browser event on every route.
• The Taiwan line is a home internet connection with limited capacity: very large runs on tw can be slower, and a region that is temporarily unavailable is reported as region_unavailable (not charged).
• Up to 10 MB per page from regional networks, 15 MB from Cloudflare; HTML output up to 2,000,000 characters.

FAQ

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

Is this a proxy or an unblocker? No. It is a fetcher that runs from several networks you can choose. It always identifies itself and never tries to look like a person's browser. If a site blocks all of our networks, you get blocked and pay nothing.

What is charged when Auto switches region? The page plus the region fee ($0.003 in total) only if the page was delivered over HTTP from our US or Taiwan network in Auto (or from the region you chose, including Singapore and Europe); pages Cloudflare gets cost $0.001, browser pages $0.003.

What happens when a site blocks Smart Fetch or answers 403 or 429? In Auto, a blocked HTTP page tries the next configured HTTP network and then a browser; a Cloudflare HTTP JavaScript shell goes straight to the browser. Browser Rendering capacity 429 uses the Taiwan Chrome pool. If every route is refused you get one row with errorType blocked or rate_limited, the list of attempts, and no charge. It does not use anti-detection tricks.

What if I enter a wrong URL or setting? The run does not fail: you get a row with errorType: "invalid_input" that says what is wrong (for a wrong setting, which one and the allowed values), and you are not charged. If one setting is wrong, nothing is fetched at all, so fix it and start again.

What if I list the same page twice? It is fetched and charged once (https://example.com, HTTPS://EXAMPLE.COM/ and example.com/#top are one page); SUMMARY.duplicatesMerged counts the merged lines and the row's inputs lists the spellings.

Can I get only metadata? Yes: turn off Markdown. The page is still charged as one page.

Support

Open an issue in the Issues tab with the URL and the region. Issues are checked regularly.