# Stealth Web Scraper & Markdown Extractor (`vietaro/cloudflare-bypass-scraper`) Actor

Self-healing, anti-bot-resistant scraper. Escalates from fast HTTP to full stealth browser only when blocked, extracts fields with adaptive (self-healing) selectors, and returns clean LLM-ready markdown.

- **URL**: https://apify.com/vietaro/cloudflare-bypass-scraper.md
- **Developed by:** [Vietaro Tools](https://apify.com/vietaro) (community)
- **Categories:** Developer tools, AI, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Stealth Web Scraper & Cloudflare Bypass — Markdown Extractor

**Fast web fetching by default. Automatic Cloudflare/anti-bot bypass when blocked. Clean Markdown for AI, RAG & automation.**

Fetch difficult, bot-protected web pages without sending every URL through a browser — and without paying stealth-browser prices for pages that didn't need one.

#### 🎬 See it in action

![Resilient Stealth Fetcher — Fast to Stealth](https://vietaro.com/assets/apify/stealth-fetcher-demo-v2.png)

**Fast → blocked → automatic stealth → HTTP 200 → clean Markdown**

The Actor starts with a lightweight fetch and automatically escalates to a stealth browser only when a site actually blocks it. You can also force stealth mode for sites known to require a browser.

***

### Why use it?

#### ⚡ Fast first, stealth only when needed

```text
URL
 ↓
Fast HTTP fetch
 ↓
Works? ─────────────── Yes → Clean Markdown
 ↓ No
Stealth browser (Cloudflare/anti-bot bypass)
 ↓
Clean Markdown
```

**Verified results from real Actor runs:**

| Task | Result | Tier used | Extraction |
|---|---|---|---|
| example.com | HTTP 200 | `fast` | `title` → "Example Domain" |
| MDN | HTTP 200 | `fast` | `title` → "Add JavaScript to your web page" |
| Cloudflare Challenge page | HTTP 200 | `stealth` | `title` → "Cloudflare Challenge" |

Every row above is a successful extraction — the Cloudflare-protected page returned a real HTTP 200 via automatic stealth escalation, not a placeholder or partial result.

You can also set `forceStealth: true` for sites already known to require a browser.

***

### 💰 Pricing

**Free tier, forever:** 300 fast-tier pages and 25 stealth-tier pages every month, on every account — enough to fully evaluate this Actor before spending anything.

| What triggers a charge | Price | Free allowance / month |
|---|---|---|
| Actor start | $0.005 | — |
| Fast-tier page (no anti-bot resistance needed) | $0.005 | 300 |
| Stealth-tier page (Cloudflare/anti-bot escalation) | $0.035 | 25 |
| Structured field extracted (optional, via `extractionSchema`) | $0.002 | — |

You're only ever charged the stealth price for pages that actually needed it — a page resolved on the fast tier is never billed at the stealth rate.

| Example volume (typical mix) | Estimated cost |
|---|---|
| 1,000 pages, ~80% fast / 20% stealth | ~$11.00 |
| 10,000 pages, ~80% fast / 20% stealth | ~$110.00 |

***

### 🧹 Clean, LLM-ready Markdown

Turn web pages into useful Markdown instead of feeding your AI pipeline raw HTML, navigation, and page chrome.

The Markdown output preserves headings, paragraphs, links, lists/tables, and code blocks — suitable for RAG pipelines, embeddings, AI agents, research, content monitoring, and knowledge-base ingestion.

***

### 🎯 Optional structured extraction

Request additional fields using simple CSS selectors:

```json
[
  { "name": "title", "selector": "h1" },
  { "name": "author", "selector": "[rel='author']" }
]
```

```json
{
  "extracted": {
    "title": "Example Domain",
    "author": "John Doe"
  }
}
```

#### Adaptive (self-healing) extraction

Fields extracted via `extractionSchema` use Scrapling's adaptive-relocation mechanism: when the exact selector stops matching, the field is relocated by DOM similarity instead of returning null. Verified end-to-end in Showcase 5, including the actual similarity score behind a real recovery (100% match). Still validate adaptive matching against your own target sites before relying on it unattended in production -- one verified site is a proof of the mechanism, not a guarantee across every possible page structure.

***

### 🚀 Quick start

```json
{ "startUrls": [{ "url": "https://example.com" }] }
```

Force stealth for a site known to require a browser:

```json
{ "startUrls": [{ "url": "https://example.com" }], "forceStealth": true }
```

With structured extraction:

```json
{
  "startUrls": [{ "url": "https://example.com" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }]
}
```

***

### 🧪 Showcases

Each showcase includes the **copyable input** and the **actual output** from a real Actor run, so you can compare your own run against it.

#### 1. Normal page → fast fetch

**Input**

```json
{
  "startUrls": [{ "url": "https://example.com" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 60,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

**Output**

```json
{
  "url": "https://example.com",
  "success": true,
  "statusCode": 200,
  "fetchTier": "fast",
  "extracted": { "title": "Example Domain" }
}
```

#### 2. MDN → fast fetch + structured extraction

**Input**

```json
{
  "startUrls": [{ "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 60,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

**Output**

```json
{
  "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page",
  "success": true,
  "statusCode": 200,
  "fetchTier": "fast",
  "extracted": { "title": "Add JavaScript to your web page" }
}
```

#### 3. Cloudflare Challenge → automatic stealth escalation

**Input**

```json
{
  "startUrls": [{ "url": "https://www.scrapingcourse.com/cloudflare-challenge" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "markdownThreshold": 45,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 60,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

**Output**

```json
{
  "url": "https://www.scrapingcourse.com/cloudflare-challenge",
  "success": true,
  "statusCode": 200,
  "fetchTier": "stealth",
  "extracted": { "title": "Cloudflare Challenge" },
  "markdown": "# Cloudflare Challenge\n## You bypassed the Cloudflare challenge! :D"
}
```

> **Note on proxy settings above:** examples 1–3 deliberately run with `useApifyProxy: false` to demonstrate the escalation ladder working even in the hardest case — no proxy at all. **For production use, set `useApifyProxy: true`** (the input schema default) — see Example 4.

#### 4. Recommended production config (residential proxy)

**Input**

```json
{
  "startUrls": [{ "url": "https://www.scrapingcourse.com/cloudflare-challenge" }],
  "extractionSchema": [{ "name": "title", "selector": "h1" }],
  "forceStealth": false,
  "markdownThreshold": 45,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 60,
  "proxyConfiguration": { "useApifyProxy": true }
}
```

**Output**

```json
{
  "url": "https://www.scrapingcourse.com/cloudflare-challenge",
  "success": true,
  "statusCode": 200,
  "fetchTier": "stealth",
  "extracted": { "title": "Cloudflare Challenge" },
  "markdown": "# Cloudflare Challenge\n## You bypassed the Cloudflare challenge! :D"
}
```

Same successful result as Example 3, confirmed with the recommended proxy setting.

#### 5. Self-healing in action -- verified

Two-run test: Run A seeds a fingerprint with a working selector, Run B repeats the same URL and field name with a **deliberately broken selector** that matches nothing on the page.

**Run A -- seed a fingerprint**

```json
{
  "startUrls": [{ "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page" }],
  "extractionSchema": [{ "name": "heading", "selector": "h1" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 60,
  "proxyConfiguration": { "useApifyProxy": true }
}
```

```json
{ "success": true, "statusCode": 200, "fetchTier": "fast", "extracted": { "heading": "Add JavaScript to your web page" } }
```

**Run B -- same URL/field, selector changed to `h1.this-class-no-longer-exists`**

```json
{
  "startUrls": [{ "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/How_to/Add_JavaScript_to_your_web_page" }],
  "extractionSchema": [{ "name": "heading", "selector": "h1.this-class-no-longer-exists" }],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 60,
  "proxyConfiguration": { "useApifyProxy": true }
}
```

```json
{ "success": true, "statusCode": 200, "fetchTier": "fast", "extracted": { "heading": "Add JavaScript to your web page" } }
```

**The field was recovered even though the selector matched nothing on the page.** This isn't just a pass/fail result -- the run log shows the actual relocation mechanism at work:

```
DEBUG: Highest probability was 100.0%
DEBUG: Top 5 best matching elements are:
100.0 -> [<data='<h1>Add JavaScript to your web page</h1>' parent='<div class="layout__header...'>]
62.24 -> [<data='<mdn-survey></mdn-survey>' ...>]
49.28 -> [<data='<p> Your blueprint for a better internet...' ...>]
```

The correct `<h1>` scored a perfect 100% match against the fingerprint saved in Run A, comfortably ahead of the next-best candidate (62%) -- a clean, unambiguous relocation, not a lucky guess near the threshold.

#### 6. Graceful failure handling

**Input**

```json
{
  "startUrls": [{ "url": "https://this-domain-does-not-exist-abcxyz123.com" }],
  "extractionSchema": [],
  "forceStealth": false,
  "maxConcurrency": 1,
  "maxUrlsPerRun": 1,
  "fetchTimeoutSecs": 30,
  "proxyConfiguration": { "useApifyProxy": true }
}
```

**Output**

```json
{
  "url": "https://this-domain-does-not-exist-abcxyz123.com",
  "success": false,
  "error": "Error: Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://this-domain-does-not-exist-abcxyz123.com/"
}
```

The run completed normally with `success: false` and a real, actionable error message — it did not crash or block other URLs in the same run.

> A failed or timed-out URL is recorded with `success: false` and an `error` message — it never stops the rest of a multi-URL run.

***

### 📤 Output

Each URL produces a result containing `url`, `fetchedAt`, `success`, `statusCode`, `fetchTier` (`fast` or `stealth`), `markdown`, `extracted`, and — only on failure — `error`.

```json
{
  "url": "https://example.com",
  "fetchedAt": "2026-08-16T14:21:22.097540+00:00",
  "success": true,
  "statusCode": 200,
  "fetchTier": "fast",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "extracted": {
    "title": "Example Domain",
    "paragraph1": "This domain is for use in documentation examples..."
  }
}
```

***

### When should I use this?

#### Good fit

- **AI / RAG** — ingest documentation, public information, and web content
- **AI agents** — provide agents with clean web-page content
- **Research** — collect readable content from multiple websites
- **Competitive intelligence** — monitor pages that may be difficult to fetch
- **SEO & agencies** — recurring content and page audits
- **Sales & RevOps** — collect public company and product information
- **Automation** — use the results in n8n, LangChain, CrewAI, MCP, or custom workflows

#### When to use something else

| If you need... | Consider instead |
|---|---|
| Login-only / account-gated pages | A custom Actor with session/auth handling — this Actor only accesses publicly reachable pages |
| Structured data from a specific major site (LinkedIn, Amazon, etc.) | A dedicated site-specific Actor — usually cheaper and more structured than generic CSS selectors |
| Guaranteed success on every request | No tool can promise this — sites change and anti-bot systems evolve |
| Pages with no anti-bot protection at all | A simpler/cheaper HTTP-only crawler may be more cost-effective than paying for stealth-tier headroom you won't use |

***

### ❓ FAQ

**Do I need my own proxy?**
No — Apify Proxy is used by default and is strongly recommended, especially for the stealth tier (see Example 4). You can supply your own proxy if you have an existing contract.

**Is this free to try?**
Yes — every account gets 300 free fast-tier pages and 25 free stealth-tier pages every month, permanently, not just during a trial window.

**What happens if a page fails or times out?**
It's recorded with `success: false` and an `error` field (see Showcase 6). The rest of the run continues normally.

**Will this work on any Cloudflare-protected site?**
The stealth tier is built to solve Cloudflare-style managed and embedded challenges automatically. Results can vary by target — validate against your specific site before relying on it for unattended production jobs.

**Is the "self-healing" claim real, or marketing?**
Real and verified -- see Showcase 5 for a reproducible before/after test, including the actual relocation confidence score (100% match) from a real run log, not just a claimed pass/fail. It's not infallible on every possible page: validate it against your own target sites, especially pages with many visually similar elements.

***

### ⚙️ Input

| Field | Purpose |
|---|---|
| `startUrls` | URLs to fetch |
| `extractionSchema` | Optional `{name, selector}` fields, matched adaptively |
| `forceStealth` | Skip directly to the stealth tier |
| `markdownThreshold` | Markdown boilerplate pruning aggressiveness |
| `maxConcurrency` | Parallel URL limit |
| `maxUrlsPerRun` | Maximum URLs processed per run |
| `fetchTimeoutSecs` | Per-URL timeout |
| `proxyConfiguration` | Apify Proxy (recommended) or your own proxy |

***

### 🧩 Built for AI and automation

**Web page → Clean Markdown → RAG / embeddings / AI agent / automation**

Compatible with n8n, LangChain, CrewAI, MCP, and custom Python/JavaScript workflows.

***

### Open source

Markdown generation uses **Crawl4AI** (Apache 2.0).

> This product includes software developed by UncleCode (https://x.com/unclecode)
> as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai).

Stealth fetching and adaptive extraction use **Scrapling** (BSD-3-Clause), with **Camoufox** (MPL-2.0) powering the stealth-browser tier.

***

### Acceptable use

You are responsible for complying with each target site's Terms of Service, `robots.txt`, applicable laws, and [Apify's Acceptable Use Policy](https://docs.apify.com/legal/acceptable-use-policy). This Actor provides a general-purpose fetching capability — use stealth features only where you have the right to access the target content.

***

### Try it

Give it a normal URL and let it stay fast. Give it a difficult or protected URL and let it escalate when necessary.

**Fast when possible. Stealth when needed. Clean Markdown every time.**

# Actor input Schema

## `startUrls` (type: `array`):

One or more URLs to fetch. Each will go through the fast->stealth escalation ladder independently.

## `extractionSchema` (type: `array`):

Optional list of {name, selector} pairs. Selectors are matched adaptively -- if the site's DOM changes later, Scrapling relocates the element by similarity instead of returning null.

## `forceStealth` (type: `boolean`):

Skip the fast HTTP attempt and go straight to the stealth browser. Use for sites you already know are hard-gated (Cloudflare Turnstile, DataDome, etc).

## `markdownThreshold` (type: `integer`):

0-100. Higher = more aggressive removal of boilerplate/nav/ads from the markdown output. 45 is a good default.

## `maxConcurrency` (type: `integer`):

How many URLs to fetch in parallel. Higher values finish faster but increase peak memory/CU usage and the chance of tripping rate limits on the same target site.

## `maxUrlsPerRun` (type: `integer`):

Hard cap on how many startUrls a single run will process, regardless of how many are supplied. Protects against an oversized input list blowing up compute cost -- applies whether or not the Actor is monetized. Set to 0 to disable the cap.

## `fetchTimeoutSecs` (type: `integer`):

Hard cap on how long a single URL's fetch attempt (including stealth-tier Cloudflare/anti-bot solving) is allowed to run before it's aborted and recorded as a timeout error. Protects against a hard-gated site hanging the whole run indefinitely.

## `proxyConfiguration` (type: `object`):

Apify Proxy or your own proxy list. Strongly recommended for the stealth tier.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "extractionSchema": [],
  "forceStealth": false,
  "markdownThreshold": 45,
  "maxConcurrency": 5,
  "maxUrlsPerRun": 50,
  "fetchTimeoutSecs": 150,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Fetched pages including fetch status, fetch tier, clean Markdown, and optional structured extraction.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("vietaro/cloudflare-bypass-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("vietaro/cloudflare-bypass-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call vietaro/cloudflare-bypass-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vietaro/cloudflare-bypass-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/NJNOLaicbKNzkNt3W/builds/hkyi6zuBaWPRkyyB4/openapi.json