# Stealth Scraper MD - blocks to Markdown/JSON/CSV (`enheartening_loam/stealth-scraper-md`) Actor

Auto-escalating web scraper for anti-bot protected pages (Cloudflare/Turnstile/Akamai/DataDome). Clean Markdown, HTML, JSON or CSV with pagination, CSS-selectors, caching, webhooks. HTTP->dynamic->stealth, no proxy setup.

- **URL**: https://apify.com/enheartening\_loam/stealth-scraper-md.md
- **Developed by:** [Gv Gn](https://apify.com/enheartening_loam) (community)
- **Categories:** E-commerce, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Stealth Scraper MD

Extract **public** web data from sites that block bots (Cloudflare / Turnstile / Akamai / DataDome). Get clean **Markdown, HTML, JSON or CSV** automatically.

Tries fast HTTP first; if a page blocks it, escalates to a browser, then a stealth Chromium. No proxy/browser management on your side.

### Features

- Auto-escalation: `HTTP → dynamic → stealth`
- Formats: Markdown, HTML, JSON (CSS selectors, `@attr`), CSV
- Pagination (`next`-links), Apify Proxy, result cache, webhooks
- Custom user-agent / headers / cookies
- Optional ZenRows / Bright Data Web Unlocker fallback
- Free tier + promo codes (beta)

### Quick start

```json
{
  "urls": ["https://books.toscrape.com"],
  "outputFormat": "json",
  "jsonStructure": { "title": "h3 a", "price": "p.price_color" }
}
```

One dataset row per product. Paste, run, done.

**CLI:** `apify call YOUR-USERNAME/stealth-scraper-md --default-result`

### Presets

| Preset | Result |
|---|---|
| `examples/demo_books_json.json` | Product cards → JSON |
| `examples/demo_article_markdown.json` | Page → clean Markdown |
| `examples/demo_pagination_csv.json` | Paginated list → CSV |
| `examples/preset_ecommerce_product.json` | Product card (price/rating/SKU) → JSON |
| `examples/preset_llm_rag_markdown.json` | Article → clean Markdown (LLM/RAG) |
| `examples/preset_seo_audit.json` | Page SEO audit → JSON |

### Pricing

| Tier | Price |
|---|---|
| Free | 100 results / mo |
| Promo (`REDDIT5K`, `DEVTO5K`, `TELEGRAM5K`, `X5K`, `PH5K`, `BETA5K`) | 5,000 results |
| Early-bird | $0.39 / 1000 successful results |

Only successful, usable pages are billed. Blocked or empty pages are not charged.

### Input options

`urls` (required) · `mode` (auto/stealth/dynamic/http) · `cssSelector` · `maxUrls` · `outputFormat` (markdown/html/json) · `jsonStructure` · `pagination` · `useProxy` / `proxyCountry` / `proxyGroups` · `userAgent` / `headers` / `cookies` · `cache` · `webhookUrl` / `webhookToken` · `unblocker` · `promoCode`

### Can / cannot

| Can | Cannot |
|---|---|
| Plain HTTP + auto browser escalation | Log in & scrape *behind* a session (unless you pass cookies) |
| Cloudflare / Turnstile / Akamai / DataDome | Infinite-scroll pagination, mandatory 2FA / manual CAPTCHAs |
| Markdown / HTML / JSON / CSV / tables | Fill login forms |
| Pagination via `next`-links | Promise 100% success on every site |

### Compliance

For **lawful collection of public web data**. Respect each site's ToS, robots.txt, GDPR/CCPA and platform rules. Not for login/paywall bypass or personal data without a legal basis.

### Support

- **Report a bug:** GitHub Issues → <https://github.com/gal001/stealth-scraper-md/issues>
- **Beta / promo:** message via Issues with your source + use case.
- Known limits: HTTP proxy won't bypass Cloudflare IP/TLS blocks; infinite-scroll pages stop at page 1.

# Actor input Schema

## `urls` (type: `array`):

List of page URLs to fetch. You can also provide a single 'url' field instead.

## `url` (type: `string`):

Minimal input: one URL. If both url and urls are provided, urls wins.

## `preset` (type: `string`):

Ready-made profile. ecommerce\_product = product card JSON; article\_llm = clean markdown for LLM/RAG; table = extract tables; seo\_audit = SEO fields; reviews = generic reviews.

## `discoverSitemap` (type: `boolean`):

Expand the URL list automatically from the first URL's robots.txt/sitemap.xml. Great for catalogues and e-commerce — given one homepage it finds many product/article pages. Capped by 'max items to scrape'.

## `mode` (type: `string`):

auto/standard = try cheap HTTP first, escalate if blocked; http/fast = plain HTTP only; dynamic = JS browser; stealth/max = strongest anti-bot bypass.

## `cssSelector` (type: `string`):

Extract only matching elements, e.g. article or .main-content. Leave empty for whole page.

## `maxUrls` (type: `integer`):

Safety cap on how many input URLs are processed in one run.

## `maxResults` (type: `integer`):

Hard cap on successful result rows. 0 means use quota/promo default.

## `maxOutputBytes` (type: `integer`):

Stop after total output exceeds this many bytes. 0 means default 512 MB.

## `timeoutSeconds` (type: `integer`):

Per-request timeout. Browser stealth may internally use a larger budget derived from this value.

## `maxCostUnits` (type: `integer`):

If the pre-run estimate exceeds this value, the actor will not start scraping. 0 disables the guard.

## `simulate` (type: `boolean`):

If enabled, actor validates input and returns cost estimate without scraping.

## `outputFormat` (type: `string`):

markdown = clean markdown; html = full HTML; json = structured fields from jsonStructure/tables.

## `jsonStructure` (type: `object`):

Used when outputFormat=json. Supports simple selectors ({"title":"h1"}) or typed specs ({"price":{"selector":".price","type":"price","required":true}}). Types: string, integer, number, price, bool, url, list.

## `tables` (type: `object`):

When outputFormat=json, extract HTML tables as lists of row dicts. Format: {"name": "css-selector-of-table"}.

## `includeMetadata` (type: `boolean`):

Add metadata dict with title, description, canonical, og tags, author, publishedAt, lang, JSON-LD snippet.

## `seoAudit` (type: `boolean`):

Add SEO diagnostics: h1/h2, text length, links count, images missing alt, canonical, meta description.

## `pagination` (type: `boolean`):

Walk each input URL as a chain of pages via next links. Respects mode escalation. Stops at last page, maxUrls, maxResults, or output limit.

## `useProxy` (type: `boolean`):

When enabled, requests are chained through Apify Proxy after local SSRF filtering. Requires APIFY\_PROXY\_\* env.

## `proxyCountry` (type: `string`):

e.g. US, DE, FR. Only used when useProxy=true.

## `proxyGroups` (type: `string`):

e.g. RESIDENTIAL. Only used when useProxy=true.

## `userAgent` (type: `string`):

Override default User-Agent for HTTP and browser modes.

## `headers` (type: `object`):

Extra HTTP headers, e.g. {"Accept-Language": "en-US"}.

## `cookies` (type: `string`):

Cookie header string or JSON object for authorized pages. Sensitive: not logged.

## `webhookUrl` (type: `string`):

POST compact run summary to this URL after finish.

## `webhookToken` (type: `string`):

Sent as Authorization: Bearer <token>. Sensitive.

## `cache` (type: `object`):

Disk cache for repeat requests. Disabled by default. Only high-quality successful rows are cached.

## `unblocker` (type: `object`):

External anti-bot service config: {"provider": "zenrows" | "webunlocker", "apiKey": "...", "zone": "..."}. Alternatively use top-level unblockerProvider/unblockerApiKey/unblockerZone.

## `unblockerProvider` (type: `string`):

zenrows or webunlocker.

## `unblockerApiKey` (type: `string`):

Sensitive key for external unblocker.

## `unblockerZone` (type: `string`):

Required only for webunlocker (Bright Data zone).

## `promoCode` (type: `string`):

Promo code increases free result cap for this run.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://quotes.toscrape.com"
  ],
  "preset": "none",
  "discoverSitemap": false,
  "mode": "auto",
  "maxUrls": 50,
  "maxResults": 0,
  "maxOutputBytes": 0,
  "timeoutSeconds": 30,
  "maxCostUnits": 0,
  "simulate": false,
  "outputFormat": "markdown",
  "jsonStructure": {
    "title": "title",
    "quotes": "div.quote span.text"
  },
  "tables": {
    "products": "#products-table"
  },
  "includeMetadata": false,
  "seoAudit": false,
  "pagination": false,
  "useProxy": false,
  "headers": {},
  "cache": {
    "enabled": false,
    "ttl": 3600
  },
  "unblocker": {
    "provider": "",
    "apiKey": "",
    "zone": ""
  },
  "unblockerProvider": ""
}
```

# Actor output Schema

## `url` (type: `string`):

No description

## `finalUrl` (type: `string`):

No description

## `status` (type: `string`):

No description

## `method` (type: `string`):

No description

## `blocked` (type: `string`):

No description

## `quality` (type: `string`):

No description

## `markdown` (type: `string`):

No description

## `html` (type: `string`):

No description

## `json` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `title` (type: `string`):

No description

## `timestamp` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com",
        "https://quotes.toscrape.com"
    ],
    "jsonStructure": {
        "title": "title",
        "quotes": "div.quote span.text"
    },
    "tables": {
        "products": "#products-table"
    },
    "headers": {},
    "cache": {
        "enabled": false,
        "ttl": 3600
    },
    "unblocker": {
        "provider": "",
        "apiKey": "",
        "zone": ""
    },
    "promoCode": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("enheartening_loam/stealth-scraper-md").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://example.com",
        "https://quotes.toscrape.com",
    ],
    "jsonStructure": {
        "title": "title",
        "quotes": "div.quote span.text",
    },
    "tables": { "products": "#products-table" },
    "headers": {},
    "cache": {
        "enabled": False,
        "ttl": 3600,
    },
    "unblocker": {
        "provider": "",
        "apiKey": "",
        "zone": "",
    },
    "promoCode": "",
}

# Run the Actor and wait for it to finish
run = client.actor("enheartening_loam/stealth-scraper-md").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com",
    "https://quotes.toscrape.com"
  ],
  "jsonStructure": {
    "title": "title",
    "quotes": "div.quote span.text"
  },
  "tables": {
    "products": "#products-table"
  },
  "headers": {},
  "cache": {
    "enabled": false,
    "ttl": 3600
  },
  "unblocker": {
    "provider": "",
    "apiKey": "",
    "zone": ""
  },
  "promoCode": ""
}' |
apify call enheartening_loam/stealth-scraper-md --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,enheartening_loam/stealth-scraper-md"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gemHpHeAUAgcG5A1e/builds/Kj5ieputRzFggz6Qk/openapi.json