World
"}' ## Using your API key curl -X POST https://imapp--web-markdown-api.apify.actor/v1/markdown \ -H "Authorization: Bearer your_key" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "options": {"output_format": "text"}}' ```` *** ### Features | Feature | Description | |---------|-------------| | **Clean Markdown** | Navigation, sidebars, and ads stripped automatically | | **Multi-format output** | `markdown`, `text` (plain), or `json` (structured) | | **CSS selector filtering** | Extract specific elements (`article.main`, `#content`) | | **Token budget control** | Set `max_length` to fit LLM context windows (100–1M chars) | | **Metadata extraction** | Title, description, author, date, word count, language | | **Bot evasion stack** | curl\_cffi TLS impersonation → httpx → Playwright browser | | **Browser rendering** | Optional headless Chromium for JS-heavy SPAs | | **Free trial** | 7-day Pro access, no credit card required | | **Rate limiting** | Per-client RPS + monthly quota with tiered limits | #### Bot Evasion Stack The API automatically falls through three layers to get content: ``` Fetch attempt ──► curl_cffi (Chrome TLS fingerprint) │ success? ──► Return content │ fail ▼ httpx (plain HTTP) │ success? ──► Return content │ fail (use_browser=true) ▼ Playwright (headless Chromium) │ success? ──► Return content ``` - **Layer 1: curl\_cffi** — Impersonates Chrome 125 TLS fingerprint. Passes Cloudflare's first line of defense. Handles ~80% of sites. - **Layer 2: httpx** — Plain HTTP fallback for simple sites. - **Layer 3: Playwright** — Full browser rendering for JavaScript-heavy SPAs (React, Vue, Angular). Patches `navigator.webdriver`, WebGL, permissions, and plugins for stealth. *** ### Pricing **Pay-per-result** — you only pay for successful conversions. | Event | Price | What You Get | |-------|-------|-------------| | **API result** | **$0.0004** | Per successful URL → Markdown conversion (10K = $4) | | **Actor start** | **$0.00005** | One-time per session (first 5 seconds waived) | #### Tier Limits | Tier | Calls/month | RPS | Example monthly cost at max usage | |------|:-----------:|:---:|:---------------------------------:| | **Free** | 10,000 | 5 | $0 (always) | | **Starter** | 25,000 | 10 | ~$10 | | **Pro** | 100,000 | 25 | ~$40 | | **Enterprise** | 500,000 | 50 | ~$200 | > Prices shown are base rates before Apify platform discounts (Bronze/Silver/Gold). > **Apify user discounts:** Apify subscription tiers (Bronze/Silver/Gold) apply > a discount to the per-result price automatically. *** ### Authentication Pass your API key via one of two methods: ```bash ## Option A: Authorization header (recommended) curl -H "Authorization: Bearer sk_pro_abc123" ... ## Option B: x-api-key header curl -H "x-api-key: sk_pro_abc123" ... ``` **No key?** Unauthenticated requests are rate-limited to **5 RPS / 10K calls/mo** (free tier). Get a **free 7-day Pro trial** at `POST /v1/trial` to unlock 25 RPS and 100K calls/mo. #### Free Trial ```bash curl -X POST https://imapp--web-markdown-api.apify.actor/v1/trial ``` ```json { "api_key": "trial_a1b2c3d4e5f6g7h8", "tier": "pro", "expires_in_days": 7, "usage_instructions": { "authorization": "Bearer trial_a1b2c3d4e5f6g7h8", "endpoints": ["POST /v1/markdown"], "note": "POST /v1/html also available for converting raw HTML", "docs": "/docs" } } ``` **Trial limits:** 1 key per IP address · 1 request per 60 seconds. *** ### API Reference All endpoints are versioned under `/v1`. Full interactive docs at [`/docs`](https://imapp--web-markdown-api.apify.actor/docs) (Swagger UI) or [`/redoc`](https://imapp--web-markdown-api.apify.actor/redoc). #### `GET /health` Health check for load-balancer probes. Unversioned — always accessible. ```bash curl https://imapp--web-markdown-api.apify.actor/health ``` ```json {"status":"ok","service":"web-markdown-api","version":"0.1.0"} ``` #### `POST /v1/markdown` Convert a public URL to clean Markdown, plain text, or structured JSON. ##### Request Body | Field | Type | Default | Description | |-------|------|---------|-------------| | `url` | string | *required* | The URL to convert (http/https) | | `options.output_format` | string | `"markdown"` | `markdown`, `text`, or `json` | | `options.css_selector` | string | `null` | CSS selector for a specific element | | `options.max_length` | int | `null` | Max output characters (100–1,000,000) | | `options.include_links` | bool | `true` | Preserve hyperlinks in output | | `options.include_images` | bool | `false` | Include image references | | `options.exclude_navigation` | bool | `true` | Strip nav/sidebar/footer | | `options.use_browser` | bool | `false` | Enable headless Chromium for JS-rendered pages | ##### Example: Markdown output ```bash curl -X POST https://imapp--web-markdown-api.apify.actor/v1/markdown \ -H "Authorization: Bearer your_key" \ -H "Content-Type: application/json" \ -d '{ "url": "https://en.wikipedia.org/wiki/Markdown", "options": { "output_format": "markdown", "max_length": 5000, "include_links": true, "exclude_navigation": true } }' ``` ```json { "success": true, "url": "https://en.wikipedia.org/wiki/Markdown", "markdown": "# Markdown\n\n**Markdown** is a lightweight markup language...", "metadata": { "title": "Markdown - Wikipedia", "description": "Lightweight markup language for plain text formatting", "word_count": 1450, "char_count": 8900, "image_count": 3, "link_count": 45, "language": "en" }, "output_format": "markdown", "source": "trafilatura" } ``` ##### Example: Plain text output ```bash curl -X POST https://imapp--web-markdown-api.apify.actor/v1/markdown \ -H "Authorization: Bearer your_key" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "options": {"output_format": "text"}}' ``` ```json { "success": true, "url": "https://example.com", "text": "Example Domain\n\nThis domain is for use in illustrative examples...", "metadata": {"word_count": 17}, "output_format": "text", "source": "trafilatura" } ``` ##### Example: Browser rendering for JS-heavy pages ```bash curl -X POST https://imapp--web-markdown-api.apify.actor/v1/markdown \ -H "Authorization: Bearer your_key" \ -H "Content-Type: application/json" \ -d '{ "url": "https://react.dev/reference/react/hooks", "options": { "output_format": "markdown", "use_browser": true, "max_length": 2000 } }' ``` > **Note:** `use_browser: true` enables headless Chromium via Playwright. > This is slower (3-10s per page) but can handle JavaScript-rendered SPAs > that HTTP-only fetchers can't parse. #### `POST /v1/html` Convert raw HTML directly to Markdown — no fetch step. Useful when you already have the HTML content and want to skip the network request. Same options as `/v1/markdown` (minus the `url` field). ```bash curl -X POST https://imapp--web-markdown-api.apify.actor/v1/html \ -H "Authorization: Bearer your_key" \ -H "Content-Type: application/json" \ -d '{ "html": "This is bold text.
", "options": {"output_format": "markdown"} }' ``` ```json { "success": true, "url": "", "markdown": "# Hello\n\nThis is **bold** text.", "metadata": {"word_count": 5}, "output_format": "markdown", "source": "trafilatura" } ``` #### `POST /v1/trial` Get a free 7-day Pro trial key. See [Authentication](#authentication) above for the full response format and limits. #### `GET /v1/usage` Rate-limit usage statistics — for monitoring and debugging. ```bash curl https://imapp--web-markdown-api.apify.actor/v1/usage \ -H "Authorization: Bearer your_key" ``` ```json { "active_clients": 12, "total_monthly_calls": 3421, "by_tier": { "free": {"clients": 8, "calls_this_month": 1200}, "pro": {"clients": 4, "calls_this_month": 2221} } } ``` #### Error Responses | Status | Meaning | |--------|---------| | 400 | Bad request (invalid parameters) | | 404 | URL returned HTTP 404 | | 422 | Validation error (invalid URL format, empty HTML) | | 429 | Rate limit exceeded (RPS or monthly quota) | | 502 | Failed to fetch, render, or convert the URL | Rate-limited responses include retry information: ```json { "detail": "RPS limit exceeded", "retry_after_seconds": 0.85, "upgrade_url": "https://console.apify.com/actors/imapp~web-markdown-api", "tier": "free" } ``` All responses include rate-limit headers: ```text X-RateLimit-Limit-RPS: 5 X-RateLimit-Remaining-RPS: 4 X-RateLimit-Limit-Monthly: 10000 X-RateLimit-Remaining-Monthly: 9997 X-Tier: free ``` *** ### Use Cases | Use Case | How It Helps | |----------|-------------| | **RAG Pipelines** | Ingest web content into vector databases (Pinecone, Weaviate, Qdrant) for semantic search | | **AI Agents** | Provide clean, token-efficient context for Claude, GPT, Gemini reasoning | | **Content Analysis** | Extract and analyze article text, sentiment, or topics at scale | | **Documentation Ingestion** | Convert docs to Markdown for LLM fine-tuning or RAG | | **Data Collection** | Build structured datasets from web sources without browser overhead | | **Competitor Monitoring** | Track pricing, features, and content changes on competitor sites | *** ### Rate Limits Each tier has a **requests-per-second** (RPS) cap and a **monthly quota**. See the [Pricing section](#pricing) for tier-specific limits. - Free tier: 5 RPS, 10K calls/month - Pro trial: 25 RPS, 100K calls/month (7 days) - Paid keys: Configured per customer When you exceed either limit, all endpoints return `HTTP 429` with a `retry_after_seconds` field and a link to upgrade. *** ### Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `LOG_LEVEL` | `info` | Log verbosity (`debug`, `info`, `warning`, `error`) | | `DEFAULT_RPS` | `5` | Free tier requests/second limit | | `DEFAULT_MONTHLY_CALLS` | `10000` | Free tier monthly call limit | | `PRO_RPS` | `25` | Pro tier requests/second | | `PRO_MONTHLY_CALLS` | `100000` | Pro tier monthly calls | | `HTTP_TIMEOUT` | `30` | HTTP fetch timeout in seconds | | `MAX_RESPONSE_SIZE` | `5242880` | Max response size in bytes (5 MB) | | `PRIMARY_FETCHER` | `curl_cffi` | Primary HTTP client (`curl_cffi` or `httpx`) | | `USER_AGENT` | — | Custom User-Agent override (disables rotation) | | `API_KEYS_JSON` | `{}` | JSON mapping of API keys to tiers (set as **Secret**) | | `ENABLE_PROXY` | auto | Enable Apify proxy (`1`, `true`, or auto-detect) | | `PROXY_GROUPS` | `RESIDENTIAL` | Apify proxy group | | `TRIAL_DURATION_DAYS` | `7` | Trial key validity period | > **Secrets:** `API_KEYS_JSON` should be set as a **Secret** in the > Apify Console (Settings → Secrets), not as a regular environment variable. *** *** ### Supported Sites | Type | Quality | Notes | |------|---------|-------| | Blog posts, news, docs | ⭐ Excellent | Trafilatura excels at article extraction | | Static HTML sites | ⭐ Excellent | Fast, reliable, low latency | | JavaScript SPAs (React, Vue) | ⭐ Good | Enable `use_browser: true` for JS rendering | | Cloudflare-protected sites | ⭐ Good | TLS impersonation handles most challenges | | PDFs, images, binaries | ❌ Not supported | Return HTTP 502 with error detail | *** ### About Built by [imapp](https://apify.com/imapp). Part of the IM App suite of developer APIs on the Apify platform. **Questions?** [Open an issue](https://github.com/imapp/web-markdown-api/issues) or reach out via the Apify Console. # Actor input Schema ## `logLevel` (type: `string`): Verbosity of server logs. ## `httpTimeout` (type: `integer`): Timeout for fetching web pages. ## `defaultRps` (type: `integer`): Requests per second limit for free tier users. ## `defaultMonthlyCalls` (type: `integer`): Monthly call limit for free tier users. ## `apiKeysJson` (type: `string`): JSON mapping of API keys to tier names for paid access. ## Actor input object example ```json { "logLevel": "info", "httpTimeout": 30, "defaultRps": 5, "defaultMonthlyCalls": 10000, "apiKeysJson": "{}" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '