🚀 Cloudflare Crawl Router for Apify

Cloudflare Crawl Router is an intelligent README.md for web scraping, website crawling, structured data extraction, and content collection. It inspects each target URL, detects anti-bot friction and frontend complexity, and automatically routes the request to the most efficient compliant strategy: lightweight HTTP, Cheerio-style parsing, browser rendering, or a safe early abort when the site is not suitable for the current crawl settings.

Instead of paying browser costs for every page, this Actor helps optimize crawl performance, extraction quality, and infrastructure spend. It is especially useful for automation teams, AI pipelines, RAG workflows, monitoring systems, lead-enrichment jobs, and research use cases that need clean output plus transparent crawl diagnostics.

🌟 Why this Apify Actor stands out

This Actor is built for users who want:

• ⚡ Crawl simple sites cheaply with HTTP before escalating to browser rendering when needed
• 🛡️ Detect Cloudflare and JavaScript-heavy pages early and route accordingly
• 📄 Generate AI-ready markdown and cleaned text for LLM and RAG pipelines
• 🔍 Understand why a route succeeded, failed, or escalated through transparent diagnostics

It does not claim to bypass protections unlawfully. Instead, it detects Cloudflare-related signals, challenge hints, login walls, and crawl friction, then chooses the best legal and technically appropriate extraction strategy available under your configuration.

⚙️ How it works

1. Analyze the target URL and detect site characteristics
2. Detect frontend technology, Cloudflare signals, and protection levels
3. Choose the most appropriate crawl strategy (HTTP, Cheerio, or Browser)
4. Extract content, metadata, and links from the page
5. Return structured data plus routing diagnostics explaining the decisions made

🎯 Best use cases

• Website content extraction for AI and LLM pipelines
• Product page scraping and structured product data collection
• Article and blog extraction for RAG or search indexing
• Compliance-aware website crawling on mixed static/dynamic sites
• Cost-optimized browser-vs-HTTP crawl orchestration
• Crawl diagnostics for protected or JavaScript-heavy targets
• Apify workflows that need structured content plus explanation of routing decisions

📈 Apify Marketplace positioning

Cloudflare Crawl Router is positioned as an intelligent Apify web scraping Actor for users who need more than a generic crawler. It helps turn websites into clean structured data while improving cost efficiency, crawl transparency, and strategy selection. This makes it well suited for the Apify Marketplace under categories such as website crawler, content scraper, product scraper, AI data extraction, and smart browser automation.

✅ Features

• 🤖 Automatic crawl strategy routing across HTTP fetch, Cheerio parsing, Playwright rendering, and queue-based site crawling
• ☁️ Cloudflare-aware signal detection using headers, challenge markers, Turnstile hints, and suspicious response patterns
• 🧩 Technology fingerprinting for React, Next.js, Nuxt, Vue, Angular, SvelteKit, Astro, WordPress, Shopify, and similar stacks
• 💸 Cost-aware browser escalation and proxy recommendations
• 📦 Extraction-ready output for articles, products, jobs, profiles, and generic content pages
• 📝 Clean text and markdown generation for AI agents, LLM pipelines, summarization, and RAG ingestion
• 📊 Crawl summaries, routing summaries, domain profiles, and optional HTML or screenshot artifacts
• 🤝 Robots.txt-aware and sitemap-aware discovery flows for compliant crawling

📥 Input Configuration

The Actor accepts a JSON input with grouped sections. Here is a complete example:

{
  "routing": {
    "fallbackToBrowser": true,
    "mode": "site-crawl",
    "renderJavaScript": true
  },
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "limits": {
    "maxDepth": 1,
    "maxPages": 20,
    "timeoutSecs": 45
  },
  "extraction": {
    "extractionType": "auto"
  },
  "output": {
    "outputFormat": "json",
    "includeMetadata": true,
    "includeDiagnostics": true
  },
  "networking": {
    "useProxy": "auto"
  },
  "compliance": {
    "obeyRobotsTxt": true,
    "sameDomainOnly": true
  },
  "artifacts": {
    "saveHtml": false,
    "saveMarkdown": true,
    "saveScreenshot": false
  },
  "diagnostics": {
    "detectCloudflare": true
  }
}

💬 Feedback & Reviews

Found a routing edge case or want support for a new extraction pattern? Open an issue or leave a review on Apify. Feedback helps improve strategy detection, extraction quality, and marketplace usability.

📤 Output Example

📤 Example Dataset Output

Each dataset item includes structured content plus routing diagnostics:

{
  "url": "https://example.com/article/123",
  "finalUrl": "https://example.com/article/123",
  "title": "Example Title",
  "detectedEntityType": "article",
  "crawlStrategyUsed": "browser",
  "routeReason": "Detected SPA hydration markers and deferred content rendering",
  "cloudflareDetected": true,
  "challengeSuspected": false,
  "protectionLevel": "medium",
  "markdown": "# Example Title\n\nArticle content...",
  "text": "Example Title Article content...",
  "links": [],
  "timestamp": "2026-03-16T10:00:00.000Z"
}

⚠️ Limitations

• Does not bypass CAPTCHAs or authentication walls
• Success depends on target site structure and crawl permissions
• Browser rendering increases runtime and cost
• Some protected sites may be skipped with diagnostics instead of forced crawling