Pricing

Pay per usage

Stealth Website Crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Nocturne

Actor stats

Bookmarked

Total users

Monthly active users

24 days ago

Last modified

Categories

Automation

Developer tools

URLs

startUrls

Required

URLs to process.

Type:array

Actions (interactive mode)

actions

Optional

Sequence of actions to perform on each page. If provided, runs in interactive mode. Types: click, fill, type, scroll, hover, select, wait, screenshot, javascript, extractHtml, extractContent, captureNetwork, humanActivity, mouseMove.

Type:array

Capture network responses

captureNetwork

Optional

Record JSON/API responses made by the page. Useful for intercepting data APIs.

Type:boolean

Default:false

Max pages to crawl

maxCrawlPages

Optional

Maximum pages to crawl (crawl mode only).

Type:integer

Minimum:1

Maximum:10000

Default:10

Max crawl depth

maxCrawlDepth

Optional

How many links deep to follow from start URLs. 0 = start URLs only.

Type:integer

Minimum:0

Maximum:100

Default:20

Content output format

outputFormat

Optional

Format for extracted content.

Type:string

Default:markdown

Options:

markdownhtmltext

Follow links

followLinks

Optional

Follow same-domain links to discover more pages (crawl mode).

Type:boolean

Default:true

Include URL patterns

includeUrlGlobs

Optional

Only crawl URLs matching these glob patterns. Example: https://example.com/blog/*

Type:array

Exclude URL patterns

excludeUrlGlobs

Optional

Skip URLs matching these glob patterns. Example: /login

Type:array

Wait for selector

waitForSelector

Optional

CSS selector to wait for before processing each page. Useful for JS-heavy sites.

Type:string

Take screenshots

takeScreenshots

Optional

Capture a screenshot of each page. Stored in key-value store.

Type:boolean

Default:false

Block images, fonts, CSS

blockResources

Optional

Block images, media, fonts, and stylesheets to reduce proxy bandwidth and speed up crawling.

Type:boolean

Default:false

Initial cookies

initialCookies

Optional

Cookies to set before crawling. Format: [{name, value, domain, path}]

Type:array

Max concurrent pages

maxConcurrency

Optional

Number of pages to load simultaneously. Lower = safer stealth, higher = faster.

Type:integer

Minimum:1

Maximum:10

Default:3

Request timeout (seconds)

requestTimeoutSecs

Optional

Max time to wait for a page to load.

Type:integer

Minimum:5

Maximum:120

Default:30

Max retries per page

maxRetries

Optional

Number of retry attempts for failed pages. Failed pages are not charged.

Type:integer

Minimum:0

Maximum:5

Default:1

Proxy configuration

proxyConfig

Optional

Residential proxies strongly recommended for anti-bot sites.

Type:object

Default:

{
  "useApifyProxy": true,
  "apifyProxyGroups": [
    "RESIDENTIAL"
  ]
}

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Anti-Bot Bypass Service

h4sh/anti-bot-bypass

Bypass Cloudflare, DataDome, PerimeterX, Akamai and other anti-bot protections. Get clean HTML from any protected website using Camoufox stealth browser.

Dominique

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

5.0

Website Content Crawler for RAG

qingyun/website-content-crawler

Crawl documentation sites, help centers, blogs, and websites, then extract clean markdown, text, or HTML for RAG pipelines, vector databases, and LLM applications.

yun qing

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

mikolabs

Website Content Crawler

tugelbay/website-content-crawler

Crawl websites and extract clean Markdown/text content for RAG pipelines and LLMs. HTTP-first, 10x faster than browser-based crawlers.

Tugelbay Konabayev

Stealth Web Scraper

lentic_clockss/stealth-web-scraper

Scrape Cloudflare-protected and anti-bot websites with built-in residential proxies, stealth browser fingerprints, and Turnstile handling. Pay only for successful pages.

kane liu

Website Content Crawler for LLM's

salesblaster-ai/website-content-crawler

Extract contact information + turn any website into clean, structured content ready for LLM's (e.g. AI lead magnets, RAG pipelines, and outbound personalization). Most web scrapers dump raw HTML or unstructured text. This crawler is purpose-built for LLM's, and optimized for lead generation.

SalesBlaster AI

AI Content Crawler

kai-agent/ai-content-crawler

Crawl any website and get clean, AI-ready content in markdown format. Perfect for RAG pipelines, LLM training data, and vector database ingestion. Features smart chunking, metadata extraction, and multiple output formats.