Under maintenance

Pricing

Pay per usage

Try for free

Go to Apify Store

Web Search & Scrape by XCrawl Proxy

Under maintenance

Try for free

Search the web or scrape any URL using XCrawl residential proxy network. Bypass anti-bot systems with automatic JS rendering fallback, global IP rotation, and configurable concurrency (1-20). Perfect for market research, LLM data collection, and content aggregation.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Charles

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Web Search & Scrape by XCrawl Proxy — Apify Actor

Search the web or scrape any URL using XCrawl's residential proxy network. This Actor combines Google-quality search with intelligent anti-bot page scraping — all in one call.

Why This Actor?

Feature	Benefit
Search + Scrape in One	No need to chain separate tools. Search results are automatically enriched with full page content.
Residential IP Rotation	Every request goes through true residential proxies — far fewer blocks than datacenter IPs.
Smart Anti-Bot Retry	Automatically detects Cloudflare, DataDome, captcha pages and retries with JS rendering.
Configurable Concurrency	1–20 parallel scrapes — tune for speed vs. stealth.
30+ Block Heuristics	Detects and bypasses the most common anti-scraping systems.
Global Geo-Targeting	Search from US, UK, JP, DE, AU — get localized results.
Markdown Output	Clean markdown content, not messy HTML — ready for LLM consumption.

Quick Start

1. Get an XCrawl API Key

2. Set Environment Variable

XCRAWL_API_KEY=your-api-key-here

3. Run the Actor

Search the web:

{
  "action": "search",
  "query": "latest AI startup funding 2026",
  "location": "US",
  "limit": 5,
  "withContent": true
}

Scrape a single page:

{
  "action": "scrape",
  "url": "https://example.com/article",
  "formats": "markdown,summary"
}

Input Parameters

Parameter	Type	Default	Description
`action`	`"search"` or `"scrape"`	`"search"`	Choose operation mode
`query`	string	—	Search query (required for `search`)
`url`	string	—	Target URL (required for `scrape`)
`location`	string	`"US"`	Geo-location code (US, UK, CN, JP, DE, AU...)
`language`	string	`"en"`	Search language code
`limit`	integer	`10`	Max results (1–50)
`withContent`	boolean	`true`	Auto-fetch full page content for each search result
`formats`	string	`"markdown,summary"`	Comma-separated: `markdown`, `summary`, `html`
`render`	boolean	`false`	Enable JS rendering for heavily protected sites
`screenshot`	boolean	`false`	Capture PNG screenshot (requires `render=true`)
`concurrency`	integer	`5`	Parallel scrapes (1–20)

Output

Each result contains:

{
  "title":      "Page title",
  "url":        "https://...",
  "snippet":    "Search snippet or description",
  "markdown":   "Full page content in markdown (up to 100K chars)",
  "summary":    "AI-generated page summary",
  "status":     "completed" | "failed",
  "credits":    "XCrawl credits used",
  "source":     "search" | "scrape"
}

Use Cases

Market Research — Scrape competitor pages, track pricing
LLM Training Data — Clean markdown ready for fine-tuning
Content Aggregation — Build news/trending feeds
Lead Generation — Search + scrape prospect pages in one workflow
SEO Monitoring — Track SERP positions and page changes

Pricing

This Actor bills via Apify platform usage plus XCrawl API credits (charged by dash.xcrawl.com). The XCrawl free tier is generous enough to get started — no upfront payment needed.

Environment Variables

Variable	Required	Source
`XCRAWL_API_KEY`	✅	dash.xcrawl.com

Links

Built on XCrawl — the web scraping proxy platform with built-in anti-bot protection.

Xcrawl Search Scrape Actor

empathetic_chorus/xcrawl-search-scrape-actor

Charles

Universal AI Web Scraper

stanvanrooy6/universal-ai-web-scraper

Turn any website into an API. Extract structured data using plain English. Features anti-bot bypass, dynamic rendering, and web search. No coding needed.

Stan Van Rooy

1.5

Firecrawl MCP

red.cars/firecrawl-mcp

AI agents that need web data without anti-bot headaches. 20 tools for API-based web scraping, crawl, search, and extract — no proxy rotation, no stealth needed.

AutomateLab

Article Extractor & News Scraper

web.harvester/article-extractor-news-scraper

Extract articles from any news site, blog, or webpage. Get title, full text, author, date, images & metadata using 7 extraction engines (Newspaper4k, Trafilatura, Goose3). Anti-bot bypass, proxy rotation, automatic fallback. Perfect for news monitoring, NLP datasets & content aggregation.

Web Harvester

5.0

Web Article Content Extractor

vulnv/web-article-content-extractor

Extract clean, readable content from news articles, blog posts, and web pages. Batch process multiple URLs, download images, bypass bot protection with proxy support. Perfect for content curation, research, and data analysis.

VulnV

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Scrape GPT - Universal AI Web Scraper Agent

paradox-analytics/scrape-gpt---universal-ai-web-scraper-agent

AI-powered universal web scraper that works on ANY website without configuration. Extract data from e-commerce, news sites, social media, and more using intelligent LLM-based field mapping. Features JSON-first extraction, automatic pagination, anti-bot bypass, and cost-effective caching.

Paradox Analytics

LinkedIn Company Data Scraper API

moving_beacon-owner1/my-actor-7

Scrape company profiles, industries, employees, and more with scalable concurrency and anti-blocking proxy support.

Jamshaid Arif

Firecrawl AI-Powered Web Search & Scrape

alizarin_refrigerator-owner/firecrawl-ai-powered-web-search-scrape

Search the web and get clean, LLM-ready content in one API call. Powered by Firecrawl's /v1/search endpoint. Returns markdown, HTML, or extracted data. Perfect for SEO research, competitor analysis, and AI training data collection.