All Web Search Scraper

All Web Search Scraper is a multi engine web search that sends one query to ten search platforms and returns each platform's ranked results in one Dataset. Every row keeps the platform and position that produced it, and can include the destination page as Markdown for research, monitoring, and retrieval workflows.

• Ten search platforms in one run — Google, Bing, Baidu, Yandex, Naver, Sogou's WeChat search, Seznam, So.com, DuckDuckGo, and Brave share one input and one output contract.
• Every platform's own ranking, kept — a page three platforms returned is three rows, each with that platform's real position. Nothing is collapsed, so no rank is lost or guessed.
• Paging that actually pages — nine of the ten support a page parameter and the run walks it, so a request for more results than one page holds returns more than one page's worth.
• Page text travels with every readable result — each returned URL is fetched, content_status records the outcome, and readable pages include Markdown with a word count, so a retrieval pipeline needs one run instead of two.

The smallest useful run is one result: $0.04 for the Actor Start plus $0.00200 for the row, or $0.04200 in total on the FREE tier.

Why Choose All Web Search Scraper

• Regional reach a single-engine tool cannot reproduce. Baidu and So.com serve the Chinese web, Yandex the Russian web, Naver the Korean web, Seznam the Czech web, and Sogou's WeChat search reaches public articles within that ecosystem. A query run across all ten can surface pages that one platform alone misses.
• Agreement is measurable and checkable. Group the Dataset by url and the count shows how many platforms returned that page, with each platform's position beside it rather than averaged away.
• Source overlap stays visible instead of being oversold. Brave documents an independent index, while DuckDuckGo says its traditional links are largely sourced from Bing. The Actor preserves both result sets and their platform labels so you can measure overlap yourself.
• Redirecting links are resolved to the real page. Several platforms publish a tracker address instead of the destination; where the destination can be recovered it is, so rows from different platforms line up on the same url and can be grouped.
• The page text ships with the result, at no extra charge. Every row carries the page as Markdown with a word count and a content_status saying whether it could be read. A page several platforms returned is downloaded once and attached to each of those rows, so breadth costs requests only where it adds pages.

Quick Start Guide

1. Configure

Open the Actor. keyword, max_results and platforms arrive prefilled with a working example, so the first run needs no editing. Type your search phrase into keyword, raise max_results, and either pick specific platforms in platforms or clear the field to search all ten.

2. Run

Press ▷ Start. All selected platforms are searched at the same time, each one paging until it has enough results or stops producing new URLs. Each distinct page is then read into Markdown, and every row is written the moment its page comes back, so rows arrive in reading order rather than platform order — each one carries its platform and position to sort by afterwards. The log prints each saved result and names any platform that produced nothing.

3. Collect

Open the Dataset tab and use the Overview view, or export to JSON, CSV, XLSX, XML or HTML. Group by url to see which pages the most platforms agreed on, sort by position within a platform to read that platform's ranking, or filter content_status to ok for the rows whose page text came back.

Input Parameters

The Actor takes 3 inputs, of which keyword and max_results are required; platforms narrows the search to a subset of the ten.

Parameter	Type	Required	Description	Example
keyword	string	Yes	The search phrase, sent to every selected platform exactly as typed.	open source ai
max_results	integer	Yes	How many rows to return per platform. Minimum 1; the run pages each platform until it has them.	1
platforms	array	No	Which platforms to search, from 10 options: google, bing, baidu, yandex, naver, sogou_weixin, seznam, so360, duckduckgo, brave. Empty searches all ten.	["brave"]

{
  "keyword": "open source ai",
  "max_results": 1,
  "platforms": ["brave"]
}

max_results is a per-platform budget rather than a run total, so ten platforms at 20 can return up to 200 rows. A page returned by three platforms remains three rows because each row carries a different ranking. A platform name this build no longer serves is skipped with a printed notice while valid selections continue; the run ends early only when nothing in platforms is supported.

Output Data Schema

One row is one result from one platform, with the destination page read when available. Every row follows the same 13-field contract.

Field	Type	Description
title	string or null	The result headline as the engine rendered it
snippet	string or null	The description the engine printed under the result, when it printed one
url	string or null	The result address, and the key to group on to compare platforms
source_name	string or null	Publisher or site name shown by the platform, when available
date_text	string or null	Freshness text exactly as the platform displayed it, when available
platform	string or null	Which platform returned this result
position	integer or null	Organic rank on that platform; paid placements use null
is_ad	boolean or null	Whether the platform marked the row as a paid placement
markdown	string or null	The result page converted to Markdown when readable
word_count	integer or null	Words in that Markdown rendering
content_status	string or null	Whether the page could be read: ok, empty, too_large, unconvertible, or unavailable
processor	string or null	The Actor URL that produced the row
processed_at	string or null	UTC timestamp when the row was produced

This sample is abbreviated: the markdown body is truncated to one line. Ten selected platforms with max_results: 12 can return up to 120 rows.

{
  "title": "Open Source AI – Open Source Initiative",
  "snippet": "Open Source AI gives developers access to resources for context-specific applications.",
  "url": "https://opensource.org/ai",
  "source_name": "Open Source Initiative",
  "date_text": "May 28, 2025",
  "platform": "brave",
  "position": 1,
  "is_ad": false,
  "markdown": "# Open Source AI\n\nOpen source enables diverse communities ...",
  "word_count": 1432,
  "content_status": "ok",
  "processor": "https://apify.com/username/actor_name",
  "processed_at": "2026-09-01T13:43:54+00:00"
}

A value the engine did not print stays empty rather than being filled in: snippet is absent on results that genuinely carry no description, such as image and video hits, and markdown is empty whenever content_status is anything other than ok. Export formats: JSON, JSON Lines, CSV, XLSX, XML, RSS and HTML table.

Integration Examples

Every example here runs the same verified scenario — the keyword open source ai against Brave for one result — using the Actor ID pa0BDQPgWUaFPghFw or its name form agentx/all-web-search-scraper, so the input you test in the console is the input your code sends.

Actor ID

pa0BDQPgWUaFPghFw

The name form agentx/all-web-search-scraper works in the same positions as pa0BDQPgWUaFPghFw in API clients, integrations, and MCP calls.

HTTP

Start the run, poll it until it finishes, then read the Dataset. This asynchronous flow is the path to use in production, because a run is never bounded by the lifetime of one HTTP connection (Apify runs and builds).

RUN=$(curl -s -X POST "https://api.apify.com/v2/acts/agentx~all-web-search-scraper/runs?token=APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keyword":"open source ai","max_results":1,"platforms":["brave"]}' \
  | python -c "import sys,json; print(json.load(sys.stdin)['data']['id'])")

curl -s "https://api.apify.com/v2/actor-runs/$RUN?token=APIFY_TOKEN"

curl -s "https://api.apify.com/v2/actor-runs/$RUN/dataset/items?token=APIFY_TOKEN"

Poll the second call until data.status is SUCCEEDED, or register an ACTOR.RUN.SUCCEEDED webhook and skip polling entirely.

Quick test only: POST /v2/acts/agentx~all-web-search-scraper/run-sync-get-dataset-items returns rows on one connection, and is safe here only because a one-result run finishes well under the limit. The platform cuts a sync connection at 300 seconds without aborting the run, a client-side timeout never stops a billed run, and every client or gateway retry starts another billed run.

Python

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("agentx/all-web-search-scraper").start(run_input={
    "keyword": "open source ai",
    "max_results": 1,
    "platforms": ["brave"],
})
run = client.run(run["id"]).wait_for_finish()
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["platform"], item["position"], item["title"], item["url"])

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'APIFY_TOKEN' });
let run = await client.actor('agentx/all-web-search-scraper').start({
  keyword: 'open source ai',
  max_results: 1,
  platforms: ['brave'],
});
run = await client.run(run.id).waitForFinish();
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((row) => [row.platform, row.position, row.url]));

Make.com

1. Add the Apify → Run an Actor module.
2. Paste pa0BDQPgWUaFPghFw into the Actor field and leave "Wait for the Actor to finish" enabled so the scenario resumes on completion rather than on connection close.
3. Put {"keyword":"open source ai","max_results":1,"platforms":["brave"]} into the Input JSON field, replacing the query and raising max_results.
4. Chain Apify → Get Dataset Items and map title, snippet, url, platform, position and markdown into your scenario.

n8n

1. Add the Apify node and choose the Run Actor operation.
2. Set the Actor to pa0BDQPgWUaFPghFw.
3. Paste {"keyword":"open source ai","max_results":1,"platforms":["brave"]} as Custom Body, and enable the node's wait-for-finish option.
4. Add a second Apify node with Get Dataset Items to pull the 13-field rows into the workflow.

MCP

Point an MCP client at https://mcp.apify.com, then call the Actor by its name form:

{
  "actor": "agentx/all-web-search-scraper",
  "input": {
    "keyword": "open source ai",
    "max_results": 1,
    "platforms": ["brave"]
  }
}

Pricing

Each result row costs $0.00200 on the FREE tier and each run adds $0.04 of Actor Start, so the smallest useful run — one result — totals $0.04200.

Event	FREE	BRONZE	SILVER	GOLD	PLATINUM	DIAMOND
Result (one row)	$0.00200	$0.00180	$0.00160	$0.00150	$0.00150	$0.00150
Actor Start (per GB of run memory)	$0.01	$0.01	$0.01	$0.01	$0.01	$0.01

Actor Start is charged once per gigabyte of run memory, with a minimum of one event. This Actor runs at 4 GB, so every run pays four Actor Start events: 4 × $0.01 = $0.04, the same on every tier. Result charges apply only to rows delivered to the default Dataset.

max_results is per platform, so ten platforms at 12 have a 120-row ceiling. If all 120 rows are delivered, the FREE-tier charge is $0.04 plus 120 × $0.00200, or $0.28000. A page several platforms returned is several rows and is charged per row, because each one carries a different platform ranking. Reading the result page carries no separate event charge, and a page returned by several platforms is downloaded once, so one run across ten platforms costs less than ten single-platform runs that each pay their own Actor Start.

Prices can change; the live figures are always on the Actor pricing page.

Use Cases

• Cross-market SEO visibility. Run a brand or product term across all ten platforms and filter the Dataset to your own domain: every row is one platform's real ranking for it, so a pivot on platform against position is a rank table no single-platform tracker can produce.
• Grounding data for retrieval pipelines. Every run returns the URL list and each page's Markdown together; count how many platforms returned a url to get a corroboration score before the page enters a context window.
• Regional research. A question about the Chinese, Russian, Korean or Czech web is answered far better by the indexes that crawl it, and this Actor reaches five such indexes in one call.
• Competitor discovery outside your bubble. Different crawlers surface different long-tail sites; the pages returned by exactly one index are often the most interesting part of the result set.
• Broken-narrative and reputation checks. Search a claim or a name across ten indexes and look at what only one of them holds, which is where fringe or newly published material tends to appear first.

Alternatives

Searching each platform by hand is realistic for one query and one platform, and hopeless for ten. It also gives you no rank comparison and no export.

The platforms' own source disclosures are worth reading before you compare overlap: Brave documents its independent index, while DuckDuckGo explains that its traditional links are largely sourced from Bing. Going to a platform directly is better when you need its advanced operators or vertical results.

Single-platform SERP tools return far more per result — ads, related questions, featured panels, per-country rank — for one platform at a time. When your question is about one market and one platform, that depth is worth more than breadth. This Actor is the opposite trade: fewer columns, ten platforms, one call.

Choose something else if you need shopping listings, People Also Ask panels, featured snippets, knowledge panels, or per-country and per-language targeting. None of those are returned here. Sponsored results are returned when a platform prints them: they carry is_ad: true and a null position, so they never enter the organic ranking, and one filter on is_ad leaves you the organic set.

Limits and Troubleshooting

• Fewer rows than max_results → a selected platform had no more distinct result URLs for that query → the log prints the count. Each platform stops when it reaches its own limit or a page yields no new URLs.
• The log names an index with no results → that index produced nothing for this run → it is reported separately from an index that answered with zero matches. Coverage varies by query and by market, and the run tells you which indexes actually contributed.
• sogou_weixin returns unfamiliar results → that index covers WeChat articles rather than the general web → deselect it if you want web pages only. It is included because it reaches material no general crawler holds, and excluded from a run in one click.
• DuckDuckGo returns only one page of results → its lightweight interface has no page-forward for this kind of request → select other indexes for depth. This is stated rather than worked around, so the row count is never quietly capped without explanation.
• A run returned sponsored rows instead of the ranking you expected → the platform printed its ads above the organic block and max_results filled from the top of the page → raise max_results, or filter the export on is_ad. Sponsored rows count toward max_results and are charged like any other row; they carry is_ad: true and a null position, so which rows they are is never a guess.
• A URL still looks like a platform's own address → a few platforms publish a redirecting link whose destination cannot be recovered without following it → the link still reaches the page. Where the destination is recoverable it is unwrapped before the row is written, so rows from different platforms share the same url and group together.
• content_status is not ok → the page was unavailable, too large, empty, or held no convertible text → keep the search row and filter on content_status when Markdown is required. A content failure does not erase the platform's search result.
• Results differ between two runs of the same query → search indexes are not stable → that is the source's behaviour, not the Actor's. Ranking changes constantly, which is exactly why processed_at is stamped on every row.

Found something reproducible? Open an Issue on the Actor's Issues tab with the run ID, the exact input JSON, and the platforms you selected.

Trust and Reliability

The Actor runs on Apify's platform with the run history, logs, Dataset retention and scheduling that come with it. Billing is pay-per-event: the Actor Start events a run pays for its memory, and one charge per row that reaches the Dataset. The Dataset contract has 13 fields and every one is produced by the run; source-specific fields stay empty when a platform does not show them. Because each row names the platform that returned it and the position it held there, every ranking can be checked against that platform directly.

Legal and Compliance

Data scope. The Actor reads publicly available search result pages and extracts the outbound links, headlines and descriptions they display. It also fetches publicly reachable destination pages for Markdown enrichment. No result is stored beyond your own Dataset, and no search index is redistributed.

Privacy. A search query and its results may reference identifiable people. Do not use the output for harassment, stalking, surveillance, doxxing, spam, impersonation or unlawful profiling of a person, and apply your own retention rules to Datasets built from name-based queries.

Platform terms. Use of this Actor is subject to the Apify Terms of Service and to each search provider's own terms. Result rankings, titles and inclusion decisions belong to the platforms that produced them; the Actor preserves each platform's position and does not present a blended ranking.

Frequently Asked Questions

How do i search bing baidu and yandex in one run?

Leave platforms empty to search all ten, or set it to ["bing","baidu","yandex"] for exactly those three. All selected platforms are searched at the same time and each returns its own ranked rows, so a page all three found arrives as three rows and you can see the rank it earned on each.

Can I search multiple search engines at once?

Yes — that is the whole design. Ten platforms are queried in parallel from one run and their results arrive as rows in one Dataset, each tagged with the platform that returned it and the rank it held there. You get one export and one response shape instead of ten.

Is there a free web search API?

Some platforms publish official APIs with free tiers, and they are worth using when one platform is enough. What this Actor sells is the breadth in one call: ten result sets in one Dataset with the paging handled per platform. Assembling that yourself means ten different response shapes and ten different page parameters.

Can I schedule runs to monitor search results over time?

Yes. Apify's Scheduler runs this Actor on a cron expression, and every row is stamped with processed_at, so consecutive runs build a rank history. Watching position per platform for your own URLs across runs is a low-cost visibility tracker that covers markets a single-platform tool cannot see.

How do I see which pages several platforms agreed on?

Group the Dataset by url. A page returned by four platforms is four rows sharing that url, each carrying its own platform and position, so the group size is the agreement count and the rows show the rank each platform gave it. Nothing is collapsed for you, so the number is one you can verify rather than one the Actor asserts.

How many results can one run return?

max_results is a per-platform budget, so ten platforms at 20 return up to 200 rows, subject to what those platforms actually hold. Each one pages until it stops producing new URLs. Rows are not deduplicated across platforms, so the count is predictable: what you get is what each platform returned.

Do I get the page text as well as the link?

Yes, on every run and at no extra charge. Each returned result is fetched and converted to Markdown with a word_count, and content_status says whether the page could be read. Only pages the run is going to return are downloaded, and a page several platforms returned is downloaded once and attached to each of its rows.

Why are some results missing a snippet?

Because the engine did not print one. Descriptions are read from the text the index shows under each result, and image, video and some navigational hits genuinely carry none. The column is left empty rather than filled with the title again, so an empty snippet is a fact about that result and not a parsing failure.

Related AgentX Actors

AgentX publishes 79 Actors on Apify; this section names the three closest to this one first, then lists the full brand catalogue by category so you can find the rest without leaving the page.

Closest to this Actor:

• Google Keyword Trends — supplies the demand side of a keyword that this Actor answers the supply side of, by showing who currently ranks for it.
• Google Trends Scraper — surfaces the rising queries that are worth running through ten indexes before anyone else does.
• Medium Scraper — turns the article URLs a search run surfaces into full records for analysis.

Business and Market Intelligence

• Google Keyword Trends — Google Trends keyword interest, compared five at a time
• Google Trends Scraper — Google Trends trending searches
• Google Maps Store Scraper — Google Maps business listings
• LinkedIn Company Lookup — LinkedIn company records
• LinkedIn Profile Lookup — LinkedIn profile records
• Weather Forecast API — forecast and historical weather data

Jobs and Hiring

• All Jobs Scraper — multi-platform job listings
• Bayt Jobs Scraper — Bayt job listings
• Glassdoor Jobs Scraper — Glassdoor job listings
• Indeed Jobs Scraper — Indeed job listings
• JobStreet Company Profile — JobStreet employer records
• Jooble Jobs Scraper — Jooble job listings
• LinkedIn Jobs Scraper — LinkedIn job listings
• Naukri Jobs Scraper — Naukri job listings
• Talent Jobs Scraper — Talent.com job listings
• ZipRecruiter Jobs Scraper — ZipRecruiter job listings

Social Media

• All Forums Thread Scraper — multi-platform forum and Q&A threads
• Instagram Creator Scraper — Instagram creator profiles
• Instagram Reels Scraper — Instagram Reels metadata
• Instagram Trending Scraper — Instagram trending content
• Medium Scraper — Medium articles
• Medium Profile Scraper — Medium author profiles
• Reddit Profile Scraper — Reddit user profiles
• Reddit Viral Scraper — Reddit viral posts
• SubReddit Info Scraper — subreddit metadata
• Subreddit Members Scraper — subreddit member lists
• Subreddit Posts Scraper — subreddit post feeds
• Telegram Chat Scraper — Telegram chat messages
• Telegram Info Scraper — Telegram channel metadata
• Telegram Member Adder — Telegram group membership management
• Telegram Member Scraper — Telegram group members
• Telegram Private Group Scraper — private Telegram group data
• TikTok Creator API — TikTok creator profiles
• TikTok User Lookup — TikTok account records
• X Twitter Community API — X community data
• X Twitter Profile Lookup — X profile records
• YouTube Creator Email Scraper — YouTube creator contact data

Video, Transcripts and Downloads

• All Video Scraper — multi-platform video metadata
• Video Transcript — multi-platform transcripts
• Video Captions Downloader — caption files
• Video to Social Post — video repurposed into social copy
• YouTube Transcript — YouTube transcripts
• TikTok Transcript — TikTok transcripts
• X Twitter Transcript — X video transcripts
• Facebook Transcript — Facebook video transcripts
• Bilibili Transcript — Bilibili transcripts
• Dailymotion Transcript — Dailymotion transcripts
• Rutube Transcript — Rutube transcripts
• Loom Transcript — Loom transcripts
• Wistia Transcript — Wistia transcripts
• Instagram Reels Downloader — Reels downloads
• Kick Clip Downloader — Kick clip downloads
• Linkedin Video Downloader — LinkedIn video downloads
• Pinterest Video Downloader — Pinterest video downloads
• Reddit Video Downloader — Reddit video downloads
• Snapchat Video Downloader — Snapchat video downloads
• TED Talk Downloader — TED talk downloads
• TikTok Live Downloader — TikTok live downloads
• Twitch VOD Downloader — Twitch VOD downloads
• Zoom Recording Downloader — Zoom recording downloads

E-Commerce and Retail

• All Shopping Scraper — multi-platform product data
• AliExpress Product Scraper — AliExpress products
• Amazon Storefront Scraper — Amazon Brand Store pages
• Bol Product Scraper — Bol.com products
• eBay Seller Leads API — eBay seller records
• Hepsiburada Product Scraper — Hepsiburada products
• Kakaku Product Scraper — Kakaku.com products
• Rakuten Product Scraper — Rakuten products

Classifieds and Automotive

• All Vehicle Scraper — multi-platform vehicle listings
• AutoTrader Vehicle Search — AutoTrader US listings
• AutoTrader UK Vehicle Search — AutoTrader UK listings

Real Estate

• All Property Scraper — multi-platform property listings
• Homes Property Scraper — Homes.com listings
• MagicBricks Property Scraper — MagicBricks listings
• NoBroker Property Scraper — NoBroker listings
• Property24 Property Scraper — Property24 listings
• Realtor Property Scraper — Realtor.com listings
• Zillow Property Scraper — Zillow listings

Support and Community

Ask about platform coverage, per-platform ranking or paging in the AgentX community on Telegram; for a reproducible bug, open an Issue with the run ID and the exact input.

AgentX is an Arcyton brand — arcyton.com.

Last Updated: September 3, 2026