SERP Scraper — Google, Bing, Yahoo, DuckDuckGo, Yandex, Baidu, Ecosia

Scrape organic search results from 7 search engines in one run. Pass your queries, pick engines, get clean structured JSON: title, URL, snippet, rank, page, engine tag and the engine's estimated total-results count per result. Per-engine language/region parameters, native pagination, optional desktop/mobile device targeting, best-effort related-searches extraction on the engines that expose it, and best-effort Yandex AI-answer / entity-card extraction. No API keys, no login, no browser.

Engine parsers ported from the MIT-licensed openserp project (see NOTICE).

What SERP data does this scraper extract?

Each result is one flat JSON record per organic (or ad, if enabled) result:

Field	Meaning
query	The search query this result came from
engine	Engine that returned it: google, bing, yahoo, duckduckgo, ecosia, yandex, baidu
rank	1-based organic rank within (query, engine), across pages
page	1-based SERP page number
title	Result title as shown on the SERP
url	Target URL (redirect wrappers unwrapped where possible)
snippet	Description / snippet text
ad	true when the result is a sponsored/ad slot
serpUrl	The exact SERP URL that was fetched (for reproducibility)
device	Device profile used for the request: desktop or mobile (echoes the device input; see Device targeting below)
resultsTotal	The engine's own estimated total number of matches for the query (e.g. Bing's "About 1,230,000 results"). Best-effort — populated for google, bing, baidu, yandex when the SERP shows a count header; null for duckduckgo / ecosia / yahoo (no count header) and whenever the header wasn't present
relatedQueries	Best-effort list of related-search strings found on that SERP page. [] when the engine doesn't expose this or the run used desktop/mobile markup the parser didn't recognize — never guessed/fabricated. Currently populated for Google only; see Related searches & People Also Ask below
answer	Best-effort text of the engine's AI answer. Only ever populated for yandex (its "neuro" answer); null on every other engine. Yandex needs residential proxy from datacenter IPs, so it's null on unproxied runs
entityCard	Best-effort Yandex entity/fact card ({title, description, url}) — the Wikipedia-style knowledge panel. Only ever populated for yandex; null elsewhere and when absent

Every field above is present on every row (flat schema, no nested objects) — this keeps CSV/Excel exports clean.

How to scrape search engine results with this Actor

1. Click Try for free / Run — no login to any search engine, no cookies.
2. Enter one or more queries, tick the engines you want, optionally set languageCode / region / maxPagesPerQuery.
3. Run it and export the dataset as JSON, CSV or Excel, or read it over the API.

Run it from your own code:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("nomad-agent/serp-multi-engine-scraper").call(run_input={
    "queries": ["best python web framework"],
    "engines": ["bing", "duckduckgo", "baidu"],
    "maxPagesPerQuery": 2,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["engine"], item["rank"], item["title"], item["url"])

Or a single HTTP call that runs the Actor and returns items in one response:

curl -X POST \
  "https://api.apify.com/v2/acts/nomad-agent~serp-multi-engine-scraper/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"queries": ["best python web framework"], "engines": ["bing", "duckduckgo"]}'

Integrations: export the dataset as JSON, CSV or XLSX from the Console; call it synchronously with run-sync-get-dataset-items as shown above; wire it into Make, Zapier or n8n via their Apify connector; or drive it from any AI agent through the Apify MCP server.

Input

Field	Type	Default	Notes
queries	array	—	Search queries, one per entry. Inline operators like site: and filetype: work. Required.
engines	array	["bing","duckduckgo"]	Any of google, bing, yahoo, duckduckgo, ecosia, yandex, baidu.
device	string	"desktop"	desktop or mobile. Only changes behavior on Google and Bing — see Device targeting. Other engines accept and ignore it.
languageCode	string	""	BCP-47 code (en, de, pt-BR). Mapped per engine: Google hl/gl/lr, Bing mkt/setlang/cc, DuckDuckGo kl, Yandex lang: operator, Yahoo vl.
region	string	""	Country hint: two-letter code for Google (gl) / Bing (cc); numeric lr id for Yandex (e.g. 213 = Moscow).
maxPagesPerQuery	integer	1	SERP pages per query per engine, via each engine's native offset param. Stops early on empty/blocked pages. Max 10.
resultsPerPage	integer	0	Results per page where supported (Google num, Bing count, Baidu rn, Yahoo n). 0 = engine default. Max 50.
maxItems	integer	100	Hard cap on total results pushed across all queries and engines. Each result returned is billed as one dataset item — see Pricing below. 0 = no cap (costs then scale with queries × engines × pages — use with care). Max 1000.
includeAds	boolean	false	Include sponsored results (tagged "ad": true).
proxyConfiguration	object	off	Optional. Google/Ecosia/Yandex generally need residential proxy; Bing/DuckDuckGo/Baidu usually work without.
concurrency	integer	4	(Advanced) How many searches run at the same time. Higher is faster but more likely to get rate-limited. Extra pages within one search are always fetched one at a time with a short delay, regardless of this setting. Max 8.

Output example

Real push_data shape, captured from a live run:

{
  "query": "open source search engine",
  "engine": "duckduckgo",
  "rank": 1,
  "page": 1,
  "title": "Home - OpenSearch",
  "url": "https://opensearch.org/",
  "snippet": "OpenSearch is an open-source search and analytics suite…",
  "ad": false,
  "serpUrl": "https://html.duckduckgo.com/html/?q=open+source+search+engine&kl=us-en",
  "device": "desktop",
  "resultsTotal": null,
  "relatedQueries": [],
  "answer": null,
  "entityCard": null
}

relatedQueries is only ever non-empty on Google rows (see below) — it's [] on every other engine's rows, always, by design (not a bug). resultsTotal is null on DuckDuckGo/Ecosia (no count header on their endpoints); answer and entityCard are only ever populated on Yandex rows.

Pricing

Pay per event: $0.05 per Actor start and $0.004 per result returned. 100 results ≈ $0.45. No subscription, no rental — you pay only for what you fetch.

Use cases

• Rank tracking / SEO monitoring across engines, not just Google
• Comparing result overlap between Google, Bing, Yahoo, DuckDuckGo, Yandex, Baidu, Ecosia
• Regional / language SERP snapshots (languageCode + region)
• Feeding organic results into research, lead-gen or LLM pipelines
• Cheap keyword-to-URL discovery without a paid SERP API

Engines & reliability

Engine	Direct (no proxy)	Notes
Bing	✅	HTML first, automatic RSS-output fallback when Bing serves its JS bot-wall
DuckDuckGo	✅	Uses the JS-free html.duckduckgo.com endpoint
Baidu	✅	Real target URLs taken from the result mu attribute where present
Yahoo	✅	r.search.yahoo.com redirect wrappers unwrapped to the real target URL
Google	⚠️ needs residential proxy	Serves a JS interstitial / captcha to datacenter IPs
Ecosia	⚠️ needs residential proxy	Cloudflare Turnstile on datacenter IPs
Yandex	⚠️ needs residential proxy	SmartCaptcha on datacenter IPs; when reachable, best-effort answer (AI) and entityCard extraction

Blocked engines are logged and skipped — they never fail the run or charge you for results they didn't return.

Device targeting

The device input (desktop / mobile) is real, not a cosmetic toggle — but it only does something on the engines that actually respond differently to it:

Engine	device: mobile effect
Bing	Real, verified. Switches to a mobile browser User-Agent. This also happens to help get past Bing's JS bot-wall page that otherwise blocks datacenter IPs (mobile UA returned real results directly in testing; desktop UA needed the RSS fallback).
Google	Switches to a mobile browser User-Agent — the documented way to request Google's mobile SERP. Google blocks this Actor's requests at the interstitial stage from non-residential IPs regardless of device, so the effect on a real (proxied) run isn't independently verified here — use a residential proxy and compare.
DuckDuckGo	No-op. The html.duckduckgo.com endpoint this Actor uses is a single, JS-free layout with no separate mobile version. The field is accepted but ignored.
Yahoo / Ecosia / Yandex / Baidu	No-op. Not implemented — out of scope for this pass. The field is accepted but ignored; it never causes an error.

Related searches & People Also Ask

Engine	relatedQueries	peopleAlsoAsk
Google	Best-effort extraction of the "related searches" links at the bottom of the page. Not independently verified live (Google blocks this Actor's requests before parsing gets that far — see above), built on the structural pattern (internal /search?q=... links outside the organic results) that has stayed stable across Google's redesigns. Returns [], never errors, if the page layout doesn't match.	Not implemented. Google's "People also ask" box is populated by JavaScript after the page loads; a plain HTTP request (what this Actor uses) never receives it.
Bing	Not implemented. Verified live: Bing's related-searches/PAA widgets are lazy-loaded via JavaScript after the initial response — the static HTML this Actor fetches only ever contains the organic results.	Not implemented — same reason.
DuckDuckGo	Not implemented. The JS-free html.duckduckgo.com endpoint never renders a related-searches section — that's a duckduckgo.com JS-app-only feature.	Not implemented — same reason (no PAA on the JS-free endpoint).
Yahoo	Not implemented. Yahoo's "Also Try" / related widgets are JavaScript-lazy-loaded like Bing's — absent from the static HTML this Actor fetches.	Not implemented — same reason.
Ecosia / Yandex / Baidu	Not implemented — out of scope for this pass. Yandex answer (AI) and entityCard are extracted separately, see the output fields.	Not implemented — out of scope for this pass.

peopleAlsoAsk is intentionally not a dataset field: since it isn't populated on any engine, adding it would just be a permanently-empty column. relatedQueries is a real field ([] by default, populated on Google when the block is present).

FAQ

Is it legal to scrape search engine results? This Actor reads only publicly available SERP data — what any visitor sees without logging in. No personal data behind authentication is touched. Review the target engines' terms and your local regulations for your specific use case.

Do I need an account or API key? No. Results are fetched from public SERP pages — no login, no API keys, no cookies.

Why does Google return no results? Google (and Ecosia, Yandex) aggressively challenge datacenter IPs. Enable Apify residential proxy in proxyConfiguration. Bing, DuckDuckGo, Baidu and Yahoo normally work without any proxy.

How many results can I get? Up to maxPagesPerQuery pages per engine per query (~10 results/page, ~25 for DuckDuckGo), capped by maxItems (default 100, max 1000). Each result is billed as one dataset item — see Pricing above.

Are ads included? Only when includeAds is on; ad rows are tagged "ad": true and never consume organic ranks.

Can I get mobile search results? Yes for Bing (and best-effort for Google): set device to mobile. It's a no-op on Yahoo, DuckDuckGo, Ecosia, Yandex and Baidu — see Device targeting.

Does this Actor extract related searches or People Also Ask? relatedQueries is populated on Google rows on a best-effort basis. peopleAlsoAsk isn't implemented on any engine — it's rendered by JavaScript after the page loads, which a plain HTTP fetch (what this Actor uses) never receives. See Related searches & People Also Ask.

Something broken or missing? Search engines change their markup often. Open an issue on the Actor's Issues tab — it is monitored and reliability fixes ship fast.

Attribution

SERP URL construction and CSS selectors are a Python port of openserp by Rustem Kamalov, MIT License. See LICENSE and NOTICE.

Related Actors

• AI Job Search Agent — Web Job Finder (BYOK)
• LinkedIn Jobs Scraper — No Login, No Cookies
• Hacker News Who's Hiring Scraper