AI Web Search Tool: Real-Time Search Results for Agents and Apps

AI Web Search Tool searches several engines, including DuckDuckGo, Google, Brave, Yahoo and Bing, with automatic fallback when one is unavailable, and returns web pages, images, videos, news or books as structured JSON. Feed it into an AI agent, a RAG pipeline or your own app, then read the results from the dataset, or pull the complete object, including run metadata, from the key-value store.

Why This Scraper?

• Multiple engines, automatic fallback. Web and news searches draw on several engines, including DuckDuckGo, Google, Brave, Yahoo and Bing, so one engine being unavailable does not stop the run.
• Five result types. Web pages, images, videos, news articles or books, switchable with a single field.
• Locale and freshness control. Pick a region and language code, and optionally restrict results to the last day, week, month or year.
• Safe search built in. Off, moderate or strict, set per run.
• Structured for machines. Every result is a flat JSON record ready for an LLM prompt, a RAG index or a spreadsheet, with no HTML to parse.
• Two ways to read the output. Individual records in the dataset, or one combined object with results and run metadata in the key-value store.
• Field shaping on demand. Trim each record down to only the fields you need with dot notation paths, no post-processing required.

Use Cases

• AI agents and RAG pipelines: feed live web, news or image results into an LLM's context so it can answer with current information.
• Fact checking and research: pull recent news and multiple sources on a claim before writing it up.
• Content and media discovery: find images, videos or news items on a topic for social posts, blogs or newsletters.
• Monitoring and reporting: run the same query on a schedule and export results into your own pipeline or spreadsheet.
• App and chatbot backends: power a "search the web" feature in your own product through the API.

Data You Get

Sample shape: values are illustrative placeholders, not from a live record.

Field	Example
position	1
title	"Understanding Artificial Intelligence"
link	"https://example.com/articles/understanding-ai"
snippet	"Artificial intelligence refers to computer systems that perform tasks typically requiring human intelligence."
domain	"example.com"
query	"artificial intelligence trends"
image	"https://example.com/media/photo.jpg"
thumbnail	"https://example.com/media/photo-thumb.jpg"
source	"example.com"
url	"https://example.com/articles/understanding-ai" (same value as link)
hostname	"example.com"
icon	"https://external-content.duckduckgo.com/ip3/example.com.ico" (site favicon)
rawDescription	same text as the description or snippet
image_token	"0123abcd..." (videos only)
height	720
width	1280
description	"A short clip explaining the topic."
duration	"5:32"
published	"2024-06-01"
publisher	"Sample Publisher"
date	"2024-06-01"
author	"Sample Author"
info	"English [en], .epub, 0.5MB"

Which fields show up depends on Result type: web results carry snippet and domain, news results carry snippet, date and source, images carry image, thumbnail, source, height and width, videos carry description, thumbnail, duration, published and publisher, and books carry author, publisher, info and thumbnail. position and query are on every record. Every record also carries url (same as link), description (same text as snippet where there is one), rawDescription, hostname (host of link), icon, image, height, image_token, excerpt (same as description), publishedOn (the published or date value) and viewCount (videos only); a field that does not apply to the mode is null, so the shape is the same across modes.

How to Use

1. Enter a Search query, the only required input.
2. Pick a Result type: web, images, videos, news or books.
3. Adjust Max results, Region, Safe search and Time range as needed.
4. Click Start, then read the results from the dataset, or fetch the combined object from the key-value store.

Basic web search:

{
  "query": "artificial intelligence trends",
  "maxResults": 10,
  "resultType": "web"
}

Recent news with a time filter:

{
  "query": "renewable energy policy",
  "resultType": "news",
  "timeRange": "w",
  "maxResults": 20
}

Image search with region and safe search:

{
  "query": "mountain landscape photography",
  "resultType": "images",
  "region": "uk-en",
  "safeSearch": "strict",
  "maxResults": 30
}

Web search through Apify Proxy:

{
  "query": "machine learning datasets",
  "resultType": "web",
  "maxResults": 50,
  "proxy": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

Run it from your code

Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("abotapi/ai-web-search-tool").call(run_input={"query": "artificial intelligence trends", "resultType": "web"})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["link"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('abotapi/ai-web-search-tool').call({ query: 'artificial intelligence trends', resultType: 'web' });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Or connect it to Make, Zapier, n8n, Google Sheets or webhooks from the Integrations tab.

Shaping and reusing output

• dev_transform_fields trims every record down to only the field paths you list, using dot notation for nested values, so you don't have to post-process the dataset yourself.
• dev_dataset_name saves a run into a named dataset instead of the default one, with {ACTOR}, {DATE} and {TIME} placeholders for organizing repeated runs.
• dev_dataset_clear empties that dataset before the new results are inserted, useful when a named dataset is reused as a "latest results" table.

Send results into your apps (MCP connectors)

Optionally pipe the results into the apps you already use, via Model Context Protocol (MCP) connectors. This is an extra delivery step after the search: the Apify dataset is never changed and nothing extra is charged.

1. Authorize a connector once under Apify, Settings, API & Integrations (Notion, Linear, Airtable, or Apify).
2. Select it in the "Pipe results into your apps" input field (mcpConnectors).
3. For Notion, also set notionParentPageUrl to the page under which result pages are created. maxNotifyListings caps how many results are written per connector (default 50).

Each result becomes one entry with a title and its key fields as plain text; the complete record stays in the dataset. Leave mcpConnectors empty to skip.

Input Parameters

Parameter	Type	Default	Description
query	string	"Tesla"	The search query to run. Alias: search.
search	string	(none)	Alias of query, used when query is left at its default.
maxResults	integer	50	Maximum number of results to return (1 to 500). Alias: maxItems.
maxItems	integer	(none)	Alias of maxResults, used when maxResults is left at its default.
endPage	integer	(none)	Stop after this many result pages (about 10 results per page); caps the result count at endPage x 10.
region	string	"us-en"	Region and language code for results, free text, for example wt-wt (no region), us-en, uk-en, de-de, fr-fr, jp-jp. The 63 known codes are listed in the input form; an unknown code is still used but logs a warning.
safeSearch	string	"moderate"	Safe search filter level: off, moderate or strict (upper case such as OFF also works).
timeRange	string	""	Restrict results by recency: empty for any time, d (past day), w (past week), m (past month) or y (past year).
resultType	string	"web"	Type of results to return: web, images, videos, news or books. Alias: searchMode (default means web).
searchMode	string	(none)	Alias of resultType: default (same as web), web, images, news, videos or books. Used when resultType is left at web.
proxy	object	{"useApifyProxy": true}	Connection settings: Apify Proxy (automatic datacenter, escalating when needed) or your own proxy URLs. Left empty, the search runs on a direct connection, which is often blocked.
dev_transform_fields	array	[]	Advanced. Keep only the listed field paths in each output record, using dot notation for nested objects.
mcpConnectors	array	[]	Optional. Connectors (authorized under Apify, Settings, API & Integrations) that receive a summary of each result. Never changes the dataset.
notionParentPageUrl	string	(none)	Notion connector only: page URL or id under which result pages are created.
maxNotifyListings	integer	50	Cap on results written to each connector per run (1 to 1000).
dev_dataset_name	string	"default"	Advanced. Save results into a named dataset instead of the default one. Supports {ACTOR}, {DATE} and {TIME} placeholders.
dev_dataset_clear	boolean	false	Advanced. Clear the named dataset before inserting the new results.

Output Example

Sample shape: values are illustrative placeholders, not from a live record.

{
  "position": 1,
  "title": "Understanding Artificial Intelligence",
  "link": "https://example.com/articles/understanding-ai",
  "snippet": "Artificial intelligence refers to computer systems that perform tasks typically requiring human intelligence.",
  "domain": "example.com",
  "query": "artificial intelligence trends"
}

Output views

The dataset has one view per result type: Overview (web), Images, Videos, News and Books, each showing the columns that apply to that mode. The full record keeps the same fields in every mode (see Data You Get).

Plan Requirement

The search engines block plain connections from many networks (a direct request is often refused with an HTTP 403 and a short "Error getting results" page), so this actor does not rely on a bare connection. Turn on Apify Proxy under Proxy & connection (it is on by default in the form). The run then tries the connection types in order and moves on only when one is refused or fails:

1. Apify Proxy with the group you selected (automatic datacenter by default).
2. Apify residential proxy, for accounts whose plan includes it.
3. A backup connection when neither of the above can load results.

Datacenter-only access is not guaranteed to be enough. The block was measured from a plain local connection, not through Apify's datacenter or residential pools, so no tier is promised to work on every query. Apify free plans do not include residential proxies; on a free plan the run logs "Apify free plans do not include residential proxies" and continues on the backup connection, and some pages may still be unavailable. A paid Apify plan, or your own proxy URLs, gives the most reliable results. The proxy group and country are your choice and your cost.

A search whose page loads but has no matches is an empty search: the run ends SUCCEEDED with 0 items, logs "No results matched this query", and is not retried on other connections. A page that answered with a challenge, a rate limit or any non-success response is never treated as empty: the run moves on to the next connection and, if none loads, fails with the connection message below.

If no page can be loaded at all, the run fails (no empty dataset) with one of these messages:

• Plan refusal: "The proxy connection was refused for this account, so no page could be read. Your Apify plan does not include proxy access; a paid plan is required". If you entered your own proxy URLs, check their credentials.
• No residential access: "No results: residential proxies are unavailable on this run (Apify free plans do not include them), and the backup connection could not load the requested DuckDuckGo pages."
• Connection failure: "Could not load any results page (the requests failed), so nothing was collected. This is a connection problem, not an empty search. Please try again later."

Each search has a 25 second budget across all connection attempts.

Pricing

This actor is pay per event. Registered prices, exactly as live:

Event	Price
Result (apify-default-dataset-item), per result in the dataset	$0.0005
Actor start (apify-actor-start), once per run	FREE plan $0.1, BRONZE $0.06767, SILVER $0.03533, GOLD / PLATINUM / DIAMOND $0.003

Max results caps how many results a run can produce. Check the actor's Pricing tab for the current rates.

FAQ

How much does it cost?

Pay per event: $0.0005 per result plus a once-per-run start fee that depends on your Apify plan (see Pricing above). Max results caps how large a single run can get.

Is it legal to search DuckDuckGo, Bing, Google and the other engines this way?

This actor reads the same public result pages a normal search returns to any visitor, and it does not extract content from the sites those results link to. You are responsible for how you use the results: follow each search engine's own terms and the laws that apply to you, especially before republishing or redistributing what you find.

Can I schedule this to get fresh results automatically?

Yes, schedule it from the Schedules tab like any Apify actor. There is no incremental mode built in, so every scheduled run returns a fresh, complete set of results for the query rather than only what changed since the last run.

Why does the Search query field show "Tesla" by default?

That's a sample value so the form has something to run out of the box. query is required, so replace it with your own search before a real run; leaving the sample in place will search for "Tesla".

Why did my run fail instead of returning an empty dataset?

The run stops with an error when query is empty, or when no results page could be loaded at all (see the messages under Plan Requirement). A page that loads but has no matches is an empty search, not a failure: the run succeeds with 0 items. In the failure cases nothing is written to the dataset. Try again in a few minutes, switch the proxy group, or use a paid plan.

Can I use this with AI agents or MCP?

Yes. Call it from your agent framework's tool interface or through the Apify API or client shown above, then feed the returned JSON straight into your prompt or retrieval index. It does not currently include extra connectors that push results into other apps.

🔗 Want more AI data?

Pair this actor with these related scrapers from the same team:

🤖 AI Agent Web Fetcher An advanced web fetcher that can fetch almost all websites and convert them to...	🤖 Web Scraper For Llms Stealth web scraping engine built for LLMs. Converts any web page to clean markdown or...
🧩 Yandex Search Scraper Scrape Yandex web, image and video search results with URLs, snippets and organic...	📱 Truth Social Scraper Scrape public Truth Social profiles, posts, media, metrics, author data, and complete...
📱 Lemon8 Feed Scraper Scrape Lemon8 feeds across 22 categories and 10+ regions. Extract posts, images, videos...	⚽ Globo Esporte Scraper Scrape public sports content from ge.globo.com, including news, videos, matches and feed...

👉 Browse all abotapi scrapers

💬 Support

• 🐞 Found a bug or a missing field? Open a ticket on the Issues tab. We usually reply within hours.
• 🛠️ Need another site, extra fields or a private build? Email contact@abotapi.com.
• ⭐ Enjoying it? A quick review on the actor page helps other users find it.