AI Web Search & RAG Scraper

Run web searches from code and get structured results plus a ready-to-paste Markdown line per result, built for LLM prompts, LangChain/LlamaIndex pipelines and AI agents. Batch many queries in one run and export JSON or CSV.

Scope: this Actor returns search-result snippets (rank, title, URL, snippet) from Bing web search. It does not fetch full page content. To read the pages themselves, pass the URLs to a page scraper such as Firecrawl Alternative Web Scraper.

What you get

Field	Description
query	The search query that produced the row
rank	Position within the results for that query
title, url, snippet	Result title, link and snippet text
ragMarkdown	### [title](url) plus the snippet, ready to drop into a prompt (when enabled)
scrapedAt	ISO timestamp

{
  "query": "best MCP servers 2026",
  "rank": 1,
  "title": "Example result",
  "url": "https://example.com/post",
  "snippet": "Short description of the page...",
  "ragMarkdown": "### [Example result](https://example.com/post)\nShort description of the page...",
  "scrapedAt": "2026-10-14T12:00:00.000Z"
}

(Illustrative values.)

Input

Field	Default	Notes
queries	required	List of search queries
maxResultsPerQuery	10	Organic results per query
formatAsRAGMarkdown	true	Adds the ragMarkdown field

{ "queries": ["best MCP servers 2026", "RAG pipeline tutorial"], "maxResultsPerQuery": 5, "formatAsRAGMarkdown": true }

Pricing (pay per event)

Event	Price
Actor start	$0.03 per run
Query searched	$0.02 per query that returns results
Dataset item	$0.002 per result row

Example: 100 queries with 10 results each is about $2 + $2 + $0.03, roughly $4.03. Set a maximum cost per run in the Apify console to cap spend.

Use it from Python

from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("neon_innovation_lab/ai-search-rag-scraper").call(run_input={"queries": ["best MCP servers 2026"], "maxResultsPerQuery": 5})
context = "\n".join(item["ragMarkdown"] for item in client.dataset(run["defaultDatasetId"]).iterate_items())

Limits

• Results come from Bing's public HTML results; layout changes or rate limits can reduce results. Retry or lower concurrency if rows are missing.
• No ads, no knowledge panels or images: organic web results only.
• You are responsible for complying with the search engine's terms and applicable law.

FAQ

Can I get the full text of each page? Not with this Actor. Use its URLs with a page scraper.

Why Markdown? Fewer tokens than raw HTML or JSON when injecting results into a prompt.

Something broke? Open an issue from the Issues tab with the run URL and your input.

More from Neon Innovation Lab

• Google Maps Leads Scraper: businesses from Google Maps with phones, websites and emails found on their sites.
• Website Contact Finder: emails, phones and social links from any list of company websites.
• Company Tech Stack & Sales Dossier: detect a company's tech stack and get tailored pitch angles.