AI Web Search & Read: Google results as clean Markdown

What is AI Web Search & Read?

AI Web Search & Read gives your agent the web in one call: send it a question, and it runs a Google search, fetches the top organic result pages, strips navigation/ads/boilerplate and returns clean Markdown per result — ready to drop straight into an LLM prompt, a RAG pipeline or a tool-calling agent. It also accepts a list of urls to skip the search step entirely and just convert specific pages to Markdown, and it can run in Standby mode as a live HTTP API instead of a batch run. Try it from the Input tab, or call it from code, MCP or the Apify API.

Why use AI Web Search & Read?

Most "give my agent web access" setups need two or three separate tools: a SERP API, a scraper, and a Readability/boilerplate remover. This Actor collapses all three into one call and one price. Typical users:

• AI agent builders wiring a web_search tool into a LangChain, LlamaIndex, OpenAI-function or MCP agent, without standing up their own SERP + scraping infrastructure.
• RAG pipelines that need fresh, non-indexed web content as retrieval context rather than a stale vector store.
• Researchers and analysts who want 3-5 clean summaries of "what does the web say about X" without opening ten tabs.
• Anyone already using urls-only mode as a lightweight, dependency-free "webpage to Markdown" reader.

How to use AI Web Search & Read

1. Enter a Search query (or a list of Search queries, or a list of URLs to read directly, skipping search).
2. Set Max results per query (default 3) and Output format (Markdown, plain text, or both).
3. Click Start. Each organic result becomes one dataset row with the page's title, URL, snippet and clean Markdown.
4. Export as JSON, CSV, Excel or HTML, or read the results back over the API/MCP.

Example input

{
    "query": "what is web scraping",
    "maxResults": 3,
    "outputFormat": "markdown"
}

Or read specific pages directly, no search:

{
    "urls": ["https://docs.apify.com/platform"]
}

Example output

{
    "query": "what is web scraping",
    "position": 1,
    "title": "What Is Web Scraping? - Apify",
    "url": "https://apify.com/web-scraping",
    "displayedUrl": "apify.com › web-scraping",
    "snippet": "Web scraping is the automated process of extracting data from websites...",
    "markdown": "# What Is Web Scraping?\n\nWeb scraping is the automated process of...",
    "text": null,
    "wordCount": 842,
    "charCount": 6213,
    "truncated": false,
    "lang": "en",
    "publishedAt": "2025-03-11T00:00:00Z",
    "author": null,
    "siteName": "apify.com",
    "statusCode": 200,
    "fetchedAt": "2026-09-13T10:00:04.000Z",
    "error": null,
    "scrapedAt": "2026-09-13T10:00:03.500Z"
}

You can download the dataset in JSON, CSV, Excel, HTML, XML or RSS.

Data table

Field	Type	Description
query, position	string, number	The query this row came from (null for direct urls) and its SERP rank
title, url, displayedUrl, snippet	string	Google's result title/URL/breadcrumb/snippet
markdown, text	string	Clean Markdown / plain text of the page's main content
wordCount, charCount, truncated	number, boolean	Size of the extracted content and whether it was cut to fit
lang, publishedAt, author, siteName	string	Metadata read from JSON-LD or meta tags, when present
statusCode, fetchedAt	number, string	HTTP status of the page fetch and when it happened
error, scrapedAt	string	Set when the search or page fetch failed; always-present timestamp

Input parameters

Parameter	Type	Default	Description
query	string	what is web scraping	A single search query
queries	array	–	Multiple queries in one run; overrides query
urls	array	–	Specific pages to read, skipping search entirely
maxResults	integer	3	Organic results read per query (max 20)
outputFormat	enum	markdown	markdown, text, or both
maxCharsPerResult	integer	8000	Truncate long pages for LLM context control (max 100000)
countryCode, languageCode	string	us, en	Google localisation (gl/hl)
includeSnippetOnly	boolean	false	Cheap mode: return SERP rows without fetching pages
maxConcurrency	integer	5	Parallel page fetches

Pricing

AI Web Search & Read is priced at $0.005 per result (pay-per-event), with volume discounts at higher usage tiers. A default 3-result run costs about $0.015. One SERP fetch (shared across up to 20 results for that query) uses roughly $0.0025 of Apify Proxy traffic internally — page fetches themselves are free, direct HTTP with no proxy. That works out to ~$5 per 1,000 results, in line with the median price of the top actors on Apify Store, and typically cheaper than stacking a separate SERP API + scraper + Readability service.

AI Web Search & Read vs. apify/rag-web-browser

Both turn a query into LLM-ready Markdown. This Actor additionally offers: a pure urls-only reader mode (no search needed), a dedicated includeSnippetOnly cheap mode, both markdown+text output, and JSON-LD-aware metadata (author, publishedAt, siteName) on every row — useful for citing sources back to the LLM.

Using AI Web Search & Read with AI agents

Batch (Apify API / MCP tool call):

curl -s -X POST "https://api.apify.com/v2/acts/webdatatools~ai-web-search/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "best crm for startups", "maxResults": 5}'

This Actor is also discoverable through the Apify MCP Server — an MCP-connected agent (Claude, Cursor, etc.) can call it directly as a tool once added.

LangChain (Python):

from langchain_apify import ApifyActorsTool

search_tool = ApifyActorsTool("webdatatools/ai-web-search")
result = search_tool.invoke({"query": "what is retrieval augmented generation", "maxResults": 3})

LlamaIndex:

from llama_index.tools.apify.tool import ApifyWrapper

apify = ApifyWrapper()
dataset = apify.call_actor(
    actor_id="webdatatools/ai-web-search",
    run_input={"query": "what is retrieval augmented generation", "maxResults": 3},
    dataset_mapping_function=lambda item: {"content": item["markdown"], "url": item["url"]},
)

Standby mode: call it like a live API

This Actor supports Actor Standby — start it once and call GET /search as a normal HTTP endpoint instead of starting a new batch run every time:

GET https://ai-web-search.webdatatools.apify.actor/search?query=apify+web+scraping&maxResults=3

Response is a plain JSON array of the same rows described above. GET / returns a readiness payload for health checks.

FAQ

Is this legal? Scraping public Google search results and public web pages for personal or internal use is generally accepted practice, but you are responsible for complying with the target sites' terms of service and applicable law in your jurisdiction.

Why is markdown sometimes short or empty? JavaScript-rendered pages (heavy React/Vue SPAs that inject content client-side) return little or no text to a plain HTTP fetch — this Actor does not run a browser. For those sites, wordCount will be low and text/markdown sparse; this is an honest limitation, not a bug.

Why did I get an error row instead of content? A page fetch can fail (404, timeout, blocked) or the SERP fetch itself can be rate-limited — the row still comes back with error set and every other field intact where possible, so your agent always gets a response instead of a broken run.

Does this work with n8n / Make / Zapier? Yes — use the official Apify integration in any of those tools, or call the REST API directly.

Related Actors: Google Search Results Scraper (raw SERP data), Website to Markdown Crawler (full-site crawling), Article Extractor (single-article metadata).

Found a bug or have a feature request? Use the Issues tab on this Actor's page — we read every report.

Related Actors

Part of the webdatatools web-intelligence suite — every Actor is pay-per-event, runs without proxies or a headless browser, and returns one clean row per entity:

Browse the whole suite at webdatatools, or call ten of these Actors straight from Claude, Cursor or Cline with the webdatatools MCP server.

Website & domain intelligence

• Website Contact & Social Extractor — e-mails, phones and social profiles per domain
• Website Tech Stack Detector — CMS, e-commerce, analytics, pixels and payments per domain
• Domain DNS & Email Security Checker — SPF, DKIM, DMARC, MX provider, registrar and domain age
• Domain Security Audit — TLS expiry, security headers, redirect chain, robots and llms.txt
• Subdomain Finder (Certificate Transparency) — every subdomain seen in CT logs, with a live DNS check
• Bulk Core Web Vitals & PageSpeed Audit — Lighthouse scores, LCP, CLS, INP and top fixes per URL
• On-Page SEO Audit — title, meta, headings, links, images and schema issues per page
• Sitemap URL Extractor & Change Monitor — every sitemap URL, or new and removed pages between runs
• Wayback Machine Snapshot & Page Change Tracker — how a page changed over time, or every archived snapshot

Content for AI, LLMs and RAG

• Website to Markdown Crawler for LLM & RAG — any site as clean Markdown per page, no browser
• Article & News Extractor — clean article text, author, date and Markdown per URL
• Structured Data & JSON-LD Extractor — Schema.org and Open Graph data from any page
• Google News Scraper — news results by keyword, topic or site
• Press Release Monitor — PR Newswire, Business Wire and GlobeNewswire releases

Search, video and social

• Google Search Results Scraper — organic SERP results per keyword and country
• YouTube Comments Scraper — comments and replies with likes, no API key
• YouTube Channel Latest Videos — the latest 15 videos of any channel from RSS
• YouTube Channel Videos Scraper — a channel's full video, shorts and stream list
• YouTube Search Results Scraper — videos, channels and playlists per query
• YouTube Video Details Scraper — views, likes, description, tags and chapters per video
• Apple Podcasts Lookup & Episodes Scraper — podcast metadata and episodes from iTunes and RSS
• Bluesky Scraper — posts, profiles, followers and threads from the AT Protocol API

Leads, jobs and company data

• Company 360 — one row per domain: contacts, tech, security, hiring and company facts
• Hiring Signals Scraper — open jobs and hiring velocity from 10 public ATS boards
• Y Combinator Companies & Founders Scraper — YC startups by batch, industry and hiring status
• Wikidata Entity & Company Enrichment — HQ, founders, employees, revenue and social IDs per company
• Bulk Email Validator — syntax, MX, disposable, role and free-provider checks
• OpenStreetMap POI Extractor — shops and amenities by radius, bbox or area

Developer, app and research data

• npm, PyPI & Crates.io Package Health Checker — releases, downloads, deprecation and a health score
• GitHub Repository Health & Activity Report — stars, commits, contributors and risk flags per repo
• VS Code Marketplace Extension Scraper — installs, ratings and versions per extension
• Chrome Web Store Extension Scraper — users, rating, version and developer per extension
• Google Play Store Scraper — apps, ratings, installs, developer contact and reviews
• App Store (iOS) App Metadata & Top Charts — ratings, price, version and charts per app
• CrossRef DOI & Citation Metadata Lookup — papers, authors, journals and citation counts
• FDA Recalls & Adverse Events Monitor — food, drug and device recalls from openFDA
• iCal / ICS Calendar Feed to Events Extractor — any public calendar feed as event rows
• Shopify Store Products Scraper — catalog, prices, variants and stock per store