RAG Web Browser

Search the web and extract clean Markdown content for AI applications and RAG pipelines.

What it does

Accepts a search query or direct URL, fetches web pages, cleans HTML with Mozilla Readability, and returns LLM-ready Markdown.

Features

• Search mode (DuckDuckGo) or direct URL mode
• Mozilla Readability for content extraction
• Clean Markdown output for LLMs
• Configurable content length limits
• Works with Claude, GPT, LangChain, CrewAI

Input

Field	Type	Description
query	string	Search query or direct URL
maxResults	integer	Max search results to process
includeMetadata	boolean	Include title/description
maxContentLength	integer	Max chars per page

Output fields

url, title, description, markdown_content, content_length, scraped_at

Use cases

RAG pipelines, AI agents, research automation, content monitoring, knowledge bases.

Pricing

• $0.005 per actor start
• $0.005 per web page extracted

---

💻 Code Example — Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("nexgendata/rag-web-browser").call(run_input={
    # Fill in the input shape from the actor's input_schema
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

🌐 Code Example — cURL

curl -X POST "https://api.apify.com/v2/acts/nexgendata~rag-web-browser/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ /* input schema */ }'

❓ FAQ

Q: How do I get started? Sign up at apify.com, grab your API token from Settings → Integrations, and run the actor via the Apify console, API, Python SDK, or any integration (Zapier, Make.com, n8n).

Q: What's the typical cost per run? See the pricing section below. Most runs finish under $0.10 for typical batches.

Q: Is this actor maintained? Yes. NexGenData maintains 140+ Apify actors and ships updates regularly. Bug reports via the Apify console issues tab get responses within 24 hours.

Q: Can I use the output commercially? Yes — you own the output data. Check the target site's Terms of Service for any usage restrictions on the scraped content itself.

Q: How do I handle rate limits? Apify manages concurrency and retries automatically. For very large batches (10K+ items), run multiple smaller jobs in parallel instead of one mega-job for better reliability.

💰 Pricing

Pay-per-event pricing — you only pay for what you actually extract.

• Actor Start: $0.0050
• Web page: $0.0050

🔗 Related NexGenData Actors

• AI Web Scraper
• Hacker News Scraper

🚀 Apify Affiliate Program

New to Apify? Sign up with our referral link — you get free platform credits on signup, and you help fund the maintenance of this actor fleet.

---

Built and maintained by NexGenData — 140+ actors covering scraping, enrichment, MCP servers, and automation.