AI Agent Web Fetcher

An advanced web fetcher that can fetch almost all websites and convert them to LLM-friendly Markdown format. Perfect for AI agents, RAG systems, and integration with search actors.

Features

• 🌍 Universal Fetching: Capable of fetching content from almost any website
• 🛡️ Advanced Anti-Blocking: Automatically handles complex challenges and protections
• 📝 Markdown Output: Clean, LLM-friendly format
• 🔗 Integration Ready: Seamlessly integrates with AI Web Search Tool and other search actors
• 🎯 Dual Mode: Single URL or batch processing from datasets

Usage

Single URL Mode

{
  "url": "https://example.com"
}

Integration Mode (with AI Web Search Tool)

When triggered by the AI Web Search Tool, the actor automatically:

1. Receives the search results dataset
2. Extracts URLs from the link field
3. Fetches and converts each page to Markdown

No configuration needed - the integration is automatic!

You can also manually specify a dataset:

{
  "datasetId": "abc123",
  "urlField": "link",
  "maxUrls": 10
}

Input Parameters

Parameter	Type	Description	Default
url	string	Single URL to fetch	-
datasetId	string	Dataset ID containing URLs (integration mode)	-
urlField	string	Field name containing URLs in dataset	link
maxUrls	integer	Max URLs to fetch from dataset (1-100)	10
proxy	object	Proxy configuration	-
waitUntil	string	Navigation wait condition	domcontentloaded
debug	boolean	Save HTML content in output	false

Output Format

Each fetched URL produces:

{
  "url": "https://example.com",
  "markdown": "# Page Title\n\nContent...",
  "metadata": {
    "title": "Page Title",
    "description": "Page description",
    "author": "Author name"
  }
}

Integration with AI Web Search Tool

1. Run the AI Web Search Tool
2. Add this actor as an integration
3. The actor automatically fetches and converts all search results to Markdown

Perfect for building AI agents that need to:

• Research topics with real-time web data
• Extract content from search results
• Build RAG systems with fresh web content

Standby Mode (HTTP API)

This Actor supports Standby mode for real-time HTTP API access with low latency. When enabled, the Actor runs as a persistent HTTP server.

Base URL

https://<your-username>--ai-fetch-python.apify.actor

Endpoints

Endpoint	Method	Description
/	GET	API information and available endpoints
/health	GET	Health check endpoint
/fetch	GET/POST	Fetch a URL and return Markdown

Fetch Parameters

Parameter	Type	Description
url	string	URL to fetch (required)
waitUntil	string	Wait condition: networkidle, load, domcontentloaded (default: networkidle)

Example Requests

Fetch a URL (GET):

$curl "https://<username>--ai-fetch-python.apify.actor/fetch?url=https://example.com&token=YOUR_TOKEN"

Fetch with POST:

curl -X POST "https://<username>--ai-fetch-python.apify.actor/fetch?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "waitUntil": "networkidle"}'

Response Format

{
  "url": "https://example.com",
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
  "metadata": {
    "title": "Example Domain",
    "description": "",
    "author": ""
  }
}