🧠 Ask Website with AI

The Ask Website with AI Apify Actor is designed to analyze website content and answer specific questions using advanced AI models. It combines web scraping with AI-powered natural language understanding to generate accurate answers—even exploring internal links when needed.

---

🚀 Features

• Apify SDK for Python — A robust toolkit for building Apify Actors and scrapers in Python.
• Input Schema — Defines and validates the structure of the Actor's input.
• Request Queue — Manages the list of URLs to be scraped efficiently.
• Dataset — Stores structured data with uniform attributes.
• Playwright — A browser automation library used for dynamic web scraping.
• AI Integration — Supports Google Gemini and OpenAI GPT models for analysis.
• Intelligent Link Exploration — Optionally explores relevant internal links if no answer is found on the initial page, using semantic relevance prioritization.

---

🧭 How It Works

1. Initial Page Scraping
   Navigates to the given URL using Playwright and scrapes the HTML content, which is cleaned and structured into a JSON object.

2. Initial AI Analysis
   The AI model attempts to answer the provided questions using only the scraped content from the initial page.

3. Internal Link Exploration (Optional)
   If no clear answer is found and topNLinksToExplore > 0, the Actor identifies and ranks internal links. The AI chooses the most relevant one(s) to explore further.

4. Iterative Analysis
   The scraping and AI analysis process repeats across selected links, up to the number defined in topNLinksToExplore.

5. Combined Content Analysis
   If no definitive answer is found from individual pages, the Actor combines all scraped content and requests a final answer from the AI model.

---

⚙️ Input Configuration

The Actor expects input in JSON format, validated by input_schema.json:

Field	Type	Required	Description
start_urls	array	✅ Yes	A list of objects, each with a url string. Example: [{ "url": "https://apify.com" }]
questions	array	✅ Yes	List of questions for the AI model to answer. Example: ["What is the topic of the site?", "What is the email of the site?"]
instruction	string	❌ No	Extra instructions for the AI, e.g., "return only the specific answer without any explanation or extra text"
AIModel	string	✅ Yes	Choose from: "gemini-1.5-flash" (Recommended) "gpt-4o-mini" (Recommended) "gemini-1.5-pro" "gpt-4.1-mini" "gpt-4.1"
apiKey	string	✅ Yes	Your API key for the selected AI model
topNLinksToExplore	integer	❌ No	Max internal links to explore if no answer is found. Default: 3. Set 0 to disable

---

🛠 Getting Started

Build the Actor

Follow Apify’s documentation for instructions on building Actors.

Run the Actor

Provide the necessary start_urls, questions, AIModel, apiKey, and optional instruction and topNLinksToExplore.

---

🧪 Local Development

You can develop and test this Actor locally using the Apify CLI.

1. Install apify-cli

Using Homebrew:

$brew install apify-cli