🧠 AI Website Scraper & Smart Extractor

Turn any website into a structured API. Extract clean data, generate leads, and monitor competitors—automatically.

---

📍 What is AI Website Scraper?

AI Website Scraper is a next-generation data extraction engine designed to turn the chaos of the web into organized, usable data. It adapts to any website structure, allowing you to extract exactly what you need without writing complex code or maintaining fragile selectors.

Whether you're building a dataset for market research, feeding an LLM, or monitoring e-commerce prices, this Actor handles the heavy lifting of understanding and structuring web content.

🚀 Why use this Actor?

• Universal Extraction: Works on ANY website—from news portals and blogs to complex e-commerce stores and documentation.
• Zero Maintenance: No need to update scripts when a website changes its layout. The AI adapts automatically.
• Global Reach: Instantly translate extracted data into 50+ languages using Lingo.dev, perfect for international market analysis.
• Structured Output: Get clean, consistent JSON data ready for your API, database, or spreadsheet.

---

⬇️ Input Parameters

Configure your scrape in seconds. Below is a detailed breakdown of all available options, followed by a complete example.

⚙️ Configuration Format

Parameter	Type	Required	Description
urls	array	✅ Yes	The list of webpages (URLs) you want to extract data from.
prompt	string	✅ Yes	A plain English description of what you want to extract (e.g., "Extract product price, rating, and all reviews").
run_name	string	✅ Yes	A unique label to help you identify this batch of data later (e.g., competitor-scrape-q4).
enhance_prompt	boolean	❌ No	(Optional) Uses AI to optimize your prompt for better accuracy on complex pages. Defaults to false.
user_schema	object	❌ No	(Optional) A strict JSON schema object. If provided, the output will faithfully adhere to this structure.
translate_to	array	❌ No	(Optional) A list of ISO language codes (e.g., ['es', 'de']) to translate the results into.
proxyConfiguration	object	❌ No	(Optional) Proxy settings. Defaults to Apify Proxy (recommended) to avoid blocks.
gemini_api_key	string	❌ No	(Optional) Provide your own Gemini API Key to waive the AI Usage fee.
lingo_api_key	string	❌ No	(Optional) Your Lingo.dev API Key. Required only if you use the translate_to feature.

💡 Complete Example

This example shows a fully configured run with all options enabled.

{
  "run_name": "monitor-competitor-pricing",
  "urls": [
    "https://competitor.com/pricing",
    "https://competitor.com/products/enterprise"
  ],
  "prompt": "Extract all pricing tiers, feature lists, and hidden fees. Ensure currency is standardized.",
  "enhance_prompt": true,
  "user_schema": {
    "type": "object",
    "properties": {
      "tiers": { "type": "array" },
      "last_updated": { "type": "string" }
    }
  },
  "translate_to": ["fr", "de", "es"],
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "gemini_api_key": "AIzaSy...",
  "lingo_api_key": "lng_..."
}

---

⬆️ Output Data

You get clean, structured data delivered to your Apify Dataset. Each item in the dataset contains not just the extracted data, but also a summary of the execution phases for that specific URL.

📦 Data Structure

Field	Type	Description
url	string	The source page address that was processed.
status	string	The final outcome of the operation (e.g., success, error).
data	object	The actual structured content extracted from the page.
extracted_at	string	ISO 8601 timestamp indicating when the data was captured.
phase_1	string	Summary: Details about the page analysis and schema generation performance.
phase_2	string	Summary: Details about the extraction performance (time taken, strategy used).
translation_status	string	Summary: Status of the translation process and target languages (if active).

📄 Sample Result

{
  "url": "https://competitor.com/pricing",
  "status": "success",
  "extracted_at": "2025-12-31T15:30:00+00:00",
  "data": {
    "tiers": [
      {
        "name": "Starter",
        "price": "$29/mo",
        "features": ["Basic Support", "5 Users"]
      },
      {
        "name": "Enterprise",
        "price": "Contact Sales",
        "features": ["24/7 Support", "SSO"]
      }
    ],
    "metadata": {
      "translated_content": true,
      "translated_to_locales": ["fr", "de", "es"]
    }
  },
  "phase_1": "Schema: completed (2.1s)",
  "phase_2": "Extracted: completed (5.4s)",
  "translation_status": "completed - to ['fr', 'de', 'es']"
}

---

🌍 Multilingual Power with Lingo.dev

Unlock global insights by automatically translating your extracted data.

This Actor integrates seamlessly with Lingo.dev, a specialized localization engine. By simply providing your Lingo API Key, you can turn a single-language scrape into a multi-lingual dataset instantly.

---

💰 Pricing

Pay only for what you use.

Item	Cost	Notes
Actor Start	$0.005	Per run.
URL Processed	$0.003 FREE	Per URL successfully processed.
AI Usage	$0.030	Waived if you provide your own API Key.

---

❓ FAQ

1. Does it work on modern React/Next.js/SPA websites?

Yes. The actor uses a real browser (Headless Chrome) to render the page, scroll, and wait for dynamic content to load. It sees exactly what a user sees, making it perfect for complex Single Page Applications (SPAs).

2. Can I scrape pages behind a login?

Currently, this actor is optimized for publicly available data. It does not support handling user authentication or login flows out of the box.

3. What happens if the website layout changes?

Nothing! That's the beauty of AI extraction. Unlike traditional scrapers that rely on fragile CSS selectors (which break when a site updates), this actor "reads" the page content like a human. It will continue to find and extract your data even if the underlying HTML structure changes completely.

4. How much does it cost to scrape 1,000 pages?

It depends on your configuration. If you provide your own gemini_api_key, you only pay for the platform usage (~$3.00 + $0.005 base). If you use our AI key, it would be ~$33.00. We recommend bringing your own keys for high-volume jobs!

5. Can I integrate this with my database?

Absolutely. Apify provides integrations for Google Sheets, Airtable, Zapier, Make, and simple Webhooks. You can pipe the JSON output directly into your database or workflow the moment a job finishes.

6. Do I need to be a developer to use this?

No. You don't need to write a single line of code. Just describe what you want in plain English (e.g., "Extract all restaurant names and reviews"), and the expert AI handles the rest.

7. How fast is it?

It processes most pages in seconds. The exact speed depends on the complexity of the website and the amount of data you're extracting, but it's built for speed and scale.

8. Can I use this for lead generation?

Yes! It's perfect for building lists of leads from business directories, contact pages, and event listings. It can even format phone numbers and emails consistently for your CRM.

---

🔌 Integrations

Fits right into your workflow.

• Make / Zapier: Automate post-processing or send alerts.
• Google Sheets: Pipe data directly into your reports.
• API: Trigger and control via REST API from any application.

Get Started for Free