Smartcontext AI Web Crawler

Overview

SmartContext AI Web Crawler is a versatile web scraping tool designed to extract context-aware data from any website using AI. By providing one or more URLs and a custom natural language instruction, the crawler uses AI to analyze page content and return structured output tailored to your needs.

Whether you're researching biographies, gathering product information, or summarizing web content in a specific format, this Actor makes the process simple and intelligent.

IMPORTANT If you're having trouble scraping certain websites, please open an issue, and we'll work on resolving it as soon as possible.

Key Features

• 🌐 Scrape Any Website – Input arbitrary URLs from any domain.
• 🤖 AI-Powered Contextual Extraction – Get customized, instruction-based output using GenAI.
• 🧠 Flexible Use Cases – Extract anything from structured data to summaries, profiles, and more.
• ⚡ Fast and Scalable – Built on Apify SDK for performance and robustness.
• 📝 Custom Instruction Input – Control output structure with plain English prompts.

Use Cases

• Content Summarization – Extract bullet points, summaries, or structured overviews of web pages.
• Profile or Character Generation – Turn biography pages into RPG-style character sheets or professional profiles.
• Product/Service Research – Format product pages into comparable specs tables.
• Custom Data Pipelines – Feed specific context and extract exactly what your automation needs.

How It Works

1. Input URLs and Instructions – Provide one or more URLs and a natural language instruction for extraction.
2. Run the Actor – The Actor navigates each page and extracts content.
3. AI-Powered Structuring – GenAI interprets the content according to your prompt.
4. Get the Output – Receive structured JSON results, one per URL.

Input Configuration

SmartContext AI Web Crawler Configuration

The Actor expects the following input fields:

• start_urls (array of objects, required): List of pages to scrape. Each object must include a url key.

  Example:

  "start_urls": [
    { "url": "https://pt.wikipedia.org/wiki/Michael_Jordan" },
    { "url": "https://pt.wikipedia.org/wiki/Scottie_Pippen" }
  ]
  

• ai_input (string, required): A clear instruction for how AI should process and format the extracted content.

  Example:

  "ai_input": "Create a character sheet similar to that of an RPG character with the information on this site."
  

Output Format

Output Fields

The output is an array of structured objects, each corresponding to one input URL. The structure of each object depends on your ai_input instruction. A typical output might look like this:

{"character": {
    "name": "Michael Jordan",
    "occupation": "Entrepreneur, Former Basketball Player",
    "nickname": "Air Jordan, MJ, Black Jesus",
    "age": 62,
    "birthdate": "February 17, 1963",
    "birthplace": "Brooklyn, New York, USA",
    "height": "6 ft 6 in (1.98 m)",
    "weight": "216 lb (98 kg)",
    "attributes": {
      "strength": "Exceptional leaping ability and scoring prowess",
      "agility": "Remarkable agility and defensive skills",
      "intelligence": "Strategic player, successful businessman",
      "charisma": "Global icon, influential spokesperson"
    },
    "skills": {
      "basketball": "Elite scoring, defense, leadership, clutch performance",
      "baseball": "Minor league baseball experience",
      "business": "Successful entrepreneur and team owner",
      "racing": "NASCAR team owner"
    },
    "equipment": {
      "uniform_numbers": [
        "23",
        "45",
        "12"
      ],
      "signature_sneakers": "Air Jordan (Nike)",
      "other": "Baseball bat, racing team equipment"
    }
}}

The shape of the result field will vary based on your natural language instruction.

Get Started

1. Open SmartContext AI Web Crawler in the Apify Store.
2. Enter the URLs and define your extraction instruction.
3. Run the Actor and download structured output data.

Support & Feedback

Need help or want to suggest a feature? Reach out via Apify’s support or send feedback directly through the Actor page.