Under maintenance

Pricing

Pay per usage

Try for free

Go to Apify Store

Intelligent Website Crawler

Under maintenance

Try for free

Pricing

Pay per usage

Rating

0.0

(0)

Developer

HappiTap

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

AI-Enhanced Website Crawler

An intelligent website crawler that combines advanced crawling capabilities with AI-powered page classification, pattern detection, and content extraction. This enhanced version supports all the features of Apify's Website Crawler plus intelligent AI processing to minimize API usage and maximize data extraction efficiency.

🚀 Key Features

Advanced Crawling Capabilities

Multiple Crawler Types: Supports Playwright (adaptive, Chrome, Firefox) and Puppeteer
Smart Content Extraction: Comprehensive data extraction including metadata, markdown, text, screenshots
Dynamic Content Handling: Handles JavaScript-rendered content, infinite scroll, and expandable elements
Sitemap Support: Automatically discovers and processes XML sitemaps
Proxy Support: Built-in Apify Proxy integration with rotation
Media Blocking: Optional blocking of images/videos for faster crawling

AI-Powered Intelligence

Automatic Page Classification: AI identifies page types (product, category, service, contact, blog, FAQ, etc.)
Pattern Learning: Learns page patterns to reduce AI API calls for similar pages
Smart Content Processing: Context-aware content extraction based on page type
Token Optimization: Aggressive content pruning to minimize AI API costs
Caching System: Reuses learned patterns for similar page structures

Comprehensive Output

Multiple Formats: HTML, Markdown, clean text, structured metadata
Screenshots: Optional full-page screenshots with cloud storage URLs
Performance Metrics: Load times, content metrics, accessibility info
AI Insights: Page classification, extracted patterns, processed content
Link Analysis: Internal/external link discovery with metadata

📋 Input Configuration

Basic Crawler Settings

{
  "startUrls": [
    {
      "url": "https://example.com",
      "method": "GET"
    }
  ],
  "crawlerType": "playwright:adaptive",
  "maxCrawlPages": 100,
  "maxCrawlDepth": 3,
  "maxConcurrency": 10
}

AI Configuration

{
  "enableAI": true,
  "aiModel": "gpt-3.5-turbo",
  "enablePatternLearning": true,
  "patternSimilarityThreshold": 0.8,
  "maxAICallsPerRun": 50,
  "taskType": "auto"
}

Content Processing

{
  "saveMarkdown": true,
  "saveHtml": false,
  "saveScreenshots": false,
  "blockMedia": true,
  "removeCookieWarnings": true,
  "dynamicContentWaitSecs": 10
}

🎯 Task Types

auto: AI automatically detects the best processing approach
summarize: Creates comprehensive content summaries
extractProducts: Extracts product information (name, price, features, specs)
extractServices: Extracts service offerings and descriptions
extractFAQs: Extracts question-answer pairs and help content
extractContacts: Extracts contact information and business details

📊 Output Format

{
  "url": "https://example.com/product/123",
  "timestamp": "2024-01-15T10:30:00Z",
  "crawlTime": 2500,
  "metadata": {
    "title": "Product Name - Company",
    "description": "Product description from meta tags",
    "ogTitle": "Open Graph title",
    "schema": ["JSON-LD structured data"]
  },
  "content": {
    "html": "<html>...</html>",
    "markdown": "# Product Name\n\nProduct description...",
    "text": "Clean text content",
    "title": "Product Name",
    "headings": [
      {"level": 1, "text": "Product Name", "id": "heading-0"}
    ],
    "links": [
      {"url": "https://example.com/related", "text": "Related Product"}
    ],
    "images": [
      {"url": "https://example.com/image.jpg", "alt": "Product image"}
    ]
  },
  "screenshots": {
    "key": "screenshot-1234567890.png",
    "url": "https://api.apify.com/v2/key-value-stores/.../screenshot-1234567890.png"
  },
  "ai": {
    "classification": {
      "type": "product",
      "confidence": 0.95,
      "reasoning": "Page contains product information with pricing and specifications"
    },
    "patterns": {
      "selectors": {
        "title": ".product-title",
        "price": ".price",
        "description": ".product-description"
      },
      "confidence": 0.85
    },
    "processedContent": {
      "processed": true,
      "content": "Structured extracted data based on page type"
    }
  },
  "performance": {
    "loadTime": 1200,
    "domContentLoaded": 800,
    "firstPaint": 600
  },
  "accessibility": {
    "hasAltTexts": 5,
    "missingAltTexts": 1,
    "headingStructure": ["h1", "h2", "h2", "h3"]
  }
}

🧠 AI Features in Detail

Page Classification

The AI automatically identifies page types:

Product pages: Individual product listings with specs and pricing
Category pages: Product category or listing pages
Service pages: Service descriptions and offerings
Contact pages: Contact info, about us, company details
Blog pages: Articles, news, blog posts
FAQ pages: Help, support, and FAQ content
Home pages: Landing pages and homepages

Pattern Learning

Analyzes page structure and identifies data extraction patterns
Caches successful patterns for reuse on similar pages
Reduces AI API calls by up to 80% on sites with consistent structure
Learns CSS selectors for key data elements

Smart Content Processing

Adapts processing strategy based on detected page type
Extracts relevant structured data automatically
Minimizes token usage through intelligent content pruning
Provides confidence scores for all AI decisions

🔧 Advanced Configuration

Extract Services with Internal Link Following

{
  "startUrls": [
    { "url": "https://example.com" }
  ],
  "taskType": "extractServices",
  "followInternalLinks": true,
  "maxDepth": 2
}

Extract Products from Multiple URLs

{
  "startUrls": [
    { "url": "https://shop1.com" },
    { "url": "https://shop2.com" }
  ],
  "taskType": "extractProducts"
}

How It Works

Content Extraction: Uses Puppeteer to load pages and Cheerio to extract clean content
Intelligent Processing: LangChain processes content based on the specified task type
Structured Output: Returns processed content with metadata and original URL
Optional Crawling: Can follow internal links to gather more comprehensive data

Installation

Clone this repository
Install dependencies: npm install
Set your OPENAI_API_KEY environment variable
Run the actor: npm start

Development

npm start - Run the actor
npm run format - Format code with Prettier
npm run lint - Run ESLint
npm run lint:fix - Fix ESLint issues

Architecture

src/main.js - Main entry point and input validation
src/routes.js - Request routing
src/handlers/websiteScraper.js - Main scraping logic
src/services/langchainService.js - LangChain integration and task processing
src/puppeteerLauncher.js - Puppeteer browser configuration

Website Content Crawler

rupom888/website-content-crawler

Syed Rupom

AI Website Content Crawler

ilborso/ai-website-content-crawler

A super fast website crawler for Agentic AI integration

Fabio Borsotti

5.0

Website Crawler API — Markdown for RAG

tugelbay/website-content-crawler

Website crawler API for public pages and clean Markdown, text, or HTML output for RAG pipelines, AI agents, documentation indexing, and monitoring. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler

Tugelbay Konabayev

Website Content Crawler

bhansalisoft/website-content-crawler

Website Content Crawler : scrap any website content with meta title and meta description and site logo

bhansalisoft

Intelligent Website Scrapper

happitap/intelligent-website-scrapper

An intelligent website scraper that uses LangChain and LLM to extract and process content based on high-level goals like summarization, product extraction, service extraction, and FAQ extraction.

HappiTap

287

5.0

Website Content Crawler — Extract Full Site Content

oneary/website-content-crawler

🌐 Full website crawler that extracts structured content (text, headings, metadata, links, images) from any domain. Free platform compute pricing.

Luan M.

Website Contacts Crawler

quaking_pail/contact-crawler

Scrap website searching for contact details, emails and phone numbers

AI_Builder

Domain Contact Scraper – Intelligent Email & Phone Extractor

emastra/website-contact-scraper

Intelligent website contact scraper that extracts emails, phone numbers, WhatsApp links, and social profiles from any domain. Uses priority-first crawling and returns one structured, consolidated result per start URL with transparent crawl metadata.