LinkedIn Company Scraper

A robust LinkedIn company profile scraper built as an Apify Actor that extracts comprehensive company information from LinkedIn company pages.

Features

• Company Directory Scraping: Search and discover companies by keywords

• Profile Data Extraction: Extract detailed company information including:

  • Company name and logo
  • About us description
  • Number of employees and company size
  • Website URL
  • Industry and headquarters location
  • Company type and founding year
  • LinkedIn follower count
  • Specialties and services

• Robust Error Handling: Built-in retry mechanisms and backoff strategies

• Rate Limiting: Configurable delays and concurrent request limits

• Anti-Detection: User agent rotation and CAPTCHA detection

• Concurrent Processing: Multi-threaded scraping with configurable concurrency

Configuration

The scraper accepts configuration through Apify input. Here's the expected configuration structure:

{
  "mode": "profiles",
  "companies": [
    "Microsoft",
    "Google",
    "Apple",
    "Amazon"
  ],
  "settings": {
    "max_concurrent_requests": 2,
    "download_timeout": 30,
    "delay": {
      "min": 3,
      "max": 7
    }
  },
  "advanced": {
    "rotate_user_agents": true
  },
  "user_agents": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
  ]
}

Output Data Structure

The scraper outputs company data in the following format:

{
  "company_name": "Microsoft",
  "linkedin_followers_count": 15000000,
  "company_logo_url": "https://media.licdn.com/dms/image/company-logo_200_200/...",
  "about_us": "At Microsoft, our mission is to empower every person...",
  "num_of_employees": "100,001+ employees",
  "website": "https://www.microsoft.com",
  "industry": "Computer Software",
  "headquarters": "Redmond, Washington",
  "type": "Public Company",
  "founded": "1975",
  "specialties": "Cloud Computing, Productivity Software, Developer Tools..."
}

Rate Limiting and Best Practices

• Respect Rate Limits: The scraper includes built-in delays and retry mechanisms
• Concurrent Requests: Default is 2 concurrent requests - adjust based on your needs
• User Agent Rotation: Automatically rotates user agents to avoid detection
• Error Handling: Implements exponential backoff for failed requests

Error Handling

The scraper includes comprehensive error handling:

• Request Failures: Automatic retries with exponential backoff
• Rate Limiting: Handles 429 status codes with extended delays
• Data Parsing Errors: Continues execution and logs parsing failures
• Network Issues: Robust handling of connection timeouts and network errors

Logging

The scraper provides detailed logging:

• INFO: General progress and successful operations
• WARNING: Non-critical issues and retries
• ERROR: Failed operations and exceptions
• DEBUG: Detailed extraction information (when verbose mode enabled)

Limitations

• LinkedIn Terms of Service: Ensure compliance with LinkedIn's ToS
• Rate Limiting: LinkedIn may impose rate limits on automated requests
• Data Accuracy: Extracted data depends on LinkedIn page structure
• CAPTCHA Challenges: May encounter CAPTCHA verification