LinkedIn Job Scraper

A comprehensive LinkedIn job scraper that extracts job listings with full descriptions, company information, and detailed filtering options. Built with Playwright for reliable scraping and includes smart anti-detection measures.

Features

• Advanced Job Search: Search by keywords, location, and multiple filter criteria
• Full Job Descriptions: Optionally scrape complete job descriptions from individual job pages
• Smart Filtering: Filter by date posted, experience level, job type, remote work options, and industry
• Anti-Detection: Human-like scrolling, random delays, and modal handling to avoid blocking
• Comprehensive Data: Extract job titles, companies, locations, descriptions, salary info, and more
• Duplicate Prevention: Automatic deduplication of job listings
• Progress Tracking: Detailed logging and progress reporting throughout the scraping process

Input Parameters

Required

• Location (string): City, region, or country (e.g., "Norway", "Oslo", "Remote")

Optional

• Keywords (string): Job title or keywords to search for (leave empty for all recommendations)
• Date Posted (enum): Filter by recency
  • Past 24 hours
  • Past week (default)
  • Past 2 weeks
  • Past month
  • Past 6 weeks
• Experience Level (array): Select multiple levels
  • Internship
  • Entry level
  • Associate
  • Mid-Senior level
  • Director
  • Executive
• Job Type (array): Employment type
  • Full-time
  • Part-time
  • Contract
  • Temporary
• Remote Work (array): Workplace type
  • On-site
  • Hybrid
  • Remote
• Industry (array): Industry categories
  • Oil and Gas
  • Consulting
  • Construction
  • (More industries can be added)
• Maximum Jobs (integer): Max number of jobs to scrape (1-500, default: 50)
• Max Pages (integer): Number of result pages to process (1-20, default: 3)
• Include Job Description (boolean): Whether to fetch full job descriptions (default: true)

Output Data

Each job listing includes:

{
  "title": "Software Engineer",
  "company": "Tech Company",
  "location": "Oslo, Norway",
  "link": "https://linkedin.com/jobs/view/123456789",
  "description": "Full job description text...",
  "jobType": "Full-time",
  "salary": "500,000 - 700,000 NOK",
  "detailedLocation": "Oslo, Norway (Hybrid)",
  "companyInfo": "Tech Company",
  "descriptionFetched": true,
  "descriptionError": false,
  "scrapedAt": "2024-01-15T10:30:00.000Z",
  "position": 1,
  "searchKeywords": "software engineer",
  "searchLocation": "Norway",
  "searchFilters": {
    "datePosted": "pastWeek",
    "expLevel": "All",
    "jobType": "All",
    "remote": "All",
    "industry": "All"
  }
}

How It Works

Two-Phase Approach

1. Phase 1: Basic Job Collection

   • Scrolls through LinkedIn job search results
   • Extracts basic information (title, company, location, link)
   • Handles "See more jobs" buttons automatically
   • Implements smart scrolling with progress tracking

2. Phase 2: Detailed Job Descriptions (Optional)

   • Visits each job's individual page
   • Extracts full job descriptions and additional details
   • Includes rate limiting to avoid detection
   • Handles various LinkedIn page layouts

Anti-Detection Features

• Human-like Behavior: Random delays, mouse movements, and scrolling patterns
• Modal Handling: Automatically closes LinkedIn login popups and overlays
• Rate Limiting: Controlled request timing to avoid triggering LinkedIn's anti-bot measures
• User Agent Rotation: Uses realistic browser headers and settings
• Error Recovery: Retry logic for failed requests with exponential backoff

Usage Examples

Basic Job Search

{
  "location": "Norway",
  "keywords": "software engineer",
  "maxJobs": 25
}

Advanced Filtering

{
  "location": "Oslo, Norway",
  "keywords": "data scientist",
  "datePosted": "pastWeek",
  "expLevel": ["3", "4"],
  "jobType": ["F"],
  "remote": ["2", "3"],
  "maxJobs": 100,
  "includeJobDescription": true
}

Quick Overview (No Descriptions)

{
  "location": "Europe",
  "keywords": "",
  "maxJobs": 200,
  "includeJobDescription": false
}

Technical Details

• Browser Engine: Playwright with Chromium
• Language: JavaScript (Node.js)
• Platform: Apify Actor
• Concurrency: Sequential processing to avoid rate limiting
• Timeout Handling: Configurable timeouts with retry logic
• Memory Management: Efficient data structures and cleanup

Troubleshooting

Common Issues

1. No Jobs Found

   • Check if search criteria are too restrictive
   • Verify location spelling and format
   • Try broader keywords or remove filters

2. Incomplete Descriptions

   • LinkedIn may be rate-limiting requests
   • Try reducing maxJobs or increasing delays
   • Some job pages may have different layouts

3. Scraping Stopped Early

   • LinkedIn detected automated behavior
   • Reduce scraping speed by increasing maxJobs limit
   • Check debug screenshots in Key-Value Store

Debug Information

The actor saves debug screenshots to help troubleshoot issues:

• debug_screenshot.png: Page state during scraping
• error_screenshot.png: Page state when errors occur
• modal_still_present.png: If login modals couldn't be closed

Rate Limiting and Ethics

This scraper implements responsible scraping practices:

• Respects LinkedIn's server resources with appropriate delays
• Uses human-like interaction patterns
• Includes retry logic to handle temporary issues
• Follows robots.txt guidelines where applicable

Important: Use this tool responsibly and in compliance with LinkedIn's Terms of Service and applicable laws. Consider the impact on LinkedIn's servers and other users.

Support

For issues or questions:

1. Check the debug screenshots in the Actor's Key-Value Store
2. Review the detailed logs for error messages
3. Adjust input parameters based on the troubleshooting guide
4. Contact support if issues persist