Linkedin Profile Scraper

A robust LinkedIn profile scraper built as an Apify Actor that extracts comprehensive profile information including personal details, work experience, education, skills, and contact information.

🚀 Features

• Complete Profile Data: Extracts name, headline, summary, location, and profile picture
• Work Experience: Detailed job history with titles, companies, descriptions, and dates
• Education Background: Academic history including schools, degrees, and fields of study
• Skills & Endorsements: Professional skills with endorsement counts
• Contact Information: Email, phone, and other contact details (when available)
• Robust Error Handling: Graceful handling of missing data and API limitations
• Clean Data Structure: Well-formatted JSON output with consistent field names

📋 Requirements

• Valid LinkedIn account cookies (specifically li_at cookie)
• LinkedIn profile URLs to scrape
• Apify platform account (for deployment)

Apify Deployment

1. Upload the code to your Apify Actor
2. Configure the Actor with appropriate memory and timeout settings
3. Test with sample input data

📝 Input Configuration

The Actor expects the following input parameters:

{
  "url": "https://www.linkedin.com/in/profile-username",
  "cookies": [
    {
      "name": "li_at",
      "value": "your-li-at-cookie-value",
      "domain": ".linkedin.com",
      "path": "/"
    },
    {
      "name": "JSESSIONID", 
      "value": "your-jsessionid-value",
      "domain": ".linkedin.com",
      "path": "/"
    }
  ],
  "timestamp": "2024-01-15T10:30:00Z"
}

Required Parameters

• url (string): The LinkedIn profile URL to scrape
• cookies (array): Array of cookie objects containing authentication data

Optional Parameters

• timestamp (string): ISO timestamp for when the scraping was initiated

🍪 Getting LinkedIn Cookies

⚠️ Important: Use only your own LinkedIn account cookies. Scraping other profiles without permission may violate LinkedIn's Terms of Service.

Method 1: Browser Developer Tools

1. Log into LinkedIn in your browser
2. Open Developer Tools (F12)
3. Go to Application/Storage tab
4. Navigate to Cookies → https://www.linkedin.com
5. Copy the li_at and JSESSIONID cookie values

Method 2: Browser Extensions

Use cookie export extensions to extract cookies in JSON format.

📊 Output Format

The scraper returns structured JSON data:

{
  "firstName": "John",
  "lastName": "Doe", 
  "headline": "Software Engineer at Tech Company",
  "summary": "Experienced software engineer with expertise in...",
  "locationName": "San Francisco, CA",
  "profilePicture": "https://media.licdn.com/...",
  "experiences": [
    {
      "title": "Senior Software Engineer",
      "company": "Tech Company Inc.",
      "location": "San Francisco, CA",
      "description": "Led development of...",
      "start_date": "2022-01",
      "end_date": "2024-01",
      "duration": "2 years"
    }
  ],
  "education": [
    {
      "school": "University Name",
      "degree": "Bachelor of Science",
      "field_of_study": "Computer Science", 
      "start_date": "2018",
      "end_date": "2022",
      "description": "Relevant coursework..."
    }
  ],
  "skills": [
    {
      "name": "Python",
      "endorsements": 25
    }
  ],
  "contact_info": {
    "email": "john.doe@email.com",
    "phone": "+1234567890"
  },
  "scraped_at": "2024-01-15T10:30:00Z"
}

🔧 Configuration

Logging

The scraper includes custom logging configuration:

• INFO: General operation information
• WARNING: Non-critical issues
• ERROR: Critical errors that prevent operation

Apify-specific warnings are filtered out for cleaner logs.

Error Handling

• Invalid URLs are handled gracefully
• Missing profile sections don't crash the scraper
• Network errors are logged and reported
• Authentication failures are clearly identified

🚨 Usage Guidelines

Legal Considerations

• Only scrape profiles you have permission to access
• Respect LinkedIn's Terms of Service and robots.txt
• Use scraped data responsibly and in compliance with privacy laws
• Consider rate limiting to avoid being blocked

Best Practices

• Use your own LinkedIn account cookies only
• Don't scrape profiles too frequently
• Store cookies securely and rotate them regularly
• Handle sensitive data (emails, phone numbers) appropriately
• Implement proper data retention policies

🐛 Troubleshooting

Common Issues

"li_at cookie not found"

• Ensure you've included the li_at cookie in your cookies array
• Verify the cookie value is correct and not expired

"Failed to initialize LinkedIn client"

• Check that all required cookies are provided
• Ensure cookies are in the correct format
• Verify your LinkedIn session is still active

"Invalid profile URL format"

• Use the full LinkedIn profile URL
• Supported formats: /in/username, /pub/username
• Remove any query parameters or tracking codes

Rate Limiting

• If you encounter rate limits, reduce scraping frequency
• Use delays between requests
• Consider using different LinkedIn accounts for larger operations

Debug Mode

Enable debug logging by modifying the logging level:

logging.basicConfig(level=logging.DEBUG)

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

⚠️ Disclaimer

This tool is for educational and research purposes. Users are responsible for complying with LinkedIn's Terms of Service and applicable laws. The authors are not responsible for any misuse of this software.