Indeed Job Scraper 🔍

A powerful, reliable, and fast Indeed job scraper built with Playwright and Crawlee. Extract job listings with detailed information including salary, ratings, and full descriptions without getting blocked.

✨ Features

• 🚀 Fast & Reliable - Optimized scraping with smart panel-based extraction
• 🛡️ Anti-Bot Protection - Human-like behavior to avoid Cloudflare blocks
• 💰 Rich Data - Extract salary ranges, company ratings, and full job descriptions
• 🎯 Advanced Filtering - Filter by job type, experience level, salary range, and more
• 🌍 Global Support - Works with Indeed worldwide (US, UK, Canada, Australia, etc.)
• 📊 Clean Output - Structured JSON data ready for analysis

📦 What Data Can You Extract?

Each job listing includes:

• Basic Info: Job title, company name, location
• Details: Full job description, salary range (when available)
• Metadata: Company rating, job type (full-time, part-time, etc.), date posted
• Source: Direct job URL for easy application

Sample Output

{
    "title": "Senior Software Engineer",
    "company": "Google",
    "location": "San Francisco, CA",
    "salary": "$150,000 - $200,000 a year - Full-time",
    "rating": "4.5",
    "summary": "We are looking for a talented Senior Software Engineer...",
    "url": "https://www.indeed.com/viewjob?jk=...",
    "jobType": "Full-time",
    "postedDate": "3 days ago",
    "scrapedAt": "2025-10-08T15:30:00.000Z"
}

🚀 Quick Start

Basic Search

Simply provide a job title and location:

{
    "jobTitle": "Software Engineer",
    "location": "San Francisco, CA",
    "maxJobs": 100,
    "parseCompanyDetails": true
}

Advanced Search with Filters

{
    "jobTitle": "Data Scientist",
    "location": "New York, NY",
    "maxJobs": 200,
    "parseCompanyDetails": true,
    "remote": true,
    "jobType": "fulltime",
    "datePosted": "7",
    "experienceLevel": "senior_level",
    "minSalary": 100000,
    "maxSalary": 180000
}

Target Specific Companies

{
    "jobTitle": "Frontend Developer",
    "location": "Seattle, WA",
    "maxJobs": 50,
    "parseCompanyDetails": true,
    "companies": "Google, Meta, Amazon, Microsoft",
    "excludeCompanies": "Contractor Inc"
}

⚙️ Input Configuration

Field	Type	Required	Description
jobTitle	string	✅	Job title or keywords (e.g., "Software Engineer")
location	string	✅	City, state, or "Remote"
maxJobs	number	❌	Maximum jobs to scrape (default: 50)
parseCompanyDetails	boolean	❌	Extract detailed info by clicking job cards (default: false)
saveOnlyUniqueItems	boolean	❌	Avoid duplicates (default: true)
remote	boolean	❌	Filter for remote jobs only (default: true)
jobType	string	❌	Filter by type: all, fulltime, parttime, contract, temporary, internship
datePosted	string	❌	Days ago: any, 1, 3, 7, 14, 30
experienceLevel	string	❌	Level: all, entry_level, mid_level, senior_level, director, executive
minSalary	number	❌	Minimum annual salary in USD
maxSalary	number	❌	Maximum annual salary in USD
companies	string	❌	Comma-separated list of target companies
excludeCompanies	string	❌	Comma-separated list of companies to exclude

🎯 Use Cases

Job Market Research

Track salary trends, popular skills, and hiring patterns across industries and locations.

Job Aggregation

Build your own job board by collecting and organizing listings from Indeed.

Career Planning

Monitor job postings for specific companies or roles to plan your next career move.

Recruitment Intelligence

Identify hiring patterns, skill requirements, and compensation ranges for competitive analysis.

🔧 Performance Tips

For Speed

• Set parseCompanyDetails: false for faster scraping (basic info only)
• Increase concurrency to 5-10 for parallel processing
• Reduce requestDelayMsMin and requestDelayMsMax

For Reliability

• Keep parseCompanyDetails: true for complete data
• Use concurrency: 2 (default) to avoid detection
• Set advancedMode: true for headful browser (more human-like)
• Keep delays at default values (1200-2500ms)

📊 Output Format

Results are saved to the Apify dataset in JSON format. You can export as:

• JSON - For programmatic access
• CSV - For Excel/Google Sheets
• Excel - Ready for analysis
• HTML Table - For web display

🔒 Legal & Ethics

• ✅ Only scrape publicly available job listings
• ✅ Respect Indeed's Terms of Service
• ✅ Use reasonable rate limits to avoid overloading servers
• ✅ Don't scrape personal information beyond what's publicly posted

🐛 Troubleshooting

Getting blocked by Cloudflare?

• Enable advancedMode: true for headful browser
• Reduce concurrency to 1-2
• Increase request delays

Not finding jobs?

• Check if your search parameters are too restrictive
• Try broader keywords
• Verify location spelling matches Indeed's format

Missing salary/rating data?

• Ensure parseCompanyDetails: true is set
• Not all jobs include salary information
• Some companies hide ratings

📝 Changelog

v1.0.0 (2025-10-08)

• ✅ Panel-based scraping to avoid Cloudflare blocking
• ✅ Smart salary and rating extraction
• ✅ Optimized performance with reduced delays
• ✅ Human-like clicking behavior
• ✅ Progress logging every 5 jobs

💬 Support

Need help? Have questions?

• 📧 Contact: [Your contact info]
• 🐛 Report issues: [GitHub Issues]
• 💬 Join discussion: [Discord/Forum]

📄 License

ISC

---

Made with ❤️ using Crawlee and Apify