Fast Indeed Job Scraper

A powerful and configurable web scraper for extracting job listings from Indeed.com. This actor efficiently collects job metadata and full descriptions, supporting various search parameters and customization options for production-scale data collection.

Features

• Comprehensive Job Data: Extracts job titles, companies, locations, salaries, posting dates, and detailed descriptions.
• Flexible Search Options: Accepts full search URLs or builds queries from keywords and locations.
• Pagination Support: Automatically handles multiple pages of search results.
• Customizable Scraping: Control concurrency, proxy usage, and session management.
• Output to Dataset: Results are stored in Apify's dataset for easy export and integration.

Input Configuration

The actor accepts a JSON input object with the following parameters. All fields are optional, with sensible defaults applied.

Search Parameters

Parameter	Type	Description	Default
searchUrl	string	Full Indeed search URL. If provided, keyword, location, and country are ignored.	-
keyword	string	Search keywords for job queries.	-
location	string	Geographic location filter.	-
country	string	Country-specific Indeed domain (e.g., "United States", "United Kingdom").	"United States"
posted_date	string	Filter by posting date (e.g., "1d", "7d", "30d" for days).	-

Scraping Options

Parameter	Type	Description	Default
maxItems	number	Maximum number of job listings to collect.	100
collectDetails	boolean	Whether to fetch full job descriptions from detail pages.	false
maxConcurrency	number	Number of parallel requests to make.	5
cookies / cookiesJson	object/string	Required for high-volume runs: session cookies that keep you logged in and reduce blocking.	-
proxyConfiguration	object	Proxy settings for requests.	-

Output Schema

The actor outputs job listings to the default dataset. Each item is a JSON object with the following structure:

Field	Type	Description
title	string	Job title
company	string	Company name
location	string	Job location
postedAt	string	Human-readable posting date
salary	string	Salary information (if available)
description_html	string	Full job description in HTML format
description_text	string	Plain text version of the job description
url	string	Direct link to the job posting
source	string	Data source identifier ("indeed")
search_url	string	Original search URL where the job was found

Usage

Basic Example

To scrape software engineering jobs in remote locations:

{
  "keyword": "software engineer",
  "location": "Remote",
  "maxItems": 100,
  "collectDetails": true
}

Advanced Configuration

For large-scale scraping with proxies:

{
  "searchUrl": "https://www.indeed.com/jobs?q=developer&l=New+York&fromage=7",
  "maxItems": 500,
  "collectDetails": true,
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "groups": ["RESIDENTIAL"]
  }
}

Running the Actor

1. Via Apify Console:

   • Navigate to the actor page
   • Configure input parameters
   • Click "Run" and monitor progress

2. Via API:

   curl -X POST "https://api.apify.com/v2/acts/shahidirfan100~fast-indeed-job-scraper/runs" \
        -H "Authorization: Bearer YOUR_API_TOKEN" \
        -H "Content-Type: application/json" \
        -d '{"keyword": "data scientist", "location": "San Francisco"}'
   

3. Via Apify CLI:

   $apify run shahidirfan100/fast-indeed-job-scraper --input '{"keyword": "marketing", "maxItems": 50}'
   

Configuration Tips

• Optimizing Performance: Adjust maxConcurrency based on your proxy setup. Higher values speed up scraping but may trigger rate limits.
• Handling Large Datasets: Provide valid cookies/cookiesJson for any run above ~50 jobs; Indeed blocks anonymous traffic quickly at scale.
• Proxy Usage: Enable residential proxies for better success rates on large crawls.
• Authentication: Provide cookies from a logged-in session to access premium features or reduce blocking.
• Strict Locales: Markets such as South Africa, China, and UAE enforce heavier bot controls—keep maxConcurrency ≤ 4 and expect slower warm-up even with cookies.

Getting Cookies via Cookie-Editor (Chrome)

1. Install the free Cookie-Editor extension from the Chrome Web Store.
2. Open https://www.indeed.com (or your target country domain) and sign in manually.
3. Click the Cookie-Editor icon → Export → copy the JSON string.
4. Paste that JSON into the actor input under cookiesJson.
5. For large, long-running jobs, refresh cookies regularly to keep sessions valid.

Troubleshooting

Common Issues

• Incomplete Results: Verify search parameters and increase maxItems if needed.
• Rate Limiting: Reduce maxConcurrency or add delays between requests.
• Blocking/CAPTCHA: Use high-quality proxies and valid session cookies.
• No Results: Check if the search URL is valid and accessible.

Best Practices

• Start with small test runs to validate configuration.
• Monitor the actor's log for errors and adjust settings accordingly.
• Use the dataset export features for data analysis and integration.

Support

For issues or feature requests, please contact support through the Apify platform.

---

This actor is designed for ethical web scraping. Always respect website terms of service and robots.txt files.