Indeed Job Scraper

Scrapes job listings from Indeed.com - works globally with 60+ country domains including US, UK, Germany, Switzerland, Austria, and more.

Three Scraping Modes

This scraper supports three distinct modes for different use cases:

Mode	Input	Output	Use Case
SEARCH MODE	Search query + location + country	Basic job data from SERP	Fast discovery of new jobs
SEARCH + DETAILS MODE	includeJobDetails: true (default)	Full job data with descriptions	Complete data collection in one run
DIRECT URLS MODE	directUrls: [...]	Full job data + job_status	Still-alive checks, re-scraping specific jobs

Features

• Global coverage: Works with any Indeed country domain (ch, de, at, us, uk, fr, etc.)
• Two-phase scraping: Collects jobs from search results, then fetches full details
• Anti-bot bypass: Uses Playwright with residential proxies for reliable scraping
• Rich data extraction: Title, company, location, salary, description, company ratings, and more
• Standardized output: Consistent JobListing schema across all job scrapers

Input

Parameter	Type	Default	Description
directUrls	array	[]	Direct job URLs to scrape (skips search mode)
searchQuery	string	""	Job title, keywords, or company name
location	string	""	City, state, or postal code
country	string	"ch"	Indeed country domain (ch, de, at, us, uk, etc.)
maxResults	integer	100	Maximum number of jobs to scrape (1-1000)
includeJobDetails	boolean	true	Fetch full job descriptions (slower but more data)
proxyConfiguration	object	Residential	Apify proxy settings

Mode 1: SEARCH MODE (Fast)

Quick search without visiting detail pages:

• Use case: Fast job discovery, mass collection
• Speed: Fast - only scrapes search results pages
• Output: Basic job data (title, company, location, snippet)

{
  "searchQuery": "Developer",
  "location": "Zürich",
  "country": "ch",
  "maxResults": 200,
  "includeJobDetails": false
}

Mode 2: SEARCH + DETAILS MODE (Complete Data)

Default mode with includeJobDetails: true:

• Use case: Complete data collection in one run
• Speed: Moderate - visits detail pages for each job
• Output: Full descriptions, benefits, apply URLs

{
  "searchQuery": "Software Engineer",
  "location": "Zürich",
  "country": "ch",
  "maxResults": 50
}

Mode 3: DIRECT URLS MODE (Still Alive Checks)

When directUrls is provided, the scraper operates in direct mode:

• Skips search phase - Goes directly to provided job URLs
• Job status detection - Returns online, offline, expired, or unknown
• Full data extraction - Same as detail page scraping
• Use case: Periodic "still alive" checks, re-scraping specific jobs after deduplication

{
  "directUrls": [
    "https://ch.indeed.com/viewjob?jk=abc123def456",
    "https://de.indeed.com/viewjob?jk=xyz789uvw012",
    "https://www.indeed.com/viewjob?jk=qrs345tuv678"
  ],
  "country": "ch"
}

Direct URLs mode workflow:

1. Provide array of Indeed job detail URLs (must include jk= parameter)
2. Scraper visits each URL directly
3. Detects if job is still online or has been removed
4. Extracts full job data if available
5. Returns job_status field indicating availability

Supported Countries

• 🇨🇭 Switzerland (ch.indeed.com)
• 🇩🇪 Germany (de.indeed.com)
• 🇦🇹 Austria (at.indeed.com)
• 🇺🇸 United States (indeed.com)
• 🇬🇧 United Kingdom (uk.indeed.com)
• 🇫🇷 France (fr.indeed.com)
• 🇮🇹 Italy (it.indeed.com)
• 🇪🇸 Spain (es.indeed.com)
• 🇳🇱 Netherlands (nl.indeed.com)
• 🇧🇪 Belgium (be.indeed.com)
• 🇨🇦 Canada (ca.indeed.com)
• 🇦🇺 Australia (au.indeed.com)
• 🇮🇳 India (in.indeed.com)
• 🇸🇬 Singapore (sg.indeed.com)
• 🇯🇵 Japan (jp.indeed.com)
• 🇧🇷 Brazil (br.indeed.com)
• 🇲🇽 Mexico (mx.indeed.com)

Output

Each job listing includes:

{
  "id": "abc123xyz",
  "title": "Software Engineer",
  "company": "Tech Corp",
  "location": "Zürich, ZH",
  "canton": null,
  "job_status": "online",
  "top_listing": true,
  "employment_type": "full-time",
  "workload_min": null,
  "workload_max": null,
  "remote_option": "hybrid",
  "salary_text": "CHF 120'000 - 150'000 per year",
  "description_snippet": "We are looking for...",
  "description_full": "Full job description...",
  "requirements": [],
  "posted_at": "2024-12-04T10:00:00.000Z",
  "expires_at": null,
  "source_url": "https://ch.indeed.com/viewjob?jk=abc123xyz",
  "source_platform": "indeed.com (ch)",
  "contact_salutation": null,
  "contact_firstname": null,
  "contact_lastname": null,
  "contact_email": null,
  "contact_phone": null,
  "contact_position": null,
  "contact_raw": null,
  "apply_url": "https://company.com/apply",
  "apply_email": null,
  "company_url": null,
  "company_description": null,
  "company_rating": 4.2,
  "company_review_count": 156,
  "company_social_urls": null,
  "company_benefits": ["Health insurance", "Flexible hours"],
  "scraped_at": "2024-12-05T12:00:00.000Z"
}

Output Fields

Field	Description
id	Indeed job key (from jk= parameter)
title	Job title
company	Company name
location	City/location as displayed
canton	Swiss canton code (for Swiss jobs only)
job_status	Job availability: online, offline, expired, unknown (Direct URLs mode only)
top_listing	Boolean - if job is sponsored/featured (Search mode only)
employment_type	full-time, part-time, contract, temporary, internship, apprenticeship
remote_option	remote, hybrid, or null
salary_text	Salary as displayed
description_snippet	First 500 characters from search results
description_full	Complete job description (Detail mode only)
requirements	Array of job requirements (usually empty for Indeed)
posted_at	Publication date (parsed from relative dates like "3 days ago")
expires_at	Expiration date (usually not available)
source_url	Link to job posting on Indeed
source_platform	"indeed.com (ch)", "indeed.com (de)", etc.
company_rating	Company rating on Indeed (0-5 stars)
company_review_count	Number of company reviews
company_benefits	Array of benefits (Detail mode only)
scraped_at	Timestamp when job was scraped

Notes on job_status field:

• Only populated in Direct URLs mode (Mode 3)
• In Search modes (Mode 1 & 2), this field is null since jobs from search results are assumed to be online
• Values:
  • online - Job page loaded successfully with description
  • offline - Job page returns "not found" or similar messages
  • expired - Job explicitly marked as expired
  • unknown - Unable to load or parse job page

Example Usage

Search for IT jobs in Zurich

{
  "searchQuery": "Software Engineer",
  "location": "Zürich",
  "country": "ch",
  "maxResults": 50
}

Search for remote jobs in Germany

{
  "searchQuery": "remote developer",
  "location": "",
  "country": "de",
  "maxResults": 100
}

Quick search without full descriptions

{
  "searchQuery": "Marketing",
  "location": "Berlin",
  "country": "de",
  "maxResults": 200,
  "includeJobDetails": false
}

Direct URLs - Still Alive Check

{
  "directUrls": [
    "https://ch.indeed.com/viewjob?jk=abc123def456",
    "https://ch.indeed.com/viewjob?jk=xyz789uvw012"
  ],
  "country": "ch"
}

How It Works

1. Search Phase: Crawls Indeed search results pages, collecting job cards with basic info (title, company, location, salary snippet)
2. Detail Phase (optional): Visits each job's detail page to extract full description, benefits, and apply URL
3. Validation: Each job is validated against the JobListing schema before saving
4. Pagination: Automatically follows pagination to collect up to maxResults jobs

Search Mode (Modes 1 & 2)

• Starts from search results page with your query and location
• Extracts job cards containing basic information
• Optionally visits each detail page for complete data
• Follows pagination until maxResults is reached

Direct URLs Mode (Mode 3)

• Skips search phase entirely
• Goes directly to provided job URLs
• Checks if page loads successfully (online/offline detection)
• Extracts full job data if page is online
• Returns job_status field for monitoring

Proxy Recommendations

Indeed has anti-bot protection. For best results:

• Use Residential proxies (recommended)
• Avoid Datacenter proxies (may be blocked)
• Keep maxResults reasonable (<500 per run)
• Add delays between runs if scraping frequently

Local Development

# Install dependencies
npm install

# Build TypeScript
npm run build

# Run locally
apify run --purge

# Push to Apify
apify push

Common Use Cases

1. Job market analysis: Track hiring trends across countries
2. Competitive intelligence: Monitor competitors' hiring
3. Job aggregation: Build job search platforms
4. Still-alive monitoring: Check if previously scraped jobs are still active (Direct URLs mode)
5. Post-deduplication enrichment: Scrape basic data first, then fetch details for new jobs only
6. Career research: Analyze salary ranges and requirements
7. Global job search: Compare job markets across different countries

Notes

• Indeed's page structure changes frequently; selectors may need updates
• Some job details may only be available on the detail page
• Rate limiting applies - don't run too many concurrent requests
• Respects Indeed's robots.txt and terms of service
• Job IDs are extracted from the jk= parameter in URLs

---

Part of the Santamaria Job Scrapers Suite - Professional-grade job data for the DACH region and beyond.

Need help with integration, aggregation, or custom scraping solutions? Contact us at contact@alessandrosantamaria.com