# Indeed Job Posting Scraper with Company Contact Enrichment (`ayush_labs/job-lead-scraper`) Actor

Scrapes public job postings from Indeed and enriches each one with publicly listed company contact info (email, phone) and firmographics pulled from the hiring company's own website.

- **URL**: https://apify.com/ayush\_labs/job-lead-scraper.md
- **Developed by:** [Ayush Khatiwada](https://apify.com/ayush_labs) (community)
- **Categories:** Jobs, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Job Posting Scraper with Company Contact Enrichment

Scrapes public job postings from Indeed and enriches each one with publicly
available company contact info (email, phone) and firmographics pulled from the
hiring company's own website.

### How it works

1. You give it search queries like `"marketing manager in Austin, TX"`.
2. It searches Indeed and collects job postings: title, company, location,
   salary (parsed into min/max/period), job type, and a snippet. It pages
   through results until it has `maxResultsPerQuery` postings.
3. For each **unique** company it looks up the Indeed profile page — via the
   real company link on the job card where there is one, falling back to a
   guessed slug — and reads the company's website, industry, size, revenue,
   headquarters and founding year.
4. If enrichment is on, it visits that website's likely contact pages
   (`/contact`, `/contact-us`, `/about`, `/about-us`, homepage) and extracts a
   publicly listed email and phone number, recording **where** each was found
   and how much to trust it.
5. Everything is saved to the dataset as one row per job posting. Company
   lookups are cached, so 20 postings from one company cost one lookup.

### Output fields

| Field | Notes |
| --- | --- |
| `jobTitle`, `companyName`, `location`, `snippet`, `jobUrl` | From the search results page |
| `salary` | Raw string as Indeed shows it, e.g. `$70,000 - $90,000 a year` |
| `salaryMin`, `salaryMax`, `salaryPeriod`, `salaryCurrency` | Parsed for sorting/filtering |
| `jobType` | Non-pay chips, e.g. `Full-time, Hybrid work` |
| `companyWebsite`, `companyDomain`, `companyPageUrl` | Company's own site, plus its Indeed profile |
| `companyEmail`, `companyPhone` | `null` when nothing public was found |
| `companyEmailConfidence`, `companyPhoneConfidence` | `high` / `medium` / `low` — see below |
| `companyEmailSourceUrl`, `companyPhoneSourceUrl` | The exact page it came from |
| `industry`, `companySize`, `revenue`, `headquarters`, `founded` | For lead qualification |
| `searchQuery`, `scrapedAt` | Provenance |

#### Confidence levels

- **high** — machine-published (JSON-LD `Organization` data), or a `mailto:` /
  `tel:` link on a dedicated contact page.
- **medium** — a `mailto:` / `tel:` link elsewhere on the site, or text on a
  contact/about page.
- **low** — a regex match in body text on a page that isn't about contacting.
  Worth spot-checking before you use these.

### Project structure

```
job-lead-scraper/
├── .actor/
│   ├── actor.json          # Actor metadata + dataset table views
│   └── input_schema.json   # Input form shown in Apify Console
├── src/
│   ├── main.js             # Crawler: search, pagination, enrichment, output
│   ├── dom.js              # Browser-side selectors (fix here on an Indeed redesign)
│   ├── extract.js          # Pure parsing: emails, phones, salary, slugs
│   ├── dom.test.js
│   └── extract.test.js
├── Dockerfile
├── package.json
└── .gitignore
```

Selectors and parsing live in `dom.js` / `extract.js` and are covered by unit
tests that run against HTML fixtures — no network, no browser:

```bash
npm test
```

When Indeed redesigns, update the fixture in `src/dom.test.js` to the new
markup, watch it fail, then fix `src/dom.js` until it passes.

### Searching other countries

Indeed runs a separate site per country, and the Actor picks the right one from
the `countries` input (`us` by default). Every query runs against every country
selected, so two queries plus `["us","uk"]` is four searches.

A single query can override the list with a `country:` prefix:

```
uk: marketing manager in London
in: react developer in Bangalore
marketing manager in Austin, TX     <- uses the `countries` list
```

**62 countries are supported**, verified by probing each `<cc>.indeed.com` host
rather than assumed — the list is in `src/countries.js`.

**Indeed does not operate everywhere.** There is no Indeed site for Nepal,
Bangladesh, Sri Lanka, Kenya, or several smaller European markets, so those
cannot be scraped here at any price. Asking for one fails immediately with a
clear message rather than silently returning US results:

```
Indeed has no site for Nepal, so it cannot be searched here.
```

Covering those markets means a different source — local boards, or the ATS
platforms (Greenhouse/Lever/Ashby) that companies run their own careers pages
on. That's a separate Actor, not a flag on this one.

### Targeting a specific area

Indeed defaults to a **35-unit radius**, which turns a neighbourhood search into
a whole-metro one. Set `searchRadius` when you want a specific area:

```json
{
  "searchQueries": ["in: software engineer in Whitefield, Bengaluru"],
  "searchRadius": 5
}
```

That returns Whitefield, Kundalahalli and Varthur — not all of Bengaluru. One
query per area is the way to cover several:

```json
{
  "searchQueries": [
    "in: software engineer in Whitefield, Bengaluru",
    "in: software engineer in Koramangala, Bengaluru",
    "in: software engineer in Electronic City, Bengaluru"
  ],
  "searchRadius": 5
}
```

When a run targets exactly one country, the Actor automatically requests proxy
IPs from that country. A US IP browsing `in.indeed.com` is itself a blocking
signal — country-matched IPs got through on the first attempt where US IPs had
exhausted every retry.

### Run locally first

```bash
npm install
npx playwright install --with-deps chromium
mkdir -p storage/key_value_stores/default
cat > storage/key_value_stores/default/INPUT.json << 'EOF'
{
  "searchQueries": ["marketing manager in Austin, TX"],
  "maxResultsPerQuery": 10,
  "enrichWithCompanyContact": true,
  "maxConcurrency": 1
}
EOF
npm start
```

Results land in `storage/datasets/default/`. The final log line reports the
enrichment hit rate (`Email found for 6 (60%)`) — use real numbers from your own
runs in the Store listing instead of a guess.

### Deploy to Apify

```bash
npm install -g apify-cli
apify login
cd job-lead-scraper
apify push
```

Test with a small `maxResultsPerQuery` (5-10) before running it at scale.

### Publishing & monetization

Same flow as any Apify Actor:

1. Console → your Actor → **Publication** tab → fill in title, description,
   example input/output.
2. **Monetization** tab → pick pricing. **Pay-per-result** fits this well —
   e.g. charge per job lead returned, or a slightly higher rate per lead
   *with* a contact email found (extra value = fair to charge more for it).
   `onlyReturnEnrichedLeads` lets buyers opt into paying only for enriched rows.
3. **Settings → Payments** → connect payout method.
4. Submit for Store review.

### Measured results

From a live run (10 leads, `marketing manager in Austin, TX`, US, residential proxy):

| Field | Coverage |
| --- | --- |
| `jobTitle`, `jobUrl`, `postedAt`, `companyPageUrl` | 10/10 |
| `salary` | 9/10 |
| `companyWebsite` | 9/10 |
| `companySize` | 9/10 |
| `industry` | 8/10 |
| `companyPhone` | 5/10 |
| `companyEmail` | 3/10 |

Use numbers from your own runs in the Store listing. The email rate is the one
to be careful about: ~30% is normal and honest for this kind of enrichment.

### Proxies: residential is mandatory

Indeed 403s datacenter IP ranges outright, so the default datacenter proxy
returns nothing at all. Use the **RESIDENTIAL** group. Even then, expect the
Actor to burn several retries before it draws an IP Indeed accepts — a
successful run has been observed to take 11 attempts. That is why
`maxRequestRetries` is 20 with exponential backoff; a lower budget fails
runs that would otherwise have succeeded.

**Do not use UNBLOCKER.** It terminates TLS with its own CA (needing
`--ignore-certificate-errors`) and then proxies too slowly for a browser —
every navigation times out at 60s.

### Important limitations (be upfront about these with buyers)

- **Not every company will have a findable email/phone.** Many company sites
  hide contact info behind forms, or don't list a direct email at all. Expect
  a meaningful chunk of `companyEmail`/`companyPhone` fields to come back
  `null` — this is normal and any enrichment tool has the same ceiling. Be
  transparent about this in your Store listing rather than promising 100%
  match rates.
- **Indeed's HTML structure changes periodically.** The selectors in
  `src/dom.js` (`div.job_seen_beacon`, `h2.jobTitle`, etc.) may need updating
  if Indeed redesigns its results page. The Actor now **fails loudly** instead
  of returning an empty dataset when it can't parse any job cards, so a
  redesign shows up as a failed run rather than a silently empty one.
- **Company page slug guessing is approximate.** Where the job card links
  straight to the Indeed company page, that link is used. Otherwise the company
  name → slug conversion is a best-effort guess and some companies won't
  resolve, especially ones with special characters or very short names.
- **Contact info is scraped, not verified.** No email is sent and no
  deliverability check is run. Use the confidence field, and consider
  validating before a large send.
- **This scrapes only publicly published data**: job listings meant for
  public search, and contact info companies themselves chose to publish on
  their own websites. It does not access anything behind a login, and does
  not scrape personal social profiles. Contact info is company-level
  (`info@`, `careers@`), not personal inboxes — role addresses are ranked
  above personal-looking ones on purpose.
- **Use Apify Proxy and keep concurrency low** (1-3) to avoid IP blocks —
  already wired into the input schema by default.

### Shipped in v2

- **Pagination.** `maxResultsPerQuery` above ~15 previously returned only the
  first page. It now pages through with `&start=`.
- **Blocks fail loudly.** A CAPTCHA used to be swallowed, so a blocked run
  looked like a successful run that found nothing. It now retires the proxy
  session and retries on a fresh IP.
- **Confidence + source URL** on every email and phone.
- **Firmographics**: industry, company size, revenue, headquarters, founded.
- **`onlyReturnEnrichedLeads`** to drop rows with no email.
- **Company lookups cached** per run instead of repeated per job posting.
- **Better contact extraction**: JSON-LD first, then `mailto:`/`tel:` links,
  then visible text — so timestamps and Sentry DSNs stop being mistaken for
  phone numbers and emails, and role addresses outrank `noreply@`.
- **Structured salary** fields, and pay chips told apart from `Full-time`.
- **Indeed redirect URLs unwrapped**, so `companyWebsite` is the real site.
- **Faster**: images/fonts/CSS blocked during enrichment; enrichment runs on a
  second tab so the results page survives for pagination.
- **`maxDaysOld`** filter for fresher leads.
- **62 countries**, with a per-query `country:` prefix override.
- **Unit tests** for every selector and parser.

### Ideas for a v3

- **Other job boards.** ZipRecruiter, and company career pages directly (many
  run Greenhouse/Lever/Ashby, which have stable, predictable markup — a much
  better ROI than scraping another board's HTML). `dom.js` is the seam: add a
  per-source module behind the same shape.
- **Per-job enrichment requests.** Enrichment currently runs inline in the
  search-page handler under a time budget. Making each company its own Crawlee
  request would give per-company retries and better parallelism.
- **Deliverability scoring** on emails (MX lookup, catch-all detection) — a
  clear upsell over "we found a string that looks like an email".
- **Full job descriptions** from the posting page (JSON-LD `JobPosting` is
  usually right there), plus the apply URL.
- **Remote-only filter.** Deliberately left out: it needs an undocumented
  Indeed `sc=0kf:attr(...)` token that changes without notice. Worth adding
  only once verified against live results.
- **Contact-form URL** as a fallback when no email exists — still actionable
  for a buyer, and it would lift the effective match rate a lot.
- **Pay-per-event billing** (`Actor.charge`) instead of pay-per-result, so you
  can charge separately for a bare lead vs. an enriched one.

# Actor input Schema

## `searchQueries` (type: `array`):

One search per line. Include job title and location, e.g. 'sales manager in Chicago, IL' or 'react developer remote'.

## `countries` (type: `array`):

Which Indeed sites to search. Every query runs against each country selected here. A single query can override this with a prefix, e.g. "uk: marketing manager in London". Indeed has no site for Nepal, Bangladesh, Sri Lanka, Kenya or several smaller markets — those cannot be searched.

## `searchRadius` (type: `integer`):

How far around the location to search, in miles (km outside the US). Indeed defaults to 35, which turns a neighbourhood search into a whole-metro one — set 0 to 10 when targeting a specific area such as 'Whitefield, Bengaluru'. Leave empty for Indeed's default.

## `maxResultsPerQuery` (type: `integer`):

Maximum job postings to scrape per search query. Indeed returns 10 per page, so higher values page through the results.

## `maxDaysOld` (type: `integer`):

Restrict results to recently posted jobs. 0 means no date filter. Fresh postings usually make better leads.

## `remoteOnly` (type: `boolean`):

Keep only postings Indeed flags as remote.

## `enrichWithCompanyContact` (type: `boolean`):

Visits each hiring company's public website to look for a contact email and phone number (checks pages like /contact, /about, homepage). Turn off for a much faster, cheaper run — you still get the company website, industry and size from Indeed, just no email or phone.

## `onlyReturnEnrichedLeads` (type: `boolean`):

Drop rows where no company email was found. Useful on pay-per-result pricing if you only want fully enriched leads. Note: this filters output, not how many postings are scanned.

## `maxContactPagesPerCompany` (type: `integer`):

How many candidate pages (/contact, /contact-us, /about, /about-us, homepage) to try per company before giving up. Higher finds slightly more contacts but costs proportionally more time.

## `maxConcurrency` (type: `integer`):

How many search pages to process in parallel. Lower is safer against rate limits and CAPTCHAs.

## `debugSaveHtml` (type: `boolean`):

Saves the raw Indeed HTML to the key-value store. Use this when fields start coming back empty: the markup is what you need to repair the selectors in src/dom.js.

## `proxyConfiguration` (type: `object`):

Indeed blocks datacenter IP ranges outright — a datacenter proxy returns 403 on every request. RESIDENTIAL is the only configuration that works; getting through takes a few automatic retries while it finds a clean IP. Do not use UNBLOCKER: it proxies too slowly for a browser and every navigation times out.

## Actor input object example

```json
{
  "searchQueries": [
    "marketing manager in Austin, TX"
  ],
  "countries": [
    "us"
  ],
  "maxResultsPerQuery": 20,
  "maxDaysOld": 0,
  "remoteOnly": false,
  "enrichWithCompanyContact": true,
  "onlyReturnEnrichedLeads": false,
  "maxContactPagesPerCompany": 3,
  "maxConcurrency": 2,
  "debugSaveHtml": false,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `leads` (type: `string`):

Every scraped posting with its company contact details.

## `leadsCsv` (type: `string`):

The same leads as a spreadsheet, for importing into a CRM.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "marketing manager in Austin, TX"
    ],
    "countries": [
        "us"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("ayush_labs/job-lead-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["marketing manager in Austin, TX"],
    "countries": ["us"],
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("ayush_labs/job-lead-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "marketing manager in Austin, TX"
  ],
  "countries": [
    "us"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call ayush_labs/job-lead-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ayush_labs/job-lead-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/w35JoPXo46HFpYbHR/builds/TuFWIsQW8qdMH3Xx5/openapi.json