Job Posting Scraper with Company Contact Enrichment

Scrapes public job postings from Indeed and enriches each one with publicly available company contact info (email, phone) and firmographics pulled from the hiring company's own website.

How it works

1. You give it search queries like "marketing manager in Austin, TX".
2. It searches Indeed and collects job postings: title, company, location, salary (parsed into min/max/period), job type, and a snippet. It pages through results until it has maxResultsPerQuery postings.
3. For each unique company it looks up the Indeed profile page — via the real company link on the job card where there is one, falling back to a guessed slug — and reads the company's website, industry, size, revenue, headquarters and founding year.
4. If enrichment is on, it visits that website's likely contact pages (/contact, /contact-us, /about, /about-us, homepage) and extracts a publicly listed email and phone number, recording where each was found and how much to trust it.
5. Everything is saved to the dataset as one row per job posting. Company lookups are cached, so 20 postings from one company cost one lookup.

Output fields

Field	Notes
jobTitle, companyName, location, snippet, jobUrl	From the search results page
salary	Raw string as Indeed shows it, e.g. $70,000 - $90,000 a year
salaryMin, salaryMax, salaryPeriod, salaryCurrency	Parsed for sorting/filtering
jobType	Non-pay chips, e.g. Full-time, Hybrid work
companyWebsite, companyDomain, companyPageUrl	Company's own site, plus its Indeed profile
companyEmail, companyPhone	null when nothing public was found
companyEmailConfidence, companyPhoneConfidence	high / medium / low — see below
companyEmailSourceUrl, companyPhoneSourceUrl	The exact page it came from
industry, companySize, revenue, headquarters, founded	For lead qualification
searchQuery, scrapedAt	Provenance

Confidence levels

• high — machine-published (JSON-LD Organization data), or a mailto: / tel: link on a dedicated contact page.
• medium — a mailto: / tel: link elsewhere on the site, or text on a contact/about page.
• low — a regex match in body text on a page that isn't about contacting. Worth spot-checking before you use these.

Project structure

job-lead-scraper/
├── .actor/
│   ├── actor.json          # Actor metadata + dataset table views
│   └── input_schema.json   # Input form shown in Apify Console
├── src/
│   ├── main.js             # Crawler: search, pagination, enrichment, output
│   ├── dom.js              # Browser-side selectors (fix here on an Indeed redesign)
│   ├── extract.js          # Pure parsing: emails, phones, salary, slugs
│   ├── dom.test.js
│   └── extract.test.js
├── Dockerfile
├── package.json
└── .gitignore

Selectors and parsing live in dom.js / extract.js and are covered by unit tests that run against HTML fixtures — no network, no browser:

$npm test

When Indeed redesigns, update the fixture in src/dom.test.js to the new markup, watch it fail, then fix src/dom.js until it passes.

Searching other countries

Indeed runs a separate site per country, and the Actor picks the right one from the countries input (us by default). Every query runs against every country selected, so two queries plus ["us","uk"] is four searches.

A single query can override the list with a country: prefix:

uk: marketing manager in London
in: react developer in Bangalore
marketing manager in Austin, TX     <- uses the `countries` list

62 countries are supported, verified by probing each <cc>.indeed.com host rather than assumed — the list is in src/countries.js.

Indeed does not operate everywhere. There is no Indeed site for Nepal, Bangladesh, Sri Lanka, Kenya, or several smaller European markets, so those cannot be scraped here at any price. Asking for one fails immediately with a clear message rather than silently returning US results:

Indeed has no site for Nepal, so it cannot be searched here.

Covering those markets means a different source — local boards, or the ATS platforms (Greenhouse/Lever/Ashby) that companies run their own careers pages on. That's a separate Actor, not a flag on this one.

Targeting a specific area

Indeed defaults to a 35-unit radius, which turns a neighbourhood search into a whole-metro one. Set searchRadius when you want a specific area:

{
  "searchQueries": ["in: software engineer in Whitefield, Bengaluru"],
  "searchRadius": 5
}

That returns Whitefield, Kundalahalli and Varthur — not all of Bengaluru. One query per area is the way to cover several:

{
  "searchQueries": [
    "in: software engineer in Whitefield, Bengaluru",
    "in: software engineer in Koramangala, Bengaluru",
    "in: software engineer in Electronic City, Bengaluru"
  ],
  "searchRadius": 5
}

When a run targets exactly one country, the Actor automatically requests proxy IPs from that country. A US IP browsing in.indeed.com is itself a blocking signal — country-matched IPs got through on the first attempt where US IPs had exhausted every retry.

Run locally first

npm install
npx playwright install --with-deps chromium
mkdir -p storage/key_value_stores/default
cat > storage/key_value_stores/default/INPUT.json << 'EOF'
{
  "searchQueries": ["marketing manager in Austin, TX"],
  "maxResultsPerQuery": 10,
  "enrichWithCompanyContact": true,
  "maxConcurrency": 1
}
EOF
npm start

Results land in storage/datasets/default/. The final log line reports the enrichment hit rate (Email found for 6 (60%)) — use real numbers from your own runs in the Store listing instead of a guess.

Deploy to Apify

npm install -g apify-cli
apify login
cd job-lead-scraper
apify push

Test with a small maxResultsPerQuery (5-10) before running it at scale.

Publishing & monetization

Same flow as any Apify Actor:

1. Console → your Actor → Publication tab → fill in title, description, example input/output.
2. Monetization tab → pick pricing. Pay-per-result fits this well — e.g. charge per job lead returned, or a slightly higher rate per lead with a contact email found (extra value = fair to charge more for it). onlyReturnEnrichedLeads lets buyers opt into paying only for enriched rows.
3. Settings → Payments → connect payout method.
4. Submit for Store review.

Measured results

From a live run (10 leads, marketing manager in Austin, TX, US, residential proxy):

Field	Coverage
jobTitle, jobUrl, postedAt, companyPageUrl	10/10
salary	9/10
companyWebsite	9/10
companySize	9/10
industry	8/10
companyPhone	5/10
companyEmail	3/10

Use numbers from your own runs in the Store listing. The email rate is the one to be careful about: ~30% is normal and honest for this kind of enrichment.

Proxies: residential is mandatory

Indeed 403s datacenter IP ranges outright, so the default datacenter proxy returns nothing at all. Use the RESIDENTIAL group. Even then, expect the Actor to burn several retries before it draws an IP Indeed accepts — a successful run has been observed to take 11 attempts. That is why maxRequestRetries is 20 with exponential backoff; a lower budget fails runs that would otherwise have succeeded.

Do not use UNBLOCKER. It terminates TLS with its own CA (needing --ignore-certificate-errors) and then proxies too slowly for a browser — every navigation times out at 60s.

Important limitations (be upfront about these with buyers)

• Not every company will have a findable email/phone. Many company sites hide contact info behind forms, or don't list a direct email at all. Expect a meaningful chunk of companyEmail/companyPhone fields to come back null — this is normal and any enrichment tool has the same ceiling. Be transparent about this in your Store listing rather than promising 100% match rates.
• Indeed's HTML structure changes periodically. The selectors in src/dom.js (div.job_seen_beacon, h2.jobTitle, etc.) may need updating if Indeed redesigns its results page. The Actor now fails loudly instead of returning an empty dataset when it can't parse any job cards, so a redesign shows up as a failed run rather than a silently empty one.
• Company page slug guessing is approximate. Where the job card links straight to the Indeed company page, that link is used. Otherwise the company name → slug conversion is a best-effort guess and some companies won't resolve, especially ones with special characters or very short names.
• Contact info is scraped, not verified. No email is sent and no deliverability check is run. Use the confidence field, and consider validating before a large send.
• This scrapes only publicly published data: job listings meant for public search, and contact info companies themselves chose to publish on their own websites. It does not access anything behind a login, and does not scrape personal social profiles. Contact info is company-level (info@, careers@), not personal inboxes — role addresses are ranked above personal-looking ones on purpose.
• Use Apify Proxy and keep concurrency low (1-3) to avoid IP blocks — already wired into the input schema by default.

Shipped in v2

• Pagination. maxResultsPerQuery above ~15 previously returned only the first page. It now pages through with &start=.
• Blocks fail loudly. A CAPTCHA used to be swallowed, so a blocked run looked like a successful run that found nothing. It now retires the proxy session and retries on a fresh IP.
• Confidence + source URL on every email and phone.
• Firmographics: industry, company size, revenue, headquarters, founded.
• onlyReturnEnrichedLeads to drop rows with no email.
• Company lookups cached per run instead of repeated per job posting.
• Better contact extraction: JSON-LD first, then mailto:/tel: links, then visible text — so timestamps and Sentry DSNs stop being mistaken for phone numbers and emails, and role addresses outrank noreply@.
• Structured salary fields, and pay chips told apart from Full-time.
• Indeed redirect URLs unwrapped, so companyWebsite is the real site.
• Faster: images/fonts/CSS blocked during enrichment; enrichment runs on a second tab so the results page survives for pagination.
• maxDaysOld filter for fresher leads.
• 62 countries, with a per-query country: prefix override.
• Unit tests for every selector and parser.

Ideas for a v3

• Other job boards. ZipRecruiter, and company career pages directly (many run Greenhouse/Lever/Ashby, which have stable, predictable markup — a much better ROI than scraping another board's HTML). dom.js is the seam: add a per-source module behind the same shape.
• Per-job enrichment requests. Enrichment currently runs inline in the search-page handler under a time budget. Making each company its own Crawlee request would give per-company retries and better parallelism.
• Deliverability scoring on emails (MX lookup, catch-all detection) — a clear upsell over "we found a string that looks like an email".
• Full job descriptions from the posting page (JSON-LD JobPosting is usually right there), plus the apply URL.
• Remote-only filter. Deliberately left out: it needs an undocumented Indeed sc=0kf:attr(...) token that changes without notice. Worth adding only once verified against live results.
• Contact-form URL as a fallback when no email exists — still actionable for a buyer, and it would lift the effective match rate a lot.
• Pay-per-event billing (Actor.charge) instead of pay-per-result, so you can charge separately for a bare lead vs. an enriched one.