Haystack Jobs Scraper

An Apify Actor (Node.js / Crawlee) that scrapes tech job listings from haystackapp.io by keyword, location and filters.

How it works

haystackapp.io has two very different rendering modes, so this Actor uses a hybrid crawler:

1. Playwright crawler opens the filterable search page (https://haystackapp.io/jobs?q=...&location=...). This page's initial HTML is empty ("0 jobs found") — the real results are fetched by React after the page loads — so a real browser is required just to collect the list of job URLs.
2. Cheerio crawler then fetches each individual job page (https://haystackapp.io/jobs/{uuid}) with plain HTTP requests. These pages are fully server-rendered, so a lightweight, fast HTTP crawler is enough to extract full job details (title, description, skills, apply link, etc.) — no browser needed for this part.

This keeps the Actor fast and cheap: only one browser page is ever opened, no matter how many jobs you scrape.

Input

Field	Type	Description
keyword	string	Job title / keyword, e.g. "Software Engineer"
location	string	City, e.g. "London"
country	select	Restrict to one country
workMode	select	onsite, hybrid, or remote
maxItems	integer	Max number of jobs to return (default 50)
proxyConfiguration	object	Apify Proxy config (recommended)

Output

One dataset item per job, e.g.:

{
  "url": "https://haystackapp.io/jobs/785dd571-1826-4dbe-8008-7596b2e5d430",
  "title": "Lead Software Engineer - Proxy/SSE Network Security",
  "headerRaw": "J.P. Morgan London, UK Posted 24 Jul 2026",
  "postedDate": "24 Jul 2026",
  "workType": "On Site",
  "level": "Mid Senior",
  "skills": ["AWS", "Encryption", "OAuth", "SAML", "Agile", "DNS", "HTTP", "HTTPS", "JWT", "SAFe", "Zero Trust"],
  "description": "...",
  "applyUrl": "https://.../track-redirect?job_id=...",
  "scrapedAt": "2026-08-02T12:00:00.000Z"
}

⚠️ Before you run this in production

I built and structured this Actor entirely from the site's server-rendered HTML, fetched via a text-based tool — I was not able to render the page in a real browser to inspect it with devtools during development. That means:

• The job-link discovery selector (a[href*="/jobs/"] filtered by a UUID regex) is solid — it's based on the actual URLs the site returns.
• The "load more" / pagination handling on the search page is a best-effort guess (it tries clicking anything that looks like a "load more" button, and also scrolls, in case it's infinite-scroll instead). This is the part most likely to need adjustment.
• The detail-page field extraction (company, workType, level) uses text-pattern heuristics rather than confirmed CSS classes, since I couldn't inspect the live DOM's class names. title, skills, description, and applyUrl are extracted more robustly and should be reliable.

Recommended next step: run the Actor once with maxItems: 5 on the Apify platform (or locally with apify run), open the dataset, and compare a couple of records against the live pages. If company comes back null or workType/level look off, open devtools on a job page, find the real selectors, and I'll tighten the Cheerio parsing in src/main.js — that's a quick fix once we can see real output.

Local development

npm install
apify run

(Requires the Apify CLI. Input is read from storage/key_value_stores/default/INPUT.json — create one based on the input schema above, or set input via apify run on the platform.)

Deploy to Apify

apify login
apify push