Job Listing Scraper

This repository contains a small Apify Actor that collects public job listings from multiple fallback sources, normalizes the fields, removes duplicate URLs, and writes the results to an Apify Dataset.

It was built as a focused portfolio project to demonstrate TypeScript, Node.js, Crawlee, Playwright, Apify Actor lifecycle/storage, pagination, validation, retries, and professional delivery without adding infrastructure the scraper does not need.

Features

• Structured Actor input with configurable start URLs and result limit.
• Playwright-based browser crawling managed by Crawlee.
• Public HTML/API extraction for title, company, location, employment type, posted label, summary, URL, and visible salary.
• Three default sources: Arbeitnow HTML, Remotive's public remote-jobs API, and RemoteOK's public API. A failed source is logged while the other sources can continue.
• Whitespace cleanup, URL normalization, optional-field handling, and a 1,000-character summary cap.
• URL-based deduplication and a maximum of 500 results per run.
• Pagination through Arbeitnow's public Next link.
• Crawlee retries failed requests twice and reports requests that still fail.
• Apify Dataset output locally and in the Apify platform.
• Unit tests for normalization plus GitHub Actions CI.

Technology stack

• TypeScript and Node.js
• Apify SDK for Actor lifecycle, input, and Dataset storage
• Crawlee PlaywrightCrawler for the request queue, retries, concurrency, and crawl loop
• Playwright for browser navigation and DOM extraction
• Arbeitnow public job-board HTML as the data source
• Docker using Apify's Playwright Chrome base image

Architecture

User Input
    ↓
Apify Actor
    ↓
Apify SDK (lifecycle, input, Dataset)
    ↓
Crawlee PlaywrightCrawler (queue, retry, concurrency)
    ↓
Playwright browser
    ↓
Public Arbeitnow job listing website
    ↓
Extraction / normalization / deduplication
    ↓
Apify Dataset

Playwright is used because the Actor reads the rendered page through a real browser and stable semantic DOM attributes. Crawlee is used instead of a manual browser loop because it provides request queues, retry handling, concurrency control, and crawl statistics around Playwright.

Input

{
  "startUrls": [
    { "url": "https://www.arbeitnow.com/" }
  ],
  "maxResults": 25
}

startUrls accepts public HTTP(S) listing pages or APIs. By default the Actor tries Arbeitnow, Remotive, and RemoteOK. maxResults is clamped to 1–500.

Output

Each Dataset item has this shape. Missing values are null; the Actor does not invent data.

{
  "title": "Example job title",
  "company": "Example company",
  "location": "Berlin",
  "employmentType": "Vollzeit",
  "datePosted": "Posted 5 hours ago",
  "summary": "A cleaned summary taken from the public listing card...",
  "url": "https://www.arbeitnow.com/jobs/companies/example/example-job-123",
  "salary": null,
  "source": "arbeitnow.com"
}

Run locally

npm install
npm run build
npm test
npm start

For a small real run, provide local Apify input through the environment:

$APIFY_INPUT_JSON='{"maxResults":5}' npm start

The Apify SDK writes local Dataset output under storage/, which is ignored by Git. The local browser launch uses /usr/bin/google-chrome by default; set BROWSER_EXECUTABLE_PATH if your Chrome binary is elsewhere.

Run on Apify

After authenticating the Apify CLI, deploy from this directory:

apify login
apify push
apify call --input-file=example-input.json

The Actor input form is defined in .actor/input_schema.json. The Docker image includes the Playwright browser runtime in Apify's cloud environment.

Error handling and limitations

Crawlee retries a failed request twice. A request that still fails is logged by failedRequestHandler, while successful pages continue. The Actor intentionally uses only public pages; it does not bypass CAPTCHA, authentication, paywalls, Cloudflare challenges, robots restrictions, or rate limits. The extraction depends on Arbeitnow's current HTML attributes, so a site redesign can require selector updates. The public listing card does not always expose a full description or salary, so those fields can be null.

Testing

npm run build runs strict TypeScript checking. npm test runs four Node test-runner tests covering whitespace cleanup, URL normalization, salary parsing, and optional-field normalization. A GitHub Actions workflow runs both commands on pushes and pull requests.

Project structure

src/main.ts              Actor lifecycle, crawler, extraction, pagination
src/normalize.ts         Pure normalization and salary helpers
src/normalize.test.ts    Unit tests for pure helpers
.actor/actor.json        Apify Actor metadata and Docker build config
.actor/input_schema.json Apify input form/schema
Dockerfile               Apify Playwright Chrome image
INTERVIEW_NOTES.md       Concise explanation for technical interviews

Future improvements

If this needed to become a maintained product, I would add selector contract tests, a source-specific adapter boundary, detail-page extraction behind a configurable request budget, structured run metrics/alerts, and a responsible rate-limit policy. I would only add a database or frontend after a real consumer needed querying or workflow features.