# Job Listing Scraper (`winniekimani/job-listing-apify-actor`) Actor

Collects and normalizes publicly available job listings from multiple sources, removes duplicates, and stores structured results in an Apify Dataset.

- **URL**: https://apify.com/winniekimani/job-listing-apify-actor.md
- **Developed by:** [winnie kimani](https://apify.com/winniekimani) (community)
- **Categories:** Lead generation, Jobs
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Job Listing Scraper

This repository contains a small Apify Actor that collects public job listings from multiple fallback sources, normalizes the fields, removes duplicate URLs, and writes the results to an Apify Dataset.

It was built as a focused portfolio project to demonstrate TypeScript, Node.js, Crawlee, Playwright, Apify Actor lifecycle/storage, pagination, validation, retries, and professional delivery without adding infrastructure the scraper does not need.

### Features

- Structured Actor input with configurable start URLs and result limit.
- Playwright-based browser crawling managed by Crawlee.
- Public HTML/API extraction for title, company, location, employment type, posted label, summary, URL, and visible salary.
- Three default sources: Arbeitnow HTML, Remotive's public remote-jobs API, and RemoteOK's public API. A failed source is logged while the other sources can continue.
- Whitespace cleanup, URL normalization, optional-field handling, and a 1,000-character summary cap.
- URL-based deduplication and a maximum of 500 results per run.
- Pagination through Arbeitnow's public `Next` link.
- Crawlee retries failed requests twice and reports requests that still fail.
- Apify Dataset output locally and in the Apify platform.
- Unit tests for normalization plus GitHub Actions CI.

### Technology stack

- TypeScript and Node.js
- Apify SDK for Actor lifecycle, input, and Dataset storage
- Crawlee `PlaywrightCrawler` for the request queue, retries, concurrency, and crawl loop
- Playwright for browser navigation and DOM extraction
- Arbeitnow public job-board HTML as the data source
- Docker using Apify's Playwright Chrome base image

### Architecture

```text
User Input
    ↓
Apify Actor
    ↓
Apify SDK (lifecycle, input, Dataset)
    ↓
Crawlee PlaywrightCrawler (queue, retry, concurrency)
    ↓
Playwright browser
    ↓
Public Arbeitnow job listing website
    ↓
Extraction / normalization / deduplication
    ↓
Apify Dataset
```

Playwright is used because the Actor reads the rendered page through a real browser and stable semantic DOM attributes. Crawlee is used instead of a manual browser loop because it provides request queues, retry handling, concurrency control, and crawl statistics around Playwright.

### Input

```json
{
  "startUrls": [
    { "url": "https://www.arbeitnow.com/" }
  ],
  "maxResults": 25
}
```

`startUrls` accepts public HTTP(S) listing pages or APIs. By default the Actor tries Arbeitnow, Remotive, and RemoteOK. `maxResults` is clamped to 1–500.

### Output

Each Dataset item has this shape. Missing values are `null`; the Actor does not invent data.

```json
{
  "title": "Example job title",
  "company": "Example company",
  "location": "Berlin",
  "employmentType": "Vollzeit",
  "datePosted": "Posted 5 hours ago",
  "summary": "A cleaned summary taken from the public listing card...",
  "url": "https://www.arbeitnow.com/jobs/companies/example/example-job-123",
  "salary": null,
  "source": "arbeitnow.com"
}
```

### Run locally

```bash
npm install
npm run build
npm test
npm start
```

For a small real run, provide local Apify input through the environment:

```bash
APIFY_INPUT_JSON='{"maxResults":5}' npm start
```

The Apify SDK writes local Dataset output under `storage/`, which is ignored by Git. The local browser launch uses `/usr/bin/google-chrome` by default; set `BROWSER_EXECUTABLE_PATH` if your Chrome binary is elsewhere.

### Run on Apify

After authenticating the Apify CLI, deploy from this directory:

```bash
apify login
apify push
apify call --input-file=example-input.json
```

The Actor input form is defined in `.actor/input_schema.json`. The Docker image includes the Playwright browser runtime in Apify's cloud environment.

### Error handling and limitations

Crawlee retries a failed request twice. A request that still fails is logged by `failedRequestHandler`, while successful pages continue. The Actor intentionally uses only public pages; it does not bypass CAPTCHA, authentication, paywalls, Cloudflare challenges, robots restrictions, or rate limits. The extraction depends on Arbeitnow's current HTML attributes, so a site redesign can require selector updates. The public listing card does not always expose a full description or salary, so those fields can be `null`.

### Testing

`npm run build` runs strict TypeScript checking. `npm test` runs four Node test-runner tests covering whitespace cleanup, URL normalization, salary parsing, and optional-field normalization. A GitHub Actions workflow runs both commands on pushes and pull requests.

### Project structure

```text
src/main.ts              Actor lifecycle, crawler, extraction, pagination
src/normalize.ts         Pure normalization and salary helpers
src/normalize.test.ts    Unit tests for pure helpers
.actor/actor.json        Apify Actor metadata and Docker build config
.actor/input_schema.json Apify input form/schema
Dockerfile               Apify Playwright Chrome image
INTERVIEW_NOTES.md       Concise explanation for technical interviews
```

### Future improvements

If this needed to become a maintained product, I would add selector contract tests, a source-specific adapter boundary, detail-page extraction behind a configurable request budget, structured run metrics/alerts, and a responsible rate-limit policy. I would only add a database or frontend after a real consumer needed querying or workflow features.

# Actor input Schema

## `startUrls` (type: `array`):

Public job-board pages or APIs to crawl. By default the Actor tries Arbeitnow, Remotive, and RemoteOK so one unavailable source does not stop the run.

## `maxResults` (type: `integer`):

Maximum number of unique job listings to save (1–500).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.arbeitnow.com/"
    },
    {
      "url": "https://remotive.com/api/remote-jobs"
    },
    {
      "url": "https://remoteok.com/api"
    }
  ],
  "maxResults": 25
}
```

# Actor output Schema

## `results` (type: `string`):

The structured job listings saved in the run's default Dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.arbeitnow.com/"
        },
        {
            "url": "https://remotive.com/api/remote-jobs"
        },
        {
            "url": "https://remoteok.com/api"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("winniekimani/job-listing-apify-actor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        { "url": "https://www.arbeitnow.com/" },
        { "url": "https://remotive.com/api/remote-jobs" },
        { "url": "https://remoteok.com/api" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("winniekimani/job-listing-apify-actor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.arbeitnow.com/"
    },
    {
      "url": "https://remotive.com/api/remote-jobs"
    },
    {
      "url": "https://remoteok.com/api"
    }
  ]
}' |
apify call winniekimani/job-listing-apify-actor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,winniekimani/job-listing-apify-actor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pD2qk1h4TQOUdJLPJ/builds/aYff46qzV5GiRiKqj/openapi.json