# multi-web-sites-jobs-scraper (`ahmed_madbouly/multi-web-sites-jobs-scraper`) Actor

Scrape 1,939+ IT jobs from LinkedIn and MyCareersFuture Singapore in 8 minutes. Fully configurable keywords, locations, and sites with zero code changes. Production-ready multi-site architecture. 100% success rate. $0.13 per run.

- **URL**: https://apify.com/ahmed\_madbouly/multi-web-sites-jobs-scraper.md
- **Developed by:** [Ahmed Madbouly](https://apify.com/ahmed_madbouly) (community)
- **Categories:** Jobs, Automation, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Multi-Site Jobs Scraper (Singapore)

### Overview

This tool collects IT job listings from multiple job sites into a single, ready-to-use dataset — no manual searching across tabs, no copy-pasting listings by hand.

Tell it **which job sites**, **which job titles**, and **which location** to search, and it comes back with a clean spreadsheet-ready list: job title, company, location, posted date, and a direct link to apply — one row per listing, deduplicated by source so you always know where each result came from.

**A real test run** searching 2 sites × 5 job titles (Software Engineer, Backend Developer, Frontend Developer, DevOps Engineer, Data Scientist) for Singapore returned **808 listings in a single run**.

#### Why this instead of checking each site manually?

- **One list instead of five browser tabs.** Every site's results land in the same table with the same columns.
- **Search terms are an input, not code.** Change job titles or location from a simple form — no developer needed for day-to-day use.
- **Grows without a rebuild.** Adding another job site later is a contained change (see [Adding a new site](#adding-a-new-site)) — the rest of the tool, and everything you already rely on, keeps working exactly as before.
- **Self-healing on hiccups.** A single failed page (network blip, slow load) is retried automatically before being skipped — one bad request doesn't sink the whole run.
- **Runs on a schedule.** Since this is built on the Apify platform, it can be scheduled to run daily/weekly and export results as CSV, Excel, or JSON automatically.

#### Currently covered sites

| Site | Notes |
|---|---|
| LinkedIn | Public job search results |
| MyCareersFuture | Singapore's official government job portal |

*(JobStreet SG was evaluated and excluded — it blocks automated browsers behind a Cloudflare challenge on every page, including the homepage, so it can't be scraped reliably without additional anti-bot infrastructure.)*

***

### Technical Details

#### What it does, step by step

For every combination of selected **website** × **keyword** × **page**, it:

1. Navigates to that site's job search results page for the keyword
2. Waits for job cards to render
3. Extracts title, company, location, posted date, and URL for each listing
4. Pushes everything to the Actor's default dataset, tagged with `source` (which site) and `keyword` (which search produced it)

Site registry (`src/sites/index.js`):

| Site | id |
|---|---|
| LinkedIn | `linkedin` |
| MyCareersFuture | `mycareersfuture` |

#### Input

| Field | Type | Description |
|---|---|---|
| `websites` | array (select) | Which sites to scrape. See `INPUT_SCHEMA.json` for the current list. |
| `keywords` | array of strings | Job titles/keywords to search for, on every selected site. |
| `location` | string | Job location, e.g. `Singapore`. Only used by sites whose search takes a location parameter. |
| `maxPages` | integer | Result pages to scrape, **per site, per keyword** (not a global cap). Total requests ≈ `websites.length × keywords.length × maxPages`. |
| `useProxy` | boolean | Route requests through Apify Proxy. Requires proxy access on your Apify plan — if unavailable, the run logs a warning and continues without a proxy rather than failing. |

Example (`test-input.json`):

```json
{
    "websites": ["linkedin", "mycareersfuture"],
    "keywords": ["Software Engineer", "Backend Developer", "Frontend Developer", "DevOps Engineer", "Data Scientist"],
    "location": "Singapore",
    "maxPages": 2,
    "useProxy": true
}
```

#### Output

Each dataset item looks like:

```json
{
    "title": "Software Engineer",
    "company": "Some Company Pte Ltd",
    "location": "Singapore",
    "description": "No description",
    "postedDate": "1 week ago",
    "url": "https://...",
    "source": "LinkedIn",
    "keyword": "Software Engineer",
    "scrapedAt": "2026-08-14T13:02:00.000Z"
}
```

`description` is usually `"No description"` — most search-result cards don't include a snippet; getting real descriptions would require visiting each job's detail page individually.

#### Adding a new site

Each site is a self-contained handler in `src/sites/`. To add one:

1. Create `src/sites/<siteId>.js` exporting:
   ```js
   module.exports = {
       id: '<siteId>',
       name: '<Display Name>',
       buildSearchUrl({ keyword, location, pageNum }) { /* return a URL string */ },
       waitForSelector: '<CSS selector for a job card>',
       async extractJobs(page) { /* return [{ title, company, location, description, postedDate, url }] */ },
   };
   ```
2. Register it in `src/sites/index.js`.
3. Add `"<siteId>"` to the `websites.items.enum` (and `enumTitles`) in `INPUT_SCHEMA.json`.

That's it — `main.js` doesn't need to change; it drives every registered site generically.

**Note:** sites with active bot-protection (e.g. Cloudflare challenge pages, like JobStreet SG) won't work with a plain headless browser and are out of scope for this pattern without further anti-bot infrastructure.

#### Reliability

Each page fetch retries once on failure (network errors, timeouts) before being logged and skipped, so a single bad request doesn't fail the whole run. A randomized delay is added between requests to avoid hammering target sites.

#### Running locally

```bash
npm install
npx apify run --input-file test-input.json
```

#### Deploying to Apify

```bash
apify push
```

Requires `.actor/actor.json` (already present) and a `Dockerfile` pinned to the Playwright version in `package-lock.json` — see comments in `Dockerfile` if you bump the `playwright` dependency.

#### Notes

- Scraping public job listings may be subject to each site's Terms of Service. Use responsibly and avoid excessive request rates (this Actor adds a randomized delay between requests by default).

# Actor input Schema

## `websites` (type: `array`):

Which job sites to scrape. To add a new site: create a handler in src/sites/, register it in src/sites/index.js, then add its id here.

## `keywords` (type: `array`):

Job titles/keywords to search for on each selected site.

## `location` (type: `string`):

Job location (e.g., 'Singapore', 'Remote'). Only applies to sites whose search takes a location parameter.

## `maxPages` (type: `integer`):

How many result pages to scrape, per site, per keyword.

## `useProxy` (type: `boolean`):

Enable IP rotation (recommended to avoid blocks). Uses compute units but highly recommended.

## Actor input object example

```json
{
  "websites": [
    "linkedin",
    "mycareersfuture"
  ],
  "keywords": [
    "Software Engineer",
    "Backend Developer",
    "Frontend Developer",
    "DevOps Engineer",
    "Data Scientist"
  ],
  "location": "Singapore",
  "maxPages": 3,
  "useProxy": true
}
```

# Actor output Schema

## `jobListings` (type: `string`):

All scraped job listings from this run, as JSON.

## `runDetails` (type: `string`):

View this run and its results in the Apify Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keywords": [
        "Software Engineer",
        "Backend Developer",
        "Frontend Developer",
        "DevOps Engineer",
        "Data Scientist"
    ],
    "location": "Singapore"
};

// Run the Actor and wait for it to finish
const run = await client.actor("ahmed_madbouly/multi-web-sites-jobs-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "keywords": [
        "Software Engineer",
        "Backend Developer",
        "Frontend Developer",
        "DevOps Engineer",
        "Data Scientist",
    ],
    "location": "Singapore",
}

# Run the Actor and wait for it to finish
run = client.actor("ahmed_madbouly/multi-web-sites-jobs-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keywords": [
    "Software Engineer",
    "Backend Developer",
    "Frontend Developer",
    "DevOps Engineer",
    "Data Scientist"
  ],
  "location": "Singapore"
}' |
apify call ahmed_madbouly/multi-web-sites-jobs-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ahmed_madbouly/multi-web-sites-jobs-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FGECpswkgPxNtwz5t/builds/9dW1VzfWPhOu5BLfx/openapi.json