# Danish Trade Media Job Scraper (`centox/dk-job-scraper`) Actor

Scrapes job postings from 14 Danish trade-media job boards into one normalised dataset, with CVR numbers and named contact persons where published.

- **URL**: https://apify.com/centox/dk-job-scraper.md
- **Developed by:** [Centox ApS](https://apify.com/centox) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.01 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## dk-job-scraper

Scrapes job postings from 14 Danish trade-media job boards into one normalised
dataset, ready for enrichment and automation.

All 14 sites run the same publishing platform (Nordiske Medier), so the project
is **one adapter driven by a site registry** rather than 14 scrapers. Adding a
site that runs the same platform is a one-line change to `config/sites.json`.

***

### Quick start

```bash
pip install -r requirements.txt

## What is configured
python3 -m dk_job_scraper sites

## Scrape everything, store it, and write a JSONL file
python3 -m dk_job_scraper -v scrape --all --out data/jobs.jsonl

## Incremental run for cron: only postings never seen before
python3 -m dk_job_scraper -v scrape --all --since 14d --new-only --dedupe \
    --out data/new.jsonl
```

A full cold run over all 14 sites is about **90 seconds and ~430 requests**
(409 postings at the time of writing). Later runs stop as soon as they reach
postings already in the database, so a scheduled run costs one page per site.

***

### Sites

| Slug | Publication | Search URL |
|---|---|---|
| `licitationen` | Licitationen | https://www.licitationen.dk/job/search |
| `soefart` | Søfart | https://www.soefart.dk/job/search |
| `motormagasinet` | Motor-magasinet | https://www.motormagasinet.dk/job/search |
| `metal-supply` | Metal Supply | https://www.metal-supply.dk/job/search |
| `building-supply` | Building Supply | https://www.building-supply.dk/job/search |
| `transportmagasinet` | Transportmagasinet | https://www.transportmagasinet.dk/job/search |
| `lastbilmagasinet` | Lastbil Magasinet | https://www.lastbilmagasinet.dk/job/search |
| `food-supply` | Food Supply | https://www.food-supply.dk/job/search |
| `medtechnews` | MedTechNews | https://www.lifesciencenordic.com/job/search |
| `energy-supply` | Energy Supply | https://www.energy-supply.dk/job/search |
| `electronic-supply` | Electronic Supply | https://www.electronic-supply.dk/job/search |
| `plastforum` | Plastforum | https://www.plastforum.dk/job/search |
| `retailnews` | RetailNews | https://www.retailnews.dk/job/search |
| `wood-supply` | Wood Supply | https://www.wood-supply.dk/job/search |

***

### Output schema

One JSON object per posting. `uid` (`<site>:<job id>`) is the stable primary key.

| Field | Notes |
|---|---|
| `uid`, `job_id` | `job_id` is the platform-wide id — the same posting cross-published to several sites shares it |
| `source_slug`, `source_name`, `source_url` | Which board it came from |
| `url` | Canonical posting URL |
| `title`, `company`, `company_logo` | |
| `location` | `Danmark > Midtjylland > Lemvig Kommune` |
| `country`, `region`, `municipality` | Split out of `location` |
| `postal_code`, `city`, `street` | From the employer's address microdata |
| `cvr` | Danish company registry number — the join key for company-data enrichment |
| `company_id`, `company_url` | Employer's profile on the publishing site |
| `company_website`, `company_email`, `company_phone` | Employer's own contact details |
| `contact_name`, `contact_title`, `contact_email`, `contact_phone` | The named "Kontaktperson" on the posting |
| `published_at` | ISO 8601 with timezone |
| `deadline` | ISO date, or `null` |
| `deadline_raw`, `deadline_open_ended` | `true` for "Løbende ansøgning" / "Snarest muligt" |
| `start_date`, `start_date_raw` | Expected start |
| `positions` | Number of openings |
| `employment_type` | Ansættelsesforhold, e.g. `Fast` |
| `work_hours` | Arbejdstid, e.g. `Fuldtid` |
| `employment` | Ansættelse, e.g. `Permanent` |
| `teaser` | Short summary from the listing |
| `description_text` | Plain text body — what enrichment should read |
| `description_html` | Raw HTML body (excluded unless `--include-html`) |
| `how_to_apply_text` / `_html` | "Sådan ansøger du" block |
| `apply_url` | Outbound application link (often an ATS) |
| `also_on` | Other sites carrying the same posting (set by `--dedupe`) |
| `expired` | Posting was removed while still linked from the listing |
| `scraped_at`, `content_hash` | Bookkeeping; the hash drives change detection |

Field coverage measured over a full run (403 live postings):

```
title              100%    apply_url           94%    cvr                 24%
company            100%    positions           98%    company_email       24%
url                100%    employment_type     98%    company_phone       23%
published_at       100%    work_hours          98%    company_website     24%
teaser             100%    city / postal_code  90%    contact_name        24%
description_text   100%    region              90%    contact_email       24%
location           100%    municipality        88%    contact_phone       10%
street             100%    deadline            29%    contact_title        9%
```

`deadline` looks low because 71% of postings are open-ended ("Løbende ansøgning",
"Snarest muligt"); those are flagged with `deadline_open_ended`, and the two
together cover 100% of records.

The employer and contact-person fields come from the sidebar card's schema.org
microdata and are only present when the employer has a profile on the platform
— about a quarter of postings. Where present, they give you a CVR number and a
named person with a direct email and phone, which is the highest-value part of
the record for enrichment.

***

### Cross-posting

About **36% of postings appear on more than one of these sites** (409 records →
261 unique postings in a full run). They share the same `job_id`, so `--dedupe`
collapses them: the richest copy is kept and the other boards are listed in
`also_on`. Running it before enrichment cuts that step's cost by roughly a third.

```bash
python3 -m dk_job_scraper export --dedupe --out data/unique.jsonl
```

***

### Commands

#### `scrape`

| Flag | Meaning |
|---|---|
| `--all` / `--sites a,b` | Which boards to scrape |
| `--since 7d` | `7d`, `48h`, `2w`, `2026-08-01`, or an ISO timestamp |
| `--max-pages N` | Result pages per site (default 25) |
| `--full` | Ignore stored uids and re-scrape everything |
| `--no-details` | Listing pages only — fast, but no description or apply URL |
| `-o` / `--out`, `--stdout` | Where to write |
| `--format jsonl\|json\|csv` | Default `jsonl` |
| `--include-html` | Keep raw HTML bodies |
| `--new-only` | Export only postings not seen on a previous run |
| `--dedupe` | Collapse cross-posted duplicates |
| `--db PATH` | SQLite path (default `data/jobs.db`) |
| `--no-db` | Do not read or write the database |
| `--delay` | Seconds between requests **to the same host** (default 1.0) |
| `--site-workers` / `--detail-workers` | Parallelism (default 4 / 3) |
| `--cache-dir` | Cache fetched HTML — makes re-runs and debugging free |

#### `export`

Re-export what is already in the database without touching the network.

```bash
python3 -m dk_job_scraper export --since 30d --format csv --out data/jobs.csv
python3 -m dk_job_scraper export --new-since 24h --dedupe --stdout
python3 -m dk_job_scraper export --sites metal-supply --limit 50 --stdout
```

#### `stats`

```bash
python3 -m dk_job_scraper stats
```

Per-site counts, the newest posting seen, and the last run's summary.

***

### How incremental runs work

The search pages are requested with `sort=3` (Nyeste — newest first). Given the
set of uids already stored for a site, the scraper walks pages until it hits one
where every posting is already known, then stops. `--since` does the same thing
against publication dates. Combined with `--new-only`, a scheduled run emits a
clean delta and nothing else.

`--full` disables the early stop for a periodic full reconciliation.

***

### Design notes

**One adapter, not fourteen.** Every site is the same CMS: server-rendered HTML,
`/job/search` for listings, `/job/view/<id>/<slug>` for details, identical field
labels on the detail page. `dk_job_scraper/parser.py` is the whole adapter.

**No browser.** Everything needed is in the initial HTML response, so there is
no Selenium/Playwright dependency and no headless-browser cost.

**Politeness.** One request per second per host by default, exponential backoff
with jitter on 429/5xx, and a real browser user-agent. Sites are scraped in
parallel, but the rate limit is per host, so no single site sees a faster rate.
`robots.txt` on these domains allows `/job/search` and `/job/view`; the excluded
paths (`/job/import`, management endpoints) are never requested.

**Failure is isolated.** A dead page, a parse error or a site being down affects
only that item: the run continues and the failure is reported at the end. When a
detail page 404s — which happens routinely, because postings get pulled while
still linked from the listing — the listing-level record is kept and flagged
`expired` rather than being counted as an error.

**Selector fallbacks.** The listing parser prefers `ul.result-list` but falls
back to any `/job/view/` link, so a CSS-class rename in a redesign degrades
instead of returning nothing.

***

### Deploying to Apify

The repository is a ready-to-push Apify Actor. The scraper package stays
runnable on its own — `src/` is a thin wrapper that reads the Actor input, runs
the same code, and pushes records to the default dataset.

```bash
npm install -g apify-cli
apify login
apify push
```

`apify push` builds [Dockerfile](Dockerfile) and uploads the Actor. To check the
build locally first:

```bash
docker build -t dk-job-scraper .
apify run                      # runs the Actor with ./storage as local storage
```

#### Input

Everything is optional; the defaults scrape all 14 boards incrementally.

| Field | Default | Meaning |
|---|---|---|
| `sites` | all | Which boards to scrape |
| `onlyNewJobs` | `true` | Push only postings not seen on a previous run |
| `since` | — | `7d`, `48h`, `2w`, `2026-08-01`, or an ISO timestamp |
| `dedupeCrossPosted` | `true` | Collapse the same posting published on several boards |
| `fetchDetails` | `true` | Open each posting for description, apply URL, CVR, contact |
| `maxItems` | `0` | Cap the run (0 = no cap) |
| `maxPages` | `25` | Result pages per site |
| `includeHtml` | `false` | Add the raw HTML bodies |
| `requestDelaySecs` | `1` | Delay between requests to the same host |
| `siteConcurrency` / `detailConcurrency` | `4` / `3` | Parallelism |
| `stateRetentionDays` | `120` | How long a posting stays in the seen-before list |
| `resetState` | `false` | Forget the seen-before list for one run |
| `proxyConfiguration` | off | Optional Apify Proxy |

#### Incremental runs on the platform

The default key-value store is wiped between runs, so the set of already-seen
uids lives in a **named** store, `dk-job-scraper-state`, which persists. A
scheduled run therefore costs one page per site until something new is posted:

```
first run    409 postings   434 requests   ~90s
later runs     0-20 new      14 requests    ~6s
```

State is bounded by `stateRetentionDays` — entries first seen longer ago than
that are dropped, since those postings are long gone from the boards.

Set `resetState: true` for a one-off full re-scrape without losing the store.

#### Suggested schedule

Given the posting volume (roughly 35–45 new postings a week across all boards),
**once or twice a day is plenty**. In the Actor's *Schedules* tab:

```
0 6 * * *      # 06:00 UTC daily
```

#### Wiring it into the rest of the pipeline

Add a webhook on the Actor's `RUN.SUCCEEDED` event pointing at your automation.
The payload carries `resource.defaultDatasetId`, and the new postings from that
run — and only those — are the dataset's contents:

```
https://api.apify.com/v2/datasets/{datasetId}/items?format=json&clean=true
```

Or pull the last run's items directly:

```
https://api.apify.com/v2/acts/{username}~dk-job-scraper/runs/last/dataset/items?token={token}
```

The dataset ships two table views: **Overview** (title, company, location,
dates, source) and **Employer & contacts** (CVR, contact person, emails,
phones), so the run is readable in the Apify UI without exporting.

#### Notes

- Actor memory: 1 GB is enough — the scraper is I/O bound, not memory bound.
- A full run is ~90 seconds of mostly waiting, so compute-unit cost is low.
- No proxy is needed under normal use; the input accepts one if you ever want
  the traffic routed.
- `onlyNewJobs: false` disables the state entirely and returns the full board
  each run, if you would rather deduplicate downstream.

***

### Tests

```bash
pip install -r requirements-dev.txt
python3 -m pytest
```

67 tests, no network required — parsing is verified against saved HTML fixtures
in `tests/fixtures/` and the scraper is driven by a fake HTTP client.

***

### Project layout

```
.actor/                    Apify Actor manifest, input schema, dataset views
    actor.json
    input_schema.json
    dataset_schema.json
Dockerfile                 Apify build
src/                       Apify Actor entry point
    main.py                reads input, runs the scrape, pushes to the dataset
    state.py               cross-run "already seen" state
config/sites.json          site registry (add a site here)
dk_job_scraper/
    sites.py               registry loading and selection
    http.py                retries, backoff, per-host rate limit, cache
    parser.py              the Nordiske Medier adapter
    models.py              the normalised Job record
    scraper.py             pagination, incremental stop, detail fetching
    store.py               SQLite dedupe + change detection
    exporters.py           JSONL / JSON / CSV, cross-site dedupe
    cli.py                 command line interface
examples/
    daily_scrape.sh        cron-ready incremental run
    use_as_library.py      driving the scraper from Python
    enrich_new_jobs.py     consuming the delta downstream
tests/                     fixtures and test suite
```

***

### Maintenance

If a site changes its markup, `tests/test_parser.py` fails against the fixture
and points at the field that broke. To refresh a fixture:

```bash
curl -A "Mozilla/5.0" https://www.building-supply.dk/job/search \
    -o tests/fixtures/listing_building-supply.html
```

To add a site on the same platform, append it to `config/sites.json` — no code
change. A site on a different platform needs a new parser module following the
`parse_listing` / `parse_detail` / `build_job` interface in `parser.py`.

# Actor input Schema

## `sites` (type: `array`):

Which job boards to scrape. Leave empty for all of them.

## `onlyNewJobs` (type: `boolean`):

Push only postings that were not seen on a previous run. This also stops each site's pagination as soon as it reaches known postings, so scheduled runs are cheap. Turn off to get the full board every time.

## `since` (type: `string`):

Ignore postings older than this. Accepts 7d, 48h, 2w, a date like 2026-08-01, or an ISO timestamp. Leave empty for no limit.

## `dedupeCrossPosted` (type: `boolean`):

About a third of postings appear on more than one of these boards. When on, each posting is pushed once, with the other boards listed in the also\_on field.

## `fetchDetails` (type: `boolean`):

Open each posting to get the full description, application link, CVR number and contact person. Turning this off is much faster but returns listing data only.

## `maxItems` (type: `integer`):

Stop after this many postings. 0 means no limit.

## `maxPages` (type: `integer`):

Each page holds 24 postings.

## `includeHtml` (type: `boolean`):

Add description\_html and how\_to\_apply\_html to each record. Roughly triples the dataset size; the plain-text fields are usually what you want.

## `requestDelaySecs` (type: `integer`):

Seconds to wait between requests to the same site. Lowering this scrapes faster but is less polite; 1 second is recommended.

## `siteConcurrency` (type: `integer`):

Different sites are different hosts, so this does not increase the request rate against any single one.

## `detailConcurrency` (type: `integer`):

How many posting pages to open at once per site. Bounded by the delay above.

## `stateRetentionDays` (type: `integer`):

How long a posting stays in the seen-before list. Older entries are dropped to keep the state small.

## `resetState` (type: `boolean`):

Forget every previously seen posting for this run, so the whole board is treated as new. Use for a one-off full re-scrape.

## `proxyConfiguration` (type: `object`):

Optional. These sites are scraped politely and do not normally require a proxy.

## Actor input object example

```json
{
  "sites": [],
  "onlyNewJobs": true,
  "since": "14d",
  "dedupeCrossPosted": true,
  "fetchDetails": true,
  "maxItems": 0,
  "maxPages": 25,
  "includeHtml": false,
  "requestDelaySecs": 1,
  "siteConcurrency": 4,
  "detailConcurrency": 3,
  "stateRetentionDays": 120,
  "resetState": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `jobs` (type: `string`):

Normaliserede jobopslag fra de 14 fagmedier.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("centox/dk-job-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("centox/dk-job-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call centox/dk-job-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,centox/dk-job-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4jRjDacfbdPPIrkGU/builds/gbpyWMlw3ak6wMltp/openapi.json