dk-job-scraper

Scrapes job postings from 14 Danish trade-media job boards into one normalised dataset, ready for enrichment and automation.

All 14 sites run the same publishing platform (Nordiske Medier), so the project is one adapter driven by a site registry rather than 14 scrapers. Adding a site that runs the same platform is a one-line change to config/sites.json.

---

Quick start

pip install -r requirements.txt

# What is configured
python3 -m dk_job_scraper sites

# Scrape everything, store it, and write a JSONL file
python3 -m dk_job_scraper -v scrape --all --out data/jobs.jsonl

# Incremental run for cron: only postings never seen before
python3 -m dk_job_scraper -v scrape --all --since 14d --new-only --dedupe \
    --out data/new.jsonl

A full cold run over all 14 sites is about 90 seconds and ~430 requests (409 postings at the time of writing). Later runs stop as soon as they reach postings already in the database, so a scheduled run costs one page per site.

---

Sites

Slug	Publication	Search URL
licitationen	Licitationen	https://www.licitationen.dk/job/search
soefart	Søfart	https://www.soefart.dk/job/search
motormagasinet	Motor-magasinet	https://www.motormagasinet.dk/job/search
metal-supply	Metal Supply	https://www.metal-supply.dk/job/search
building-supply	Building Supply	https://www.building-supply.dk/job/search
transportmagasinet	Transportmagasinet	https://www.transportmagasinet.dk/job/search
lastbilmagasinet	Lastbil Magasinet	https://www.lastbilmagasinet.dk/job/search
food-supply	Food Supply	https://www.food-supply.dk/job/search
medtechnews	MedTechNews	https://www.lifesciencenordic.com/job/search
energy-supply	Energy Supply	https://www.energy-supply.dk/job/search
electronic-supply	Electronic Supply	https://www.electronic-supply.dk/job/search
plastforum	Plastforum	https://www.plastforum.dk/job/search
retailnews	RetailNews	https://www.retailnews.dk/job/search
wood-supply	Wood Supply	https://www.wood-supply.dk/job/search

---

Output schema

One JSON object per posting. uid (<site>:<job id>) is the stable primary key.

Field	Notes
uid, job_id	job_id is the platform-wide id — the same posting cross-published to several sites shares it
source_slug, source_name, source_url	Which board it came from
url	Canonical posting URL
title, company, company_logo
location	Danmark > Midtjylland > Lemvig Kommune
country, region, municipality	Split out of location
postal_code, city, street	From the employer's address microdata
cvr	Danish company registry number — the join key for company-data enrichment
company_id, company_url	Employer's profile on the publishing site
company_website, company_email, company_phone	Employer's own contact details
contact_name, contact_title, contact_email, contact_phone	The named "Kontaktperson" on the posting
published_at	ISO 8601 with timezone
deadline	ISO date, or null
deadline_raw, deadline_open_ended	true for "Løbende ansøgning" / "Snarest muligt"
start_date, start_date_raw	Expected start
positions	Number of openings
employment_type	Ansættelsesforhold, e.g. Fast
work_hours	Arbejdstid, e.g. Fuldtid
employment	Ansættelse, e.g. Permanent
teaser	Short summary from the listing
description_text	Plain text body — what enrichment should read
description_html	Raw HTML body (excluded unless --include-html)
how_to_apply_text / _html	"Sådan ansøger du" block
apply_url	Outbound application link (often an ATS)
also_on	Other sites carrying the same posting (set by --dedupe)
expired	Posting was removed while still linked from the listing
scraped_at, content_hash	Bookkeeping; the hash drives change detection

Field coverage measured over a full run (403 live postings):

title              100%    apply_url           94%    cvr                 24%
company            100%    positions           98%    company_email       24%
url                100%    employment_type     98%    company_phone       23%
published_at       100%    work_hours          98%    company_website     24%
teaser             100%    city / postal_code  90%    contact_name        24%
description_text   100%    region              90%    contact_email       24%
location           100%    municipality        88%    contact_phone       10%
street             100%    deadline            29%    contact_title        9%

deadline looks low because 71% of postings are open-ended ("Løbende ansøgning", "Snarest muligt"); those are flagged with deadline_open_ended, and the two together cover 100% of records.

The employer and contact-person fields come from the sidebar card's schema.org microdata and are only present when the employer has a profile on the platform — about a quarter of postings. Where present, they give you a CVR number and a named person with a direct email and phone, which is the highest-value part of the record for enrichment.

---

Cross-posting

About 36% of postings appear on more than one of these sites (409 records → 261 unique postings in a full run). They share the same job_id, so --dedupe collapses them: the richest copy is kept and the other boards are listed in also_on. Running it before enrichment cuts that step's cost by roughly a third.

$python3 -m dk_job_scraper export --dedupe --out data/unique.jsonl

---

Commands

scrape

Flag	Meaning
--all / --sites a,b	Which boards to scrape
--since 7d	7d, 48h, 2w, 2026-08-01, or an ISO timestamp
--max-pages N	Result pages per site (default 25)
--full	Ignore stored uids and re-scrape everything
--no-details	Listing pages only — fast, but no description or apply URL
-o / --out, --stdout	Where to write
--format jsonl|json|csv	Default jsonl
--include-html	Keep raw HTML bodies
--new-only	Export only postings not seen on a previous run
--dedupe	Collapse cross-posted duplicates
--db PATH	SQLite path (default data/jobs.db)
--no-db	Do not read or write the database
--delay	Seconds between requests to the same host (default 1.0)
--site-workers / --detail-workers	Parallelism (default 4 / 3)
--cache-dir	Cache fetched HTML — makes re-runs and debugging free

export

Re-export what is already in the database without touching the network.

python3 -m dk_job_scraper export --since 30d --format csv --out data/jobs.csv
python3 -m dk_job_scraper export --new-since 24h --dedupe --stdout
python3 -m dk_job_scraper export --sites metal-supply --limit 50 --stdout

stats

$python3 -m dk_job_scraper stats

Per-site counts, the newest posting seen, and the last run's summary.

---

How incremental runs work

The search pages are requested with sort=3 (Nyeste — newest first). Given the set of uids already stored for a site, the scraper walks pages until it hits one where every posting is already known, then stops. --since does the same thing against publication dates. Combined with --new-only, a scheduled run emits a clean delta and nothing else.

--full disables the early stop for a periodic full reconciliation.

---

Design notes

One adapter, not fourteen. Every site is the same CMS: server-rendered HTML, /job/search for listings, /job/view/<id>/<slug> for details, identical field labels on the detail page. dk_job_scraper/parser.py is the whole adapter.

No browser. Everything needed is in the initial HTML response, so there is no Selenium/Playwright dependency and no headless-browser cost.

Politeness. One request per second per host by default, exponential backoff with jitter on 429/5xx, and a real browser user-agent. Sites are scraped in parallel, but the rate limit is per host, so no single site sees a faster rate. robots.txt on these domains allows /job/search and /job/view; the excluded paths (/job/import, management endpoints) are never requested.

Failure is isolated. A dead page, a parse error or a site being down affects only that item: the run continues and the failure is reported at the end. When a detail page 404s — which happens routinely, because postings get pulled while still linked from the listing — the listing-level record is kept and flagged expired rather than being counted as an error.

Selector fallbacks. The listing parser prefers ul.result-list but falls back to any /job/view/ link, so a CSS-class rename in a redesign degrades instead of returning nothing.

---

Deploying to Apify

The repository is a ready-to-push Apify Actor. The scraper package stays runnable on its own — src/ is a thin wrapper that reads the Actor input, runs the same code, and pushes records to the default dataset.

npm install -g apify-cli
apify login
apify push

apify push builds Dockerfile and uploads the Actor. To check the build locally first:

docker build -t dk-job-scraper .
apify run                      # runs the Actor with ./storage as local storage

Input

Everything is optional; the defaults scrape all 14 boards incrementally.

Field	Default	Meaning
sites	all	Which boards to scrape
onlyNewJobs	true	Push only postings not seen on a previous run
since	—	7d, 48h, 2w, 2026-08-01, or an ISO timestamp
dedupeCrossPosted	true	Collapse the same posting published on several boards
fetchDetails	true	Open each posting for description, apply URL, CVR, contact
maxItems	0	Cap the run (0 = no cap)
maxPages	25	Result pages per site
includeHtml	false	Add the raw HTML bodies
requestDelaySecs	1	Delay between requests to the same host
siteConcurrency / detailConcurrency	4 / 3	Parallelism
stateRetentionDays	120	How long a posting stays in the seen-before list
resetState	false	Forget the seen-before list for one run
proxyConfiguration	off	Optional Apify Proxy

Incremental runs on the platform

The default key-value store is wiped between runs, so the set of already-seen uids lives in a named store, dk-job-scraper-state, which persists. A scheduled run therefore costs one page per site until something new is posted:

first run    409 postings   434 requests   ~90s
later runs     0-20 new      14 requests    ~6s

State is bounded by stateRetentionDays — entries first seen longer ago than that are dropped, since those postings are long gone from the boards.

Set resetState: true for a one-off full re-scrape without losing the store.

Suggested schedule

Given the posting volume (roughly 35–45 new postings a week across all boards), once or twice a day is plenty. In the Actor's Schedules tab:

0 6 * * *      # 06:00 UTC daily

Wiring it into the rest of the pipeline

Add a webhook on the Actor's RUN.SUCCEEDED event pointing at your automation. The payload carries resource.defaultDatasetId, and the new postings from that run — and only those — are the dataset's contents:

https://api.apify.com/v2/datasets/{datasetId}/items?format=json&clean=true

Or pull the last run's items directly:

https://api.apify.com/v2/acts/{username}~dk-job-scraper/runs/last/dataset/items?token={token}

The dataset ships two table views: Overview (title, company, location, dates, source) and Employer & contacts (CVR, contact person, emails, phones), so the run is readable in the Apify UI without exporting.

Notes

• Actor memory: 1 GB is enough — the scraper is I/O bound, not memory bound.
• A full run is ~90 seconds of mostly waiting, so compute-unit cost is low.
• No proxy is needed under normal use; the input accepts one if you ever want the traffic routed.
• onlyNewJobs: false disables the state entirely and returns the full board each run, if you would rather deduplicate downstream.

---

Tests

pip install -r requirements-dev.txt
python3 -m pytest

67 tests, no network required — parsing is verified against saved HTML fixtures in tests/fixtures/ and the scraper is driven by a fake HTTP client.

---

Project layout

.actor/                    Apify Actor manifest, input schema, dataset views
    actor.json
    input_schema.json
    dataset_schema.json
Dockerfile                 Apify build
src/                       Apify Actor entry point
    main.py                reads input, runs the scrape, pushes to the dataset
    state.py               cross-run "already seen" state
config/sites.json          site registry (add a site here)
dk_job_scraper/
    sites.py               registry loading and selection
    http.py                retries, backoff, per-host rate limit, cache
    parser.py              the Nordiske Medier adapter
    models.py              the normalised Job record
    scraper.py             pagination, incremental stop, detail fetching
    store.py               SQLite dedupe + change detection
    exporters.py           JSONL / JSON / CSV, cross-site dedupe
    cli.py                 command line interface
examples/
    daily_scrape.sh        cron-ready incremental run
    use_as_library.py      driving the scraper from Python
    enrich_new_jobs.py     consuming the delta downstream
tests/                     fixtures and test suite

---

Maintenance

If a site changes its markup, tests/test_parser.py fails against the fixture and points at the field that broke. To refresh a fixture:

curl -A "Mozilla/5.0" https://www.building-supply.dk/job/search \
    -o tests/fixtures/listing_building-supply.html

To add a site on the same platform, append it to config/sites.json — no code change. A site on a different platform needs a new parser module following the parse_listing / parse_detail / build_job interface in parser.py.