GovData Germany Scraper - CKAN Open Data Portal

GovData Germany Scraper - CKAN Open Data Portal

Scrape GovData.de, Germany's national open data portal. 156,461 Datensätze with the full DCAT-AP.de record — publisher, contact point, EU data theme, keywords, licence, temporal and spatial coverage, update frequency, High-Value Dataset flag — plus every distribution, and optional checks that tell you whether the file behind the link is actually there. Eight datasets behind one input. No login, no API key. Export to CSV, JSON, Excel or XML.

GovData is the single index of German public-sector open data: federal ministries, all sixteen Länder, the big city portals and the geodata infrastructure all harvest into it. The portal has a CKAN API, but it hands you raw CKAN — the DCAT-AP.de payload arrives as a flat key/value list with JSON arrays serialised into strings, the format field holds 403 different spellings of about thirty formats, and nothing tells you whether the download still works. This Actor turns that into typed rows.

Who uses it	What they scrape GovData for
Data engineers and analysts	A queryable index of every German public dataset, filtered to the formats they can actually ingest
GovTech and civic-tech teams	Monitoring what each Land and Kommune publishes, and when it changes
Researchers and journalists	Finding the Statistik or Geodaten behind a story, with the publisher's contact details attached
Compliance and policy teams	The 10,514 EU High-Value Datasets and their legal basis, by category
Data catalogue vendors	DCAT-AP.de metadata to seed or reconcile their own catalogue
Lead generation for public sector	175 distinct publishing bodies per 1,400 datasets, with email, phone and postal address

What it does

Pick a dataset, set filters, get rows. Every row carries the GovData page URL and a scrape timestamp.

• 🗂️ Datasets — one row per Datensatz, 84 fields: title, description, publisher, contact point with postal address, EU data theme, keywords, licences, issued and modified dates, temporal coverage, bounding box and centroid, update frequency, administrative level, regional keys (AGS), High-Value Dataset flag and legal basis, source portal, and every distribution nested with its own metadata.
• 📦 Distributions — one row per file or service, 42 fields: URL, normalised format, MIME type, byte size, checksum, licence and attribution text, dates — carrying the identity, publisher, themes and keywords of the dataset it belongs to. Datasets average 4.0 distributions each.
• 📇 Contact points — the publishing bodies deduplicated out of whatever you scanned: name, email, phone, street, postcode, city, how many datasets they are responsible for and which themes those cover.
• 🏛️ Organizations — all 76 publishing organizations with dataset counts, both catalogue-wide and within your filters.
• 🏷️ Categories — the 13 EU data themes with German and English titles, their data-theme URIs and counts.
• 🔖 Tags — the keyword vocabulary (97,863 tags) ranked by dataset count, honouring your filters.
• 📄 Formats — the distribution format directory, grouped onto a normalised name with every raw spelling the catalogue uses underneath it.
• ⚖️ Licences — the 52 registered licences with Open Definition and Open Source conformance.

What is in the catalogue

Measured 2026-08-27 against the live index:

	Datasets
Total	156,461
With at least one CSV distribution	68,582
With a geometry (bounding box + centroid on the row)	72,621
With a temporal coverage window	68,163
With a machine-readable contact email	93,876
EU High-Value Datasets	10,514
A WMS, WFS, WCS or WMTS service	14,559
JSON or GeoJSON	13,675
Metadata changed in the last 7 days	50,773
First published in the last 30 days	6,894

Largest publishers: GDI-DE (27,596), Land Schleswig-Holstein (27,546), open.bydata / Bayern (26,634), Open Data Brandenburg (19,445), Transparenzportal Hamburg (13,965), Open.NRW (10,407), Land Rheinland-Pfalz (9,221).

What you can do with GovData data

🔎 Find the datasets nobody can find on the portal.

ckan is a 6,600-a-month search and open data portal 1,000 at a $14.37 CPC — people are looking for exactly this index. GovData's own search is one box over 156,461 records. Here you can ask for datasets in the Umwelt theme, published by a Land, updated in the last month, that ship a CSV, with a contact email and get 84 typed columns back.

📊 Build a column index of German open data.

datensatz runs 1,000 searches a month in Germany at a $13.40 CPC, and the reason is that finding the right one is hard. Switch on Preview distribution columns and every CSV, TSV, JSON, GeoJSON or XML distribution comes back with its real column names, delimiter, character encoding and a sample row — read from the first 64 KB with a range request, so a 600 MB file costs 64 KB. Now "which datasets have a gemeindeschluessel column" is a query, not an afternoon.

🔗 Find the dead links before your pipeline does.

GovData harvests metadata from regional portals and never re-verifies it. Measured across 150 distributions spread over the full age range of the catalogue, 5.3% were dead — plain HTTP 404s, including federal budget files, with nothing in the metadata to warn you. Check that distribution links are alive attaches status, content type, byte size, redirect target and latency to every URL, and Drop rows whose links are dead removes the corpses.

⚖️ Pull the EU High-Value Datasets and their legal basis.

10,514 datasets carry the HVD flag from Implementing Regulation (EU) 2023/138 — the categories member states must publish openly and free of charge. Filter by category (earth-environment 9,419 · geospatial 603 · mobility 313 · statistics 217 · meteorological 81 · companies 5) and each row carries the applicable_legislation URI it was published under.

🗺️ Get geodata coverage without a GIS library.

72,621 datasets carry a spatial extent. Rows give you boundingBox, centroidLon and centroidLat computed from the geometry, plus the raw GeoJSON, the AGS regional keys and the administrative level (Bund / Land / Kreis / Gemeinde). geodaten deutschland and inspire geodaten are 110 and 40 searches a month in Germany.

📬 Build a public-sector contact list.

The Contact points dataset deduplicates publishers, contacts, authors and maintainers out of whatever you scanned. A 1,400-dataset sample held 175 distinct publishing bodies. Each row carries email, phone, street, postcode and city where the publisher supplied them — 93,876 datasets carry a machine-readable contact email.

🔔 Run it as a change monitor.

Set Metadata modified after to yesterday and sort by recently updated: 50,773 records changed in the last 7 days. Set First published after instead and you get genuinely new datasets — 6,894 in the last 30 days — rather than re-harvested ones.

Why choose this scraper

	What you get
Eight datasets, one Actor	Datasets, distributions, contacts, organizations, categories, tags, formats and licences.
84 fields, not raw CKAN	The DCAT-AP.de payload arrives as a flat key/value list with JSON arrays serialised into strings. Every one is parsed out into a named, typed field.
Filters that work	Every filter here was measured against the unfiltered total. Three of the obvious ones are broken upstream and are handled or excluded — see below.
Nothing always-null shipped	Field coverage was measured across a 2,000-dataset sample spread over the whole catalogue before any column was included.
Link checking and column previews	Both optional, both off by default, both billed only for work actually done.
Filters run before the extras are bought	Narrow by licence, format or text on the complete row; a row you filter out never costs a link check.
No login, no API key, no proxy	164 requests at up to 16 concurrent workers ran clean from a bare datacenter IP with no throttling.
Ends cleanly at your budget	At Apify's per-run charge ceiling the run stops and finishes SUCCEEDED with what it has, instead of being aborted.

Four traps this Actor handles for you

The format filter is a lie if you take it literally. res_format holds the raw string each harvesting portal supplied, with no normalisation — 403 distinct values for about thirty real formats, 103 of them full EU file-type URIs and 300 free text. Filtering res_format:"CSV" looks right and returns 7,859 datasets. The real number with a CSV distribution is 68,582: most carry http://publications.europa.eu/resource/authority/file-type/CSV instead, plus lowercase csv, .csv and text/csv. Every format name you type is expanded to all of them, and the Formats dataset shows you each variant with its own count.

Spatial bounding-box search silently returns everything. CKAN's ext_bbox parameter is accepted on this portal and answers the full 156,461 for any box you give it — the spatial extension is not installed. A bbox filter would look like it worked and quietly not filter. There isn't one here. Use Regional keys, Administrative level or the boundingBox on the row instead.

An unknown sort field does not error, it reverts to relevance. views_recent desc and relevance asc are both accepted and both return the identical first row as score desc. Only the eight sort orders verified to actually change the order are offered.

The licence is not on the dataset. Package-level license_id is populated on 1,245 of 156,461 records — 0.8%. The licence that governs the data sits on each distribution, where it is populated everywhere. Rows carry it as licenses / licenseCodes on the dataset and license on each distribution, and the Licences filter matches there.

How it compares

No other Apify Actor scrapes GovData.de. We searched the Store for govdata, open data germany, german open data, ckan and dcat. What exists is generic CKAN and Socrata exporters that point at any portal and hand back the raw package.

Feature	ParseForge (this)	doggo/ckan-opendata-exporter	straightforward_hydra/ckan-open-data-scraper	datapilot/open-data-portal-harvester	crawlerbros/eu-open-data-scraper
Source	GovData.de	Any CKAN	Any CKAN	Any portal	EU Open Data Portal
DCAT-AP.de fields parsed out	84	Raw CKAN	Raw CKAN	Raw	Raw
Format filter expanded to all 403 spellings	Yes	No	No	No	No
High-Value Dataset flag and category	Yes	No	No	No	No
Bounding box + centroid computed	Yes	No	No	No	No
Regional keys (AGS) and administrative level	Yes	No	No	No	No
Distribution link checking	Optional	No	No	No	No
Column preview of the actual file	Optional	No	No	No	No
Contact-point directory	Yes	No	No	No	No
Organization / theme / tag / format / licence directories	5 datasets	No	No	No	No
Price per 1,000 rows	$7.00	$1.00	$2.00	$2.00	$5.00

Why $7 per 1,000 when generic CKAN exporters charge $1. They are not selling the same row. Point a generic exporter at www.govdata.de/ckan and you get CKAN's package_search output: extras as a key/value list, ["http://publications.europa.eu/resource/authority/data-theme/ENVI"] as a string inside a string, no HVD flag, no bounding box, no AGS, no idea which of the 403 format spellings to filter on, and no signal that 5.3% of the links are dead. Getting from that to something you can query is the work this Actor did once. If you want a raw CKAN dump of any portal, buy the $1 exporter — it is the right tool for that job.

This Actor is HTTP-only with no browser and no proxy. Measured on the platform:

Run	Rows	Our cost per 1,000 rows
Datasets, 5,000-row sweep	5,000	$0.0054
Datasets, default input	100	$0.0080
Distributions with link check and column preview	120	$0.0171
Organizations with profiles (worst case: fixed cost over 6 rows)	6	$0.0496

Even the deliberately worst-shaped run — six rows, every optional block on — costs us five cents per thousand. We are not pricing on cost; the house floor is $5 and this row is not a commodity.

What a row looks like

Datasets (dataset: datasets) — 84 fields, abridged:

{
  "id": "182ec075-3a65-4756-9ee5-059cc888d1ae",
  "name": "mobilpunkte",
  "title": "Mobilpunkte",
  "notes": "Standorte der Mobilpunkte in der Hanse- und Universitätsstadt Rostock",
  "url": "https://www.govdata.de/suche/daten/mobilpunkte",
  "landingPage": "https://www.opendata-hro.de/dataset/9c32aba3-4380-4372-b31c-d0ba53fba326",
  "organization": "opendata-hro",
  "organizationTitle": "OpenData.HRO",
  "publisherName": "Hanse- und Universitätsstadt Rostock",
  "sourcePortal": "http://www.opendata-hro.de/",
  "categories": ["gove", "regi", "envi", "tran"],
  "categoryLabels": ["Regierung und öffentlicher Sektor", "Regionen und Städte", "Umwelt", "Verkehr"],
  "themeCodes": ["GOVE", "REGI", "TRAN", "ENVI"],
  "tags": ["automobil", "bahnhof", "busbahnhof", "elektrizität", "individualverkehr"],
  "isHighValueDataset": true,
  "hvdCategories": ["http://data.europa.eu/bna/c_b79e35eb"],
  "hvdCategoryLabels": ["Mobility"],
  "applicableLegislation": ["http://data.europa.eu/eli/reg_impl/2023/138/oj"],
  "licenses": ["http://dcat-ap.de/def/licenses/cc-zero"],
  "licenseCodes": ["cc-zero"],
  "accessRights": "http://publications.europa.eu/resource/authority/access-right/PUBLIC",
  "issued": "2020-04-02T12:51:11.247095",
  "metadataModified": "2026-08-27T07:04:23.785420",
  "languageCodes": ["DEU"],
  "spatialText": "Rostock",
  "regionalKeys": ["130030000000"],
  "administrativeLevel": "administrativeDistrict",
  "boundingBox": { "west": 11.99143, "south": 54.03856, "east": 12.31306, "north": 54.26196 },
  "centroidLon": 12.152245,
  "centroidLat": 54.15026,
  "contactName": "Hanse- und Universitätsstadt Rostock",
  "contactEmail": "geodienste@rostock.de",
  "resourceCount": 7,
  "formats": ["CSV", "XLSX", "GEOJSON", "KML", "GML"],
  "totalSizeBytes": 28673,
  "resources": [
    {
      "name": "Mobilpunkte",
      "format": "CSV",
      "formatRaw": "http://publications.europa.eu/resource/authority/file-type/CSV",
      "url": "https://geo.sv.rostock.de/download/opendata/mobilpunkte/mobilpunkte.csv",
      "license": "http://dcat-ap.de/def/licenses/cc-zero",
      "sizeBytes": 6422,
      "linkCheck": {
        "status": 200, "ok": true, "contentType": "text/x-comma-separated-values",
        "contentLength": 6422, "redirected": false, "method": "HEAD", "elapsedMs": 1176
      },
      "columnPreview": {
        "ok": true, "delimiter": ",", "encoding": "utf-8", "columnCount": 12,
        "columns": ["uuid", "bezeichnung", "latitude", "longitude", "betreiber"]
      }
    }
  ],
  "scrapedAt": "2026-08-27T14:22:32.420Z"
}

Formats (dataset: formats) — the trap, made visible. datasetCountSum adds up the per-variant counts, so it reads slightly differently from the 68,582 above: that figure comes from the expanded filter, which also catches long-tail spellings like .csv and gezippte csv-dateien that fall outside the top variants. Use largestVariantCount when you want an exact number, and the filter when you want the rows.

{
  "format": "CSV",
  "datasetCountSum": 68444,
  "largestVariantCount": 60287,
  "rank": 1,
  "variantCount": 5,
  "variants": [
    { "value": "http://publications.europa.eu/resource/authority/file-type/CSV", "count": 60287 },
    { "value": "CSV", "count": 7859 },
    { "value": "https://www.iana.org/assignments/media-types/text/csv", "count": 178 },
    { "value": "https://data-lra-cha.opendata.arcgis.com/api/feed/dcat-ap/ftype/CSV", "count": 105 },
    { "value": "http://publications.europa.eu/resource/authority/file-type/csv", "count": 15 }
  ],
  "isMachineReadable": true,
  "isService": false
}

Contact points (dataset: contacts):

{
  "role": "contact",
  "name": "Berlin.de",
  "email": "opendata@senfin.berlin.de",
  "organization": "berlin-open-data",
  "organizationTitle": "Berlin Open Data",
  "publisherName": "Senatsverwaltung für Finanzen Berlin",
  "sourcePortal": "http://datenregister.berlin.de",
  "datasetCount": 6,
  "categories": ["gove"],
  "exampleDatasetTitle": "Aktenplan der Senatsverwaltung für Finanzen"
}

Configure the run

Input	What it does
Dataset	Which of the eight surfaces to pull. Each bills its own per-row event.
Search query	Full text across titles, descriptions, tags and publishers, exactly as the portal's own box works. German stemming applies — Wasser also matches Gewässer.
Title contains / Description contains	Field-scoped search, ANDed with the above.
Organizations	Publisher slugs. Run the Organizations dataset once for the list of 76 with counts.
Categories	The 13 EU data themes.
Tags	Exact keywords, matched case-sensitively. Run the Tags dataset to see what exists.
Distribution formats	CSV, GEOJSON, WFS, PARQUET … expanded to every spelling the catalogue uses.
Exclude formats	Drop datasets publishing a format you do not want.
High-Value Datasets only / HVD categories	The EU 2023/138 mandate, whole or by category.
Administrative level	Bund, Land, Kreis, Gemeinde.
Regional keys (AGS)	130030000000 for Rostock, 11 for Berlin as a Land.
Metadata language	German covers 93,655 records; English 289.
Update frequency	Ten options, each matching both vocabularies publishers use.
Publisher name / Source portal contains	Finer than the organization: 175 distinct publishers per 1,400 datasets.
Dataset type	dct:type — dataset, series, collection.
Only datasets with a geometry	72,621 qualify, and every row then has a bounding box.
Minimum distributions	Set 1 to exclude the 239 metadata-only records.
Modified / published after and before	ISO dates. This is the change-monitor input.
Data covers time from / to	The period the DATA covers, not when the record was edited.
Licences	Substring match on the distribution licence — dl-by-de, cc-by, cc-zero.
Row must contain	Plain substring over the finished row, applied before any optional block is bought.
Filter query (advanced)	Raw Solr fq. An unknown field name returns zero rows rather than an error.
Three optional-block checkboxes	Link check, column preview, organization profile. All off by default.
Sort by	Eight verified orders. Use Stable id order for full sweeps.
Max items / Max catalogue pages	Hard ceilings on rows and cost. One page is 500 datasets.
Concurrency	5 by default, up to 12. See the throughput note below.
Proxy	Off by default. Not needed.

About throughput and throttling

Measured 2026-08-27 from a bare datacenter IP, 164 requests across 1, 2, 4, 8 and 16 concurrent workers: zero failures, zero throttles, no rate limiting of any kind. Server latency is a flat ~2.8 s per request regardless of concurrency, so parallelism scales cleanly:

Page size	Concurrency	Throughput
500	8	525 dataset rows / s
1000	8	521 rows / s
500	4	274 rows / s
100	16	182 rows / s

Above 8 workers the per-request overhead starts to dominate, which is why the input caps at 12. The portal does occasionally answer HTTP 200 with a zero-byte body under load; that is a load-shedding hiccup, not an empty page, and it is retried rather than treated as a result.

One quirk worth knowing if you script against the API yourself: GovData answers 403 to curl's default User-Agent on URLs that work perfectly with a browser one. This Actor always sends a browser UA.

Pricing

Pay-per-event. You are billed per row written, plus one small charge per catalogue page fetched and a $0.054 run-start fee. Filtered-out rows cost nothing.

Event	Price	When it fires
Run start	$0.054	Once per run
Catalogue page scanned	$0.004	Each page of up to 500 datasets, or one directory call
Dataset	$0.007	datasets
Distribution	$0.005	resources
Contact point	$0.006	contacts
Organization	$0.010	organizations
Category	$0.010	categories
Tag	$0.003	tags
Format	$0.003	formats
Licence	$0.005	licenses
Distribution link checked	$0.006	Checkbox on, per URL checked
Distribution columns previewed	$0.015	Checkbox on, per file actually read
Organization profiled	$0.012	Checkbox on, per organization

Exactly one per-row event fires on any given run — the one for the dataset you picked.

Datasets collected	Approximate cost
100	$0.76
1,000	$7.07
10,000	$70.13

The three optional blocks are off by default. A run that leaves them off pays only the run-start fee, one catalogue-page event per page, and the per-row rate — verified on the platform: a default run charged apify-actor-start=1, search-page=1, dataset-row=100 and nothing else.

Column previews are not billed when they are skipped. A distribution in a format that cannot be read as columns — XLSX, SHP, KML, GML, ZIP — returns early without making a request and costs nothing. Measured on the platform: 120 distributions, 120 link checks, 69 previews billed.

Free users

Apify's free plan caps what any paid Actor can bill in a month. If a run stops early with a usage message, that is the plan cap, not this Actor. Free-plan runs here return up to 50 rows as a preview, and the log says so.

Separately, Apify gives every run a maximum cost derived from your remaining allowance. A whole-catalogue sweep is large enough to hit that on any plan. When it does, this Actor stops and finishes cleanly with what it collected rather than letting the platform abort the run — measured 2026-08-27: an unbounded sweep hit the ceiling at 2,200 rows and ended SUCCEEDED, where the previous build ran on to 6,100 rows and was aborted by the platform. Set Max items to keep a run inside a budget you choose.

Run it

1. Pick a Dataset — start with Datasets.
2. Narrow it: a Search query, a Category, an Organization, or a Distribution format.
3. Set Max items if you want a hard ceiling on cost.
4. Run, then export as CSV, JSON, Excel or XML — or pull it from the API.

The default input — Umwelt datasets that publish a CSV — returns 100 rows in about four seconds from 15,935 matches.

Troubleshooting

Zero rows and I set a Filter query. An unknown Solr field name returns zero rows rather than an error on this API. Check the spelling. The log says this explicitly whenever a run writes nothing.

My format filter returns far fewer datasets than the portal shows. It should not — that is exactly the trap this Actor fixes. If you passed a raw value through Filter query instead of using Distribution formats, you are hitting one of 403 spellings. Run the Formats dataset to see all of them.

A licence filter returned nothing. Licences live on the distributions, not the dataset. Use the Licences input, which matches there, not license_id through Filter query — that field is populated on 0.8% of the catalogue.

A bounding-box filter isn't filtering. There isn't one, deliberately: CKAN's ext_bbox returns the whole catalogue for any box on this portal. Use Regional keys or Administrative level.

The run ended early saying it reached a charge ceiling. That is Apify's per-run cost cap, derived from your remaining plan allowance. The rows already written are yours and the shortfall was not billed. Set Max items lower, or raise the allowance.

A column preview came back ok: false. Either the format is not tabular — in which case it was skipped and not billed — or the file needs authentication, or the link is dead. Turn on the link check to tell those apart.

hvdCategories has a URI but isHighValueDataset is true with an empty array. Some publishers put a German label in that field instead of the URI. The flag counts the label; the array keeps only resolvable URIs.

FAQ

Does it need a login or an API key? No. Every surface used here is public and anonymous.

Does it need a proxy? No. 164 requests at up to 16 concurrent workers ran clean from a bare datacenter IP. Turn it on only if you are sweeping the whole catalogue repeatedly on a schedule.

Can I get the whole catalogue in one run? 156,461 datasets at 500 per page is 313 pages. Raise Max catalogue pages, sort by Stable id order, and watch your Max items so the run finishes inside your cost ceiling. Deep paging is safe here — 4,000 contiguous rows at three different sort orders returned zero duplicates.

How current is it? Live. GovData re-harvests continuously; 50,773 records changed in the last 7 days.

Is the data in German? The metadata is: titles, descriptions and keywords are German, because the datasets are. Category names come with both German and English titles, and the field names are English.

What is DCAT-AP.de? The German profile of the EU's DCAT-AP metadata standard. It is what every German portal publishes and what GovData harvests. This Actor flattens it into named fields instead of handing you the raw key/value list.

What is an AGS? The Amtlicher Gemeindeschlüssel, the official numeric key for every German municipality, district and Land. 34,436 datasets carry one, and you can filter on it.

Why is matchingDatasetCount null on a directory row? Because you ran without filters, so it would be identical to datasetCount. Set any filter and it fills in.

Related actors

• EU Open Data Portal Scraper — the European Commission's catalogue, same DCAT lineage
• Opendata Swiss Datasets Scraper — Switzerland's national portal
• Luxembourg Open Data Scraper — data.public.lu
• DWD Germany Weather Stations Scraper — German weather observations at source