GovData Germany Scraper - CKAN Open Data Portal
Pricing
Pay per event
Scrape Germany's national open data portal: 156,000+ Datensätze with full DCAT-AP.de metadata, distributions, publishers, EU data themes, High-Value Dataset flags, licences and live link checks.
Which surface of the GovData catalogue to return. 'Datasets' is one row per dataset with its distributions nested; 'Distributions' flattens to one row per file or service; 'Contact points' deduplicates the publishing bodies behind them. The remaining four are directories and each cost a single request.
Full-text search across titles, descriptions, tags and publishers, exactly as the portal's own search box works. German stemming applies, so 'Wasser' also matches 'Gewässer'. Leave empty for the whole catalogue.
Restrict the search to dataset titles. Combined with Search query using AND.
Restrict the search to dataset descriptions. Combined with Search query using AND.
Publishing organization slugs, e.g. berlin-open-data, open-data-bayern, transparenzportal-hamburg, gdi-de. Run the 'Organizations' dataset once to get the full list of 76 with their dataset counts. Multiple values are ORed.
The 13 EU data themes. Multiple values are ORed.
Keep only datasets that publish at least one distribution in these formats, e.g. CSV, JSON, GEOJSON, XLSX, SHP, WFS, PARQUET. Each name is expanded to every spelling the catalogue actually uses — the 403 raw values include full EU file-type URIs, bare names and free text, and filtering on the bare name alone would miss 88% of the CSV datasets.
Drop any dataset that publishes one of these formats. Applied after fetching, on the normalised short name (CSV, PDF, WMS, …).
dct:type as DCAT-AP.de records it. Populated on about a quarter of the catalogue; leave empty unless you specifically need it.
Keep only datasets flagged as EU High-Value Datasets under Implementing Regulation (EU) 2023/138 — the categories member states must publish openly and free of charge. 10,514 of the catalogue qualify.
Narrow to specific HVD categories. Overrides the checkbox above. Measured counts: earth-environment 9,419 · geospatial 603 · mobility 313 · statistics 217 · meteorological 81 · companies 5.
Which tier of German government the dataset covers, from the DCAT-AP.de political geocoding vocabulary. Measured: municipality 24,910 · state 19,986 · district 4,089 · federal 3,789.
Amtlicher Gemeindeschlüssel codes, e.g. 130030000000 for Rostock or 11 for Berlin as a state. Matches the dcat-ap.de regionalKey and stateKey URIs. Multiple values are ORed.
The declared language of the dataset record. German covers 93,655 records; English only 289, so this is mostly useful for excluding non-German material.
How often the publisher says the data is refreshed. Two vocabularies are in use across the catalogue for the same idea, and each option matches both.
Exact publisher name as DCAT-AP.de records it, e.g. 'Ministerium für Bildung'. This is the actual publishing body, which is finer-grained than the CKAN organization — 76 organizations front many thousands of publishers.
Substring of the regional portal GovData harvested the record from, e.g. berlin, hamburg, rlp. GovData is a harvesting catalogue and this is the trail back to the original.
Keep only records that declare the EU access-right value PUBLIC (28,951 datasets). Most publishers put free German prose in this field instead of the controlled value, so this narrows hard — leave it off unless you need the machine-readable declaration.
Keep only records carrying a spatial extent, so every row gets a bounding box and centroid. 72,621 datasets qualify.
Drop datasets with fewer than this many distributions. Set to 1 to exclude the 239 metadata-only records.
ISO date or datetime, e.g. 2026-08-01. Together with the sort order, this is how you run the Actor as a change monitor.
ISO date or datetime. Filters on when the record first appeared on GovData, which is how you find genuinely new datasets rather than re-harvested ones.
ISO date. Filters on the period the DATA covers, not on when the record was edited. A dataset matches when its coverage window overlaps yours.
Keep only datasets whose distributions carry one of these licences — substring match against the licence URI, e.g. dl-by-de, cc-by, cc-zero, official-work. Applied after fetching, because the licence lives on the distribution: package-level licence is empty on 99.2% of the catalogue.
Plain substring filter over the finished row's title, description, publisher, tags and place name. Runs after fetching and before any optional block is bought, so filtering never costs you a link check.
Raw Solr fq appended to everything above, e.g. extras_conforms_to:* or res_url:wfs. Careful: an unknown field name returns zero rows rather than an error.
HEAD every distribution URL and attach status, content type, byte size, redirect target and latency. GovData harvests metadata from ~100 regional portals and never re-verifies the links: measured 2026-08-27 across the full age range of the catalogue, 5.3% of distributions were dead 404s with nothing in the metadata to warn you. Billed per URL checked.
Read the first 64 KB of each CSV, TSV, JSON, GeoJSON or XML distribution with a range request and return its actual column names, delimiter, encoding and a sample row. The catalogue tells you a dataset is 'a CSV'; it never tells you what is inside. Billed per distribution previewed.
Requires the link check above. Discards datasets where every distribution failed, or distributions that failed individually. Dropped rows are not billed.
Only for the 'Organizations' dataset. One extra request per organization returning what it actually publishes: its themes, its top formats and its top keywords, each with counts. Billed per organization profiled.
Attach the untouched CKAN extras key/value map alongside the parsed fields. Free — the data is already in the response — but it roughly doubles the row size.
Only sort fields verified to actually change the order are offered. An unknown Solr sort field does not error on this API, it silently reverts to relevance.
How many rows to return for the Tags and Formats directories. The tag vocabulary has 97,863 entries and the long tail is mostly one-off keywords.
Hard ceiling on rows written, and therefore on cost. Leave empty for no ceiling.
Each page is 500 datasets and bills one catalogue-page event. 20 pages covers 10,000 datasets.
Parallel requests, 1 to 12. Measured on a bare datacenter IP: 8 workers moved 525 dataset rows a second with zero failures and no throttling. Above 8 the per-request overhead starts to dominate.
Soft wall-clock budget. The run stops issuing requests and flushes what it has. The platform's own run timeout still applies and whichever is nearer wins.
