CSW Catalog Harvester

What does CSW Catalog Harvester do?

It pulls dataset metadata records from OGC CSW 2.0.2 catalogs (the "Catalogue Service for the Web" endpoint behind many GeoNetwork, pycsw and ESRI Geoportal sites) and returns one flat row per record: identifier, title, type, subjects, format, abstract, bounding box, the catalog it came from and the original XML.

Typical buyer job: "Give me a spreadsheet of the datasets a geoportal publishes, with titles, keywords and footprints, so I can inventory, search, or cross-check them." No login is needed for public catalogs.

Quick start: 5 records from the pycsw demo catalog

This is the exact sample input shown on the Input tab. Press Start as-is:

{
  "catalogUrls": ["https://demo.pycsw.org/gisdata/csw"],
  "elementSetName": "summary",
  "typeNames": "csw:Record",
  "recordsPerPage": 5,
  "startPosition": 1,
  "maxRecords": 5
}

It returns 5 records (at the Free-plan price, $0.025 in record charges). On the last verified cloud run the catalog reported 171 matching records in total; the run stops at 5 because of maxRecords.

Output example (real row from that sample run)

Trimmed for length (abstract and fields shortened, _raw cut, most empty Dublin Core fields omitted):

{
  "identifier": "urn:uuid:dc9d02a2-932a-11ea-ad6f-823cf448c401",
  "title": "AG_PRESERVE_CONTRACTS",
  "type": "vector digital data",
  "subject": ["Ag Contracts", "Williamson Act", "Contract Lands"],
  "format": "vector digital data",
  "abstract": "Williamson Contract lands with Preserve Numbers. ...",
  "bbox": [-117.35, 32.6, -116.29, 33.45],
  "bboxCrs": "urn:x-ogc:def:crs:EPSG:6.11:4326",
  "date": null,
  "creator": null,
  "_catalogUrl": "https://demo.pycsw.org/gisdata/csw",
  "_elementSetName": "summary",
  "_recordType": "SummaryRecord",
  "_recordPosition": 2,
  "_numberOfRecordsMatched": 171,
  "fields": {"identifier": ["urn:uuid:dc9d02a2-932a-11ea-ad6f-823cf448c401"], "title": ["AG_PRESERVE_CONTRACTS"], "subject": ["Ag Contracts", "Williamson Act", "Contract Lands"], "...": "..."},
  "_raw": "<ns0:SummaryRecord ...><dc:identifier>urn:uuid:dc9d02a2-...</dc:identifier>..."
}

The other four sample titles were AG_PRESERVE, Agriculture_Commodity, Point_Of_Sale and Standards_Devices.

Columns you will use

Field	Meaning
identifier, title, type	Record ID, dataset name, resource type
subject, format, abstract	Keywords, format, description (richer with summary/full)
bbox, bboxCrs	Footprint as [west, south, east, north] plus the CRS the server stated
date, modified, created, creator, publisher, rights, ...	Dublin Core fields; null when the catalog does not send them
_catalogUrl, _recordPosition, _numberOfRecordsMatched	Provenance: which catalog, which position, how many matched
fields	Every Dublin Core term the record carried, with all its values
_raw	The original record XML (capped at 200,000 characters)

Dates depend on the server and the element set. The Output table shows both date (dc:date) and modified (dct:modified). The pycsw demo sends no date in summary records, so the sample shows both empty; set elementSetName to full there and date is filled (for example 20070911). The EEA GeoNetwork catalog sends modified already in summary. The Actor never fills a missing date.

Export as JSON, CSV, Excel or HTML from the Output tab or the API.

Export a catalog inventory to CSV (Python, standard library only)

This script starts one small run against the EEA GeoNetwork catalog, waits for it, and saves csw_inventory.csv with one row per record: identifier, title, date, modified and provenance. Set APIFY_TOKEN in your environment first; it is sent as a Bearer header and never printed. Change catalogUrls or maxRecords for your own catalog.

import csv, json, os, sys, time, urllib.error, urllib.request

TOKEN = os.environ["APIFY_TOKEN"]  # read from the environment, never printed
API = "https://api.apify.com/v2"
ACTOR = "datamule~csw-catalog-harvester"
RUN_INPUT = {
    "catalogUrls": ["https://sdi.eea.europa.eu/catalogue/srv/eng/csw"],
    "elementSetName": "summary",
    "recordsPerPage": 5,
    "maxRecords": 5,
}
MAX_CHARGE_USD = 0.03  # hard spend cap for this run
WAIT_SECONDS = 300
OUT = "csw_inventory.csv"
COLUMNS = ["identifier", "title", "date", "modified", "_catalogUrl",
           "_elementSetName", "_recordPosition", "_numberOfRecordsMatched"]


def call(method, path, body=None):
    req = urllib.request.Request(
        API + path, method=method,
        data=None if body is None else json.dumps(body).encode(),
        headers={"Authorization": "Bearer " + TOKEN, "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)


def cell(value):  # None stays an empty cell (= the catalog did not send it)
    if value is None:
        return ""
    return "; ".join(map(str, value)) if isinstance(value, list) else value


try:
    run = call("POST", f"/acts/{ACTOR}/runs?maxTotalChargeUsd={MAX_CHARGE_USD}", RUN_INPUT)["data"]
    deadline = time.time() + WAIT_SECONDS
    while run["status"] in ("READY", "RUNNING"):
        if time.time() > deadline:
            sys.exit(f"Run {run['id']} still {run['status']} after {WAIT_SECONDS}s; {OUT} not changed.")
        time.sleep(10)
        run = call("GET", f"/actor-runs/{run['id']}")["data"]
    if run["status"] != "SUCCEEDED":
        sys.exit(f"Run {run['id']} ended {run['status']}; {OUT} not changed.")
    rows = call("GET", f"/datasets/{run['defaultDatasetId']}/items?clean=1&format=json")
except (urllib.error.URLError, OSError, KeyError, ValueError) as err:
    sys.exit(f"API call failed ({type(err).__name__}); {OUT} not changed.")

if not isinstance(rows, list) or any("identifier" not in r for r in rows):
    sys.exit(f"Unexpected dataset response; {OUT} not changed.")
tmp = OUT + ".tmp"
with open(tmp, "w", newline="", encoding="utf-8") as fh:
    writer = csv.writer(fh)
    writer.writerow(COLUMNS)
    writer.writerows([cell(r.get(c)) for c in COLUMNS] for r in rows)
os.replace(tmp, OUT)  # the old CSV is replaced only after a complete write
print(f"Run {run['id']} SUCCEEDED: wrote {len(rows)} record(s) to {OUT}")

• Cost: 5 record events, $0.025 at the Free-plan price ($0.005 each, less on higher plans). maxTotalChargeUsd=0.03 caps the run; raise it together with maxRecords. Platform usage is included in the event price.
• Empty date/modified cells mean the catalog did not send that field for the record. The script does not guess dates. EEA summary records often carry modified but no date; use elementSetName: "full" for more Dublin Core fields where the server supports it.
• Failure is safe: a failed, aborted or timed-out run, an API error or an unexpected response exits non-zero and leaves any existing csw_inventory.csv untouched. A valid query with no matches writes a header-only CSV.
• The script stops waiting after 5 minutes; the run may still finish on Apify within the charge cap.

Use your own catalog or query

1. Replace catalogUrls with the CSW endpoint URL, the one that answers service=CSW&request=GetRecords. You can list several; each is harvested in turn. If you only have the portal's web address (a GeoNetwork catalog.search page or a pycsw site root), paste it: the run tries the standard /srv/<lang>/csw or /csw endpoint once and logs which one it used. Another catalog we verified when this Actor was built (July 2026) is the European Environment Agency GeoNetwork: https://sdi.eea.europa.eu/catalogue/srv/eng/csw.
2. Optional filters:
   • cqlFilter: a CQL constraint. On the pycsw demo, csw:AnyText like '%Williamson%' matched 2 records and dc:title like 'AG%' matched 5. The upper-case LIKE form and unprefixed names such as AnyText were rejected there as invalid. Which property names work depends on the server.
   • bbox: minLon,minLat,maxLon,maxLat, e.g. 3.3,50.7,7.2,53.6. It is ANDed with cqlFilter.
3. Set maxRecords to the total you want per catalog. If you leave it empty you get a single page (recordsPerPage, default 20). A larger value pages through the results automatically.
4. elementSetName: brief (id, title, type, bbox), summary (default) or full.

How much does it cost?

Pay per event: one record event per row written. Nothing else is charged by this Actor: no start fee and no per-page fee.

Apify plan	Price per record	1,000 records
Free	$0.005	$5.00
Bronze	$0.0045	$4.50
Silver	$0.004	$4.00
Gold	$0.0035	$3.50
Platinum	$0.003	$3.00
Diamond	$0.0025	$2.50

Max charge per run is honoured. Before each catalog and each page, the Actor checks how many records your limit still covers. It asks for no more than that and writes only records that are paid for. We checked this on the cloud: a $0.011 limit returned exactly 2 records, and a $0.004 limit (less than one record) returned 0 rows without contacting the catalog. If you restart a run, it continues after the last delivered record and does not bill any record twice.

Under Apify's pay-per-event model, platform usage for the run is covered by the event price. Your own Apify plan limits and any storage you keep afterwards still follow Apify's normal rules.

Limits and failure behaviour

• CSW 2.0.2 only, Dublin Core output (csw:Record). Every row uses the same flat shape. Asking for ISO profiles such as gmd:MD_Metadata via typeNames works only if the server supports it, and the rows are still flattened to Dublin Core fields. OGC API Records and CSW 3.0 are not supported.
• Requests are sent as XML POST, which suits POST-only servers such as GeoNetwork. Some catalogs block or rate-limit automated clients, and some lock down CSW entirely. Coverage differs from site to site, so test a small maxRecords first.
• A catalog that is unreachable, returns an HTTP 4xx, returns an OWS ExceptionReport (for example an invalid CQL filter) or returns non-CSW content is skipped with a warning and the next URL is tried. If every catalog fails, the run exits with an error (code 91). A valid query that matches nothing returns 0 rows and no charge.
• If a later page fails, the rows already delivered and billed are kept, and harvesting stops for that catalog.
• recordsPerPage is capped at 100 by the Actor; individual servers may return fewer per page.
• For gated catalogs use bearerToken or extraHeaders. Neither is logged.

Support

Report problems or request a catalog in the Issues tab. Please include the catalog URL and your input.