CSW Catalog Harvester

What does CSW Catalog Harvester do?

It pulls dataset metadata records from OGC CSW 2.0.2 catalogs (the "Catalogue Service for the Web" endpoint behind many GeoNetwork, pycsw and ESRI Geoportal sites) and returns one flat row per record: identifier, title, type, subjects, format, abstract, bounding box, the catalog it came from and the original XML.

Typical buyer job: "Give me a spreadsheet of the datasets a geoportal publishes, with titles, keywords and footprints, so I can inventory, search, or cross-check them." No login is needed for public catalogs.

Quick start: 5 records from the pycsw demo catalog

This is the exact sample input shown on the Input tab. Press Start as-is:

{
  "catalogUrls": ["https://demo.pycsw.org/gisdata/csw"],
  "elementSetName": "summary",
  "typeNames": "csw:Record",
  "recordsPerPage": 5,
  "startPosition": 1,
  "maxRecords": 5
}

It returns 5 records (at the Free-plan price, $0.025 in record charges). On the last verified cloud run the catalog reported 171 matching records in total; the run stops at 5 because of maxRecords.

Output example (real row from that sample run)

Trimmed for length (abstract and fields shortened, _raw cut, most empty Dublin Core fields omitted):

{
  "identifier": "urn:uuid:dc9d02a2-932a-11ea-ad6f-823cf448c401",
  "title": "AG_PRESERVE_CONTRACTS",
  "type": "vector digital data",
  "subject": ["Ag Contracts", "Williamson Act", "Contract Lands"],
  "format": "vector digital data",
  "abstract": "Williamson Contract lands with Preserve Numbers. ...",
  "bbox": [-117.35, 32.6, -116.29, 33.45],
  "bboxCrs": "urn:x-ogc:def:crs:EPSG:6.11:4326",
  "date": null,
  "creator": null,
  "_catalogUrl": "https://demo.pycsw.org/gisdata/csw",
  "_elementSetName": "summary",
  "_recordType": "SummaryRecord",
  "_recordPosition": 2,
  "_numberOfRecordsMatched": 171,
  "fields": {"identifier": ["urn:uuid:dc9d02a2-932a-11ea-ad6f-823cf448c401"], "title": ["AG_PRESERVE_CONTRACTS"], "subject": ["Ag Contracts", "Williamson Act", "Contract Lands"], "...": "..."},
  "_raw": "<ns0:SummaryRecord ...><dc:identifier>urn:uuid:dc9d02a2-...</dc:identifier>..."
}

The other four sample titles were AG_PRESERVE, Agriculture_Commodity, Point_Of_Sale and Standards_Devices.

Columns you will use

Field	Meaning
identifier, title, type	Record ID, dataset name, resource type
subject, format, abstract	Keywords, format, description (richer with summary/full)
bbox, bboxCrs	Footprint as [west, south, east, north] plus the CRS the server stated
date, modified, created, creator, publisher, rights, ...	Dublin Core fields; null when the catalog does not send them
_catalogUrl, _recordPosition, _numberOfRecordsMatched	Provenance: which catalog, which position, how many matched
fields	Every Dublin Core term the record carried, with all its values
_raw	The original record XML (capped at 200,000 characters)

Export as JSON, CSV, Excel or HTML from the Output tab or the API.

Use your own catalog or query

1. Replace catalogUrls with the CSW endpoint URL, the one that answers service=CSW&request=GetRecords. You can list several; each is harvested in turn. Another catalog we verified when this Actor was built (July 2026) is the European Environment Agency GeoNetwork: https://sdi.eea.europa.eu/catalogue/srv/eng/csw.
2. Optional filters:
   • cqlFilter: a CQL constraint. On the pycsw demo, csw:AnyText like '%Williamson%' matched 2 records and dc:title like 'AG%' matched 5. The upper-case LIKE form and unprefixed names such as AnyText were rejected there as invalid. Which property names work depends on the server.
   • bbox: minLon,minLat,maxLon,maxLat, e.g. 3.3,50.7,7.2,53.6. It is ANDed with cqlFilter.
3. Set maxRecords to the total you want per catalog. If you leave it empty you get a single page (recordsPerPage, default 20). A larger value pages through the results automatically.
4. elementSetName: brief (id, title, type, bbox), summary (default) or full.

How much does it cost?

Pay per event: one record event per row written. Nothing else is charged by this Actor: no start fee and no per-page fee.

Apify plan	Price per record	1,000 records
Free	$0.005	$5.00
Bronze	$0.0045	$4.50
Silver	$0.004	$4.00
Gold	$0.0035	$3.50
Platinum	$0.003	$3.00
Diamond	$0.0025	$2.50

Max charge per run is honoured. Before each catalog and each page, the Actor checks how many records your limit still covers. It asks for no more than that and writes only records that are paid for. We checked this on the cloud: a $0.011 limit returned exactly 2 records, and a $0.004 limit (less than one record) returned 0 rows without contacting the catalog. If you restart a run, it continues after the last delivered record and does not bill any record twice.

Under Apify's pay-per-event model, platform usage for the run is covered by the event price. Your own Apify plan limits and any storage you keep afterwards still follow Apify's normal rules.

Limits and failure behaviour

• CSW 2.0.2 only, Dublin Core output (csw:Record). Every row uses the same flat shape. Asking for ISO profiles such as gmd:MD_Metadata via typeNames works only if the server supports it, and the rows are still flattened to Dublin Core fields. OGC API Records and CSW 3.0 are not supported.
• Requests are sent as XML POST, which suits POST-only servers such as GeoNetwork. Some catalogs block or rate-limit automated clients, and some lock down CSW entirely. Coverage differs from site to site, so test a small maxRecords first.
• A catalog that is unreachable, returns an HTTP 4xx, returns an OWS ExceptionReport (for example an invalid CQL filter) or returns non-CSW content is skipped with a warning and the next URL is tried. If every catalog fails, the run exits with an error (code 91). A valid query that matches nothing returns 0 rows and no charge.
• If a later page fails, the rows already delivered and billed are kept, and harvesting stops for that catalog.
• recordsPerPage is capped at 100 by the Actor; individual servers may return fewer per page.
• For gated catalogs use bearerToken or extraHeaders. Neither is logged.

Support

Report problems or request a catalog in the Issues tab. Please include the catalog URL and your input.