Beacon v2 Genomics Extractor

Point at ANY GA4GH Beacon v2 endpoint and pull genomics-discovery data into clean, flat, tabular rows — no per-resource scraper. GA4GH Beacon v2 is the global standard for federated genomics discovery: cancer-genomics resources, national biobanks, reference-population projects and ELIXIR nodes all expose the same wire grammar, and this one Actor speaks it universally.

Why one Actor works everywhere

Every conformant Beacon v2 endpoint answers with the same envelope:

GET {base}/{endpoint}?[filters=…][&params][&skip=&limit=]
  → { meta:{ apiVersion, beaconId, returnedGranularity, returnedSchemas },
      responseSummary:{ exists, numTotalResults },
      response:{ … one of the containers below … } }

The Actor routes on whichever container the beacon returns, so it handles every Beacon entry type with one code path:

Endpoint(s)	Container	One row per…
biosamples, individuals, g_variants, analyses, runs	response.resultSets[].results[]	matched record
cohorts, datasets	response.collections[]	collection
filtering_terms	response.filteringTerms[]	queryable ontology term
entry_types, configuration	response.entryTypes{}	advertised entry type
info, map, service-info	singleton object	the object

The pagination-semantics auto-detect (why it's genuinely universal)

Beacon v2 pages record endpoints with skip + limit, but independent operator families disagree on what skip counts — and a runner that hard-codes either model silently loses or duplicates data on the other:

• ELIXIR / BSC (beacon-python): skip is a raw record offset (skip=1 shifts the window by one record).
• bycon (Progenetix, Cancer Cell Lines): skip is a page index (skip=1 is the disjoint next page; effective offset = skip × limit).

The Actor reads page 0, then — only if more records exist — sends one skip=1 probe and classifies the mode from the returned ids. Every row is then deduped by (resultSetId, id), so an overlap never double-emits and a server that ignores skip terminates cleanly instead of looping. Verified live against all three families.

Input

Field	Type	Notes
sources	array (required)	Beacon v2 base URLs, e.g. https://progenetix.org/beacon, https://beacons.bsc.es/beacon/v2.0.0. Point at the base — don't append an endpoint.
endpoint	select	biosamples (default), individuals, g_variants, analyses, runs, cohorts, datasets, filtering_terms, info, configuration, entry_types, map.
filters	array	CURIE ontology IDs (e.g. NCIT:C7376). Some beacons (bycon family) require a filter for record endpoints — use filtering_terms to discover valid IDs.
requestParameters	object	Extra Beacon query params. For g_variants: { "referenceName": "9", "start": "21000000", "end": "22000000", "variantType": "DEL" }.
granularity	select	record (default), count, boolean, aggregated.
maxRecords	integer	Total row cap across all sources. Default 1000.
bearerToken	string (secret)	Optional OAuth2 bearer for auth-gated beacons. Never required for public beacons; never logged.
extraHeaders	object	Optional extra request headers.

Output

One flat row per record/collection/term, with id, label, name, type, count, full provenance (_source, _beaconId, _apiVersion, _endpoint, _container, _entryType, _beaconSoftware, _resultSetId, _numTotalResults), and the complete original record losslessly in _raw.

Honesty

• A top-level Beacon error envelope (e.g. HTTP 422 no valid query) is surfaced as a real error and skipped-with-warning, never emitted as a fabricated row.
• A valid-but-empty response is a real 0-result, exit 0.
• A run where every source fails exits non-zero (91) so nothing broken ships.

Pricing

Pay-per-event: one record charge per emitted row.

Notes

Beacon v2 exists precisely to share genomics discovery (counts, boolean existence, and record-level metadata over consented cohorts) without moving patient-level genomes. This is a generic protocol client; auth is optional and only needed for beacons that expose protected datasets.