# IVOA TAP Catalog Extractor (`datamule/ivoa-tap-extractor`) Actor Point at ANY IVOA TAP service (VizieR, Gaia, GAVO, MAST, ESO, CADC and thousands of observatory archives) and run an ADQL query over the Table Access Protocol. Query mode returns dynamic columns per your SELECT; tables mode discovers the full schema. One actor, every astronomical catalog. - **URL**: https://apify.com/datamule/ivoa-tap-extractor.md - **Developed by:** [Datamule](https://apify.com/datamule) (community) - **Categories:** Developer tools, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.28 / 1,000 result records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## IVOA TAP Catalog Extractor **Point at ANY IVOA TAP service and run an ADQL query — one actor, every astronomical catalog.** The [IVOA Table Access Protocol](https://www.ivoa.net/documents/TAP/) (TAP) is the standard query protocol of the Virtual Observatory. VizieR (CDS), Gaia (ESA), GAVO, MAST (STScI), ESO, NOIRLab, CADC and **thousands** of observatory and national data centres all expose the *identical* protocol. So instead of a separate scraper per archive, this actor speaks the protocol itself: give it a TAP service URL and an [ADQL](https://www.ivoa.net/documents/ADQL/) query, and it returns clean tabular rows — no per-site scraper, no host enumeration. ### Two modes #### `query` (default) — run ADQL, get dynamic columns Runs your ADQL query against the service's `/sync` endpoint and flattens the returned VOTable to dataset rows. **The output columns are exactly your query's `SELECT` columns** — shape the result by shaping the query. Numeric VOTable datatypes are typed to numbers, booleans to booleans, and an empty cell becomes `null`. ```` SELECT TOP 100 \* FROM tap\_schema.tables ``` Because `tap_schema` is mandatory in TAP, that query works on essentially every service. Discover a table's columns the same way: ``` SELECT \* FROM tap\_schema.columns WHERE table\_name = 'gaiadr3.gaia\_source' ``` ...then query the real data, e.g. the 20 brightest Gaia sources in a cone: ``` SELECT TOP 20 source\_id, ra, dec, phot\_g\_mean\_mag FROM gaiadr3.gaia\_source WHERE 1 = CONTAINS(POINT('ICRS', ra, dec), CIRCLE('ICRS', 56.75, 24.12, 0.2)) ORDER BY phot\_g\_mean\_mag ASC ```` #### `tables` — discover the whole service Fetches the service's VOSI `/tables` metadata and emits **one row per column** — a machine-readable data dictionary across every schema and table the service exposes (schema, table, column, datatype, unit, UCD, description). Streamed and globally capped, so even VizieR-scale services stop cleanly. ### Input | Field | Description | |---|---| | **endpoint** | A TAP service base URL, e.g. `https://dc.g-vo.org/tap`. The actor appends `/sync` or `/tables`. | | **endpoints** | Optional list — run the same query across several services at once (each row tagged with its `_endpoint`). | | **mode** | `query` (default) or `tables`. | | **adql** | The ADQL query (query mode). Use `TOP N` to limit rows server-side. | | **method** | `GET` (default) or `POST` (for long queries). | | **maxRecords** | Global cap on rows across all endpoints; also sent to the server as `MAXREC`. | | **lang** | Query language, default `ADQL`. | | **userAgent** | Optional User-Agent override. | | **timeoutSecs** | Read timeout, default 120. | | **bearer** / **extraHeaders** | Optional auth for gated deployments — never required for public services, never logged. | ### Output One dataset row per result row (query mode) or per column (tables mode). Query-mode rows carry your SELECT columns at the top level plus `_endpoint`, `_mode`, `_rowIndex`. Tables-mode rows carry `schema_name`, `table_name`, `column_name`, `datatype`, `unit`, `ucd`, `description`. ### Example TAP services | Service | Base URL | |---|---| | GAVO (Heidelberg) | `https://dc.g-vo.org/tap` | | VizieR (CDS) | `http://tapvizier.cds.unistra.fr/TAPVizieR/tap` | | Gaia (ESA) | `https://gea.esac.esa.int/tap-server/tap` | | MAST (STScI) | `https://mast.stsci.edu/vo-tap/api/v0.1/...` | ### Notes - **No fabricated output.** A query that returns nothing yields zero rows. - TAP services can return an in-band error (HTTP 200 with `QUERY_STATUS=ERROR`) for a bad ADQL query — this actor surfaces those as real errors instead of silently returning empty. - Services enforce their own server-side row limits and query timeouts; a very large query may be truncated by the server (reported as `QUERY_STATUS=OVERFLOW`). ### Pricing Pay-per-event: you are charged **per record** returned. Discovery and small queries cost almost nothing; large catalog pulls scale with the number of rows. # Actor input Schema ## `endpoint` (type: `string`): The base URL of ANY IVOA TAP service (Table Access Protocol 1.0 / 1.1). ONE actor speaks to all of them — VizieR / CDS (http://tapvizier.cds.unistra.fr/TAPVizieR/tap), Gaia / ESA (https://gea.esac.esa.int/tap-server/tap), GAVO (https://dc.g-vo.org/tap), MAST, ESO, NOIRLab, CADC and thousands of observatory data centres. Give the base ending in .../tap — the actor appends /sync for queries and /tables for discovery (a full .../tap/sync or .../tap/tables URL is also accepted). Required unless you use "Endpoints" below. For an auth-gated deployment, add a Bearer token or headers below. ## `endpoints` (type: `array`): Optional: run the SAME ADQL query across multiple TAP services at once (mirrors, or several astronomical archives). Each output row is tagged with its _endpoint. Use this OR the single "Endpoint" above (at least one is required). An endpoint that is unreachable, auth-gated, or returns a non-VOTable body is skipped with a warning and the batch continues. ## `mode` (type: `string`): query = run an ADQL query and return one row per result row (columns are your query's SELECT columns). tables = fetch the service's VOSI /tables metadata and return one row per column — a cross-service data dictionary (schema / table / column / datatype / unit / UCD / description). ## `adql` (type: `string`): The ADQL query to run (used in query mode). The output COLUMNS are exactly your SELECT columns, so shape the result by shaping the query. A portable query that works on almost every TAP service (tap_schema is mandatory in TAP): SELECT TOP 100 * FROM tap_schema.tables. Discover a table's columns with: SELECT * FROM tap_schema.columns WHERE table_name = 'your.table'. Use TOP N (ADQL) to limit rows server-side. ## `method` (type: `string`): How to send the query to /sync. GET (default) puts the query in the URL — fine for most queries. POST sends it as a form body — use it for long ADQL queries that would overflow a URL. ## `maxRecords` (type: `integer`): A GLOBAL cap on the number of rows emitted across ALL endpoints (each row is one billable event). Reached mid-endpoint, the run stops deterministically. In query mode it is also sent to the server as MAXREC so the server itself limits the result. Leave empty to emit everything the query / tableset returns. ## `lang` (type: `string`): The TAP query language. ADQL is the universal default and what almost every service speaks. Only change this for the rare service that documents another language (e.g. PQL). ## `userAgent` (type: `string`): Optional override for the request User-Agent. A descriptive User-Agent is sent by default — some data centres throttle or reject a blank / generic UA. Only override if a specific service asks you to identify differently. ## `timeoutSecs` (type: `integer`): How long to wait for the service to answer (default 120). Astronomical queries over large catalogues can be slow; services also enforce their own server-side query timeouts, so a very large query may still be cut off by the server. ## `bearer` (type: `string`): Optional bearer token for an auth-gated TAP deployment (sent as Authorization: Bearer ...). NOT required for public services. Never logged. ## `extraHeaders` (type: `object`): Optional extra HTTP headers as a JSON object, e.g. {"X-Api-Key": "..."} for a gateway-fronted service. Not required for public services. Header values are never logged. ## Actor input object example ```json { "endpoint": "https://dc.g-vo.org/tap", "mode": "query", "adql": "SELECT TOP 100 * FROM tap_schema.tables", "method": "GET", "maxRecords": 1000, "lang": "ADQL" } ```` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "endpoint": "https://dc.g-vo.org/tap", "adql": "SELECT TOP 100 * FROM tap_schema.tables", "maxRecords": 1000 }; // Run the Actor and wait for it to finish const run = await client.actor("datamule/ivoa-tap-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "endpoint": "https://dc.g-vo.org/tap", "adql": "SELECT TOP 100 * FROM tap_schema.tables", "maxRecords": 1000, } # Run the Actor and wait for it to finish run = client.actor("datamule/ivoa-tap-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "endpoint": "https://dc.g-vo.org/tap", "adql": "SELECT TOP 100 * FROM tap_schema.tables", "maxRecords": 1000 }' | apify call datamule/ivoa-tap-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=datamule/ivoa-tap-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "IVOA TAP Catalog Extractor", "description": "Point at ANY IVOA TAP service (VizieR, Gaia, GAVO, MAST, ESO, CADC and thousands of observatory archives) and run an ADQL query over the Table Access Protocol. Query mode returns dynamic columns per your SELECT; tables mode discovers the full schema. One actor, every astronomical catalog.", "version": "0.1", "x-build-id": "xgsXrwJIb8hhx6c8Y" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/datamule~ivoa-tap-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-datamule-ivoa-tap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/datamule~ivoa-tap-extractor/runs": { "post": { "operationId": "runs-sync-datamule-ivoa-tap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/datamule~ivoa-tap-extractor/run-sync": { "post": { "operationId": "run-sync-datamule-ivoa-tap-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "endpoint": { "title": "TAP service base URL", "type": "string", "description": "The base URL of ANY IVOA TAP service (Table Access Protocol 1.0 / 1.1). ONE actor speaks to all of them — VizieR / CDS (http://tapvizier.cds.unistra.fr/TAPVizieR/tap), Gaia / ESA (https://gea.esac.esa.int/tap-server/tap), GAVO (https://dc.g-vo.org/tap), MAST, ESO, NOIRLab, CADC and thousands of observatory data centres. Give the base ending in .../tap — the actor appends /sync for queries and /tables for discovery (a full .../tap/sync or .../tap/tables URL is also accepted). Required unless you use \"Endpoints\" below. For an auth-gated deployment, add a Bearer token or headers below." }, "endpoints": { "title": "Endpoints (run the same query across several)", "type": "array", "description": "Optional: run the SAME ADQL query across multiple TAP services at once (mirrors, or several astronomical archives). Each output row is tagged with its _endpoint. Use this OR the single \"Endpoint\" above (at least one is required). An endpoint that is unreachable, auth-gated, or returns a non-VOTable body is skipped with a warning and the batch continues.", "items": { "type": "string" } }, "mode": { "title": "Mode", "enum": [ "query", "tables" ], "type": "string", "description": "query = run an ADQL query and return one row per result row (columns are your query's SELECT columns). tables = fetch the service's VOSI /tables metadata and return one row per column — a cross-service data dictionary (schema / table / column / datatype / unit / UCD / description).", "default": "query" }, "adql": { "title": "ADQL query", "type": "string", "description": "The ADQL query to run (used in query mode). The output COLUMNS are exactly your SELECT columns, so shape the result by shaping the query. A portable query that works on almost every TAP service (tap_schema is mandatory in TAP): SELECT TOP 100 * FROM tap_schema.tables. Discover a table's columns with: SELECT * FROM tap_schema.columns WHERE table_name = 'your.table'. Use TOP N (ADQL) to limit rows server-side." }, "method": { "title": "HTTP method", "enum": [ "GET", "POST" ], "type": "string", "description": "How to send the query to /sync. GET (default) puts the query in the URL — fine for most queries. POST sends it as a form body — use it for long ADQL queries that would overflow a URL.", "default": "GET" }, "maxRecords": { "title": "Max records (global cap)", "minimum": 1, "type": "integer", "description": "A GLOBAL cap on the number of rows emitted across ALL endpoints (each row is one billable event). Reached mid-endpoint, the run stops deterministically. In query mode it is also sent to the server as MAXREC so the server itself limits the result. Leave empty to emit everything the query / tableset returns." }, "lang": { "title": "Query language", "type": "string", "description": "The TAP query language. ADQL is the universal default and what almost every service speaks. Only change this for the rare service that documents another language (e.g. PQL).", "default": "ADQL" }, "userAgent": { "title": "User-Agent override", "type": "string", "description": "Optional override for the request User-Agent. A descriptive User-Agent is sent by default — some data centres throttle or reject a blank / generic UA. Only override if a specific service asks you to identify differently." }, "timeoutSecs": { "title": "Read timeout (seconds)", "minimum": 1, "maximum": 900, "type": "integer", "description": "How long to wait for the service to answer (default 120). Astronomical queries over large catalogues can be slow; services also enforce their own server-side query timeouts, so a very large query may still be cut off by the server." }, "bearer": { "title": "Bearer token", "type": "string", "description": "Optional bearer token for an auth-gated TAP deployment (sent as Authorization: Bearer ...). NOT required for public services. Never logged." }, "extraHeaders": { "title": "Extra request headers", "type": "object", "description": "Optional extra HTTP headers as a JSON object, e.g. {\"X-Api-Key\": \"...\"} for a gateway-fronted service. Not required for public services. Header values are never logged." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```