# jq Helper – transform JSON with jq (`rl1987/jq-helper`) Actor Run jq programs over inline JSON or a linked Apify dataset. Built for chaining into data-processing and enrichment workflows. - **URL**: https://apify.com/rl1987/jq-helper.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Developer tools - **Stats:** 7 total users, 3 monthly users, 76.3% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.01 / successful conversion This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## jq Helper – Transform & Filter JSON with jq (no code) **Run any [jq](https://jqlang.github.io/jq/) program over your JSON and get clean, structured results back.** Reshape, filter, flatten, enrich, and aggregate JSON — from a pasted snippet or straight from another Apify Actor's dataset. The fast, no-code way to add a JSON transformation step to any scraping or data pipeline. > Map · filter · select · flatten · rename keys · merge fields · group & aggregate — all with battle-tested jq 1.8 syntax. --- ### 🧩 What is jq Helper? jq Helper is a tiny, blazing-fast Apify Actor that puts the full power of **jq** — the most popular command-line JSON processor — into a hosted, chainable step. Instead of writing a throwaway script every time a scraper returns messy JSON, drop in a jq filter and let this Actor do the transformation in the cloud. It's built to slot directly into the **Apify ecosystem**: point it at the output dataset of any scraper and use it as a JSON **post-processing, cleanup, and enrichment** stage in your automation. ### ✨ Key features - **Full jq 1.8 support** — the complete language: `select`, `map`, `group_by`, `reduce`, `sort_by`, `unique`, string ops, math, conditionals, and more. - **Two input sources** — paste **inline JSON / JSONL**, or read items from a **linked Apify dataset** by ID (perfect for chaining Actors). - **Two apply modes** — transform **each item** independently (mapping/filtering/enrichment) or process the **whole array at once** (aggregations, grouping, dedupe). - **Pipeline-ready** — writes results to its own dataset so the next Actor or integration (Zapier, Make, Google Sheets, webhooks…) can pick them up. - **Robust by default** — errors on a single bad record are skipped and logged (or fail-fast if you prefer), and results are pushed in efficient batches. - **Zero setup** — no servers, no `jq` install, no dependencies to manage. ### 🚀 Use cases - **Clean up scraped data** — drop nulls, rename fields, normalize values, flatten nested objects. - **Filter datasets** — keep only the records that match your criteria (`select(.price < 100)`). - **Reshape for export** — turn a verbose API response into a tidy, flat table for CSV/Excel/Sheets. - **Enrich records** — derive new fields (domains from emails, full location strings, computed flags). - **Aggregate & summarize** — group by a field, count, average, sum across an entire dataset. - **Glue Actors together** — sit between a scraper and your destination as a no-code transformation step. ### 🔧 How to use 1. Choose your data source: paste JSON into **`jsonInput`**, or set **`datasetId`** to an existing dataset (e.g. an upstream scraper's run). 2. Write your transformation in the **`filter`** field using jq syntax. 3. Pick an **`applyMode`** (`perItem` or `wholeInput`). 4. Run it — transformed records land in the Actor's default dataset, ready to export or pass downstream. #### ⚠️ Per-item vs. whole-input (read this first) This is the one thing to get right: | Mode | What the filter receives | Reach a field with | |------|--------------------------|--------------------| | **`perItem`** (default) | each array element, one at a time, as a single object | `.websiteUrl` ✅ | | **`wholeInput`** | the entire array as one value | `.[].websiteUrl` ✅ | In **per-item** mode the array is iterated for you, so **don't** prefix with `.[]` — `.[].websiteUrl` will fail with *"Cannot index string with string"*. Use a leading `.[]` only in **whole-input** mode. ### 📥 Input | Field | Type | Description | |-------|------|-------------| | `filter` | string (required) | The jq program. jq 1.8 syntax. May emit 0, 1, or many values per input — each becomes one output item. | | `applyMode` | `perItem` \| `wholeInput` | `perItem` (default): run on each element. `wholeInput`: feed the whole array as a single value. | | `datasetId` | string | Apify dataset to read items from (provide either this **or** `jsonInput`, not both). **Map this field to chain Actors in an integration.** Paid: $0.01 per successful run. | | `jsonInput` | string | Raw JSON (object/array/scalar) or JSONL (one value per line). Free — for prototyping. | | `wrapKey` | string | Non-object outputs are wrapped as `{ : value }` (dataset items must be objects). Default `value`. | | `failOnError` | boolean | `true` aborts on any jq runtime error; default `false` skips + logs the bad record. | ### 📤 Output Each value emitted by your filter becomes one item in the Actor's default dataset. Object outputs are stored as-is (their keys become columns); scalars and arrays are wrapped under `wrapKey`. Export the dataset as **JSON, CSV, Excel, HTML, or RSS**, or hand it to the next step in your workflow. ### 💡 Examples **Reshape & build a location string (per-item):** ```jq {fullName, currentEmployer, location: ([.city, .state, .country] | map(select(.)) | join(", "))} ```` **Keep only verified records (per-item):** ```jq select(.clearVerified == true) ``` **Pull one field as a clean column (per-item):** ```jq {websiteUrl} ``` **Group and summarize the whole dataset (whole-input):** ```jq group_by(.state) | map({state: .[0].state, count: length}) | .[] ``` **Deduplicate by a key (whole-input):** ```jq unique_by(.profileUrl) | .[] ``` ### 🔗 Chaining after another Actor (integrations) Run **jq Helper** automatically whenever an upstream scraper finishes, and pass it that run's dataset. **In the Console (no code):** 1. Open the **upstream** Actor's run config → **Integrations** tab → **Connect Actor or Task**. 2. Choose **jq Helper** to run on success (event `ACTOR.RUN.SUCCEEDED`). 3. In jq Helper's input, set the **`datasetId`** field to the upstream run's dataset using the variable: ``` {{resource.defaultDatasetId}} ``` Leave `jsonInput` empty (the two sources are mutually exclusive), then set your `filter` and `applyMode`. The transformed items land in jq Helper's own dataset — ready for the next Actor, a webhook, or an export to Sheets/Make/Zapier. > Other variables from the same run you can drop into string fields: `{{resource.id}}` (run ID), `{{resource.actId}}`, `{{resource.defaultKeyValueStoreId}}`, `{{resource.status}}`. **In code (parent orchestrator):** ```python run = await Actor.call(actor_id='you/scraper', run_input={...}) await Actor.call( actor_id='rl1987/jq-helper', run_input={ 'filter': '.websiteUrl', 'applyMode': 'perItem', 'datasetId': run.default_dataset_id, }, ) ``` > Chaining always uses the `datasetId` path, so each chained run is a paid **$0.01** conversion. Inline `jsonInput` stays free for prototyping. ### ❓ FAQ **Do I need to install jq?** No. The jq engine is bundled — just write filters. **Which jq version?** jq 1.8, the full language. **My filter returns a string/number — why is it `{"value": ...}`?** Dataset items must be JSON objects, so scalars and arrays are wrapped. Set `wrapKey` to rename the key, or emit an object (e.g. `{websiteUrl}`) for named columns. **How do I aggregate across all records?** Use `applyMode: wholeInput` and reach elements with `.[]`. **What if one record is malformed?** By default it's skipped and logged; set `failOnError: true` to stop the run instead. ### 🛠️ Local development ```bash pip install -r requirements.txt apify run # reads storage/key_value_stores/default/INPUT.json ``` *** Built with the [Apify SDK for Python](https://docs.apify.com/sdk/python/) and the [`jq` bindings](https://github.com/mwilliamson/jq.py) (bundles libjq — no system `jq` required). > **Disclaimer:** This is an independent, unofficial project. It is **not affiliated with, endorsed by, or otherwise associated with the [jq project](https://jqlang.github.io/jq/) or any of its developers.** "jq" is used here only to describe the JSON-processing language this Actor runs; all rights to jq belong to its respective authors. # Actor input Schema ## `filter` (type: `string`): The jq program to run (standard jq 1.8 syntax). Compiled once and applied per the apply mode below. A filter may emit zero, one, or many values per input — each becomes one output item; non-object values are wrapped under the scalar wrap key. IMPORTANT — do NOT prefix with `.[]` in per-item mode: each item is already a single object, so `.websiteUrl` is correct and `.[].websiteUrl` errors with 'Cannot index string with string'. Use a leading `.[]` only in whole-input mode, where the whole array is fed in at once. ## `applyMode` (type: `string`): How the filter maps over the input. Per-item (default): the array is iterated for you and the filter runs on each element as a single object — write `.field`, NOT `.[].field`. Best for mapping/filtering/enriching rows. Whole-input: the entire array is fed to the filter as one value, so use `.[]` to reach elements. Best for group\_by, reduce, sort, unique, and other cross-record aggregations. ## `jsonInput` (type: `string`): Raw JSON (object, array, or scalar) or JSONL (one JSON value per line) to transform. Provide either this or a Dataset ID, not both. In per-item mode, a top-level array is iterated element by element; a single object/scalar is treated as one item. Runs on inline input are FREE — prototype your jq snippets here at no cost. ## `datasetId` (type: `string`): ID of an Apify dataset to read items from (e.g. the default dataset of an upstream actor run). Provide either this or inline JSON, not both. This is the field to map when wiring this actor into an integration/workflow. Processing a dataset is a paid conversion ($0.01 per successful run, regardless of size). ## `wrapKey` (type: `string`): Dataset items must be JSON objects. When the filter emits a non-object value (string, number, array, boolean, null), it is wrapped as { : value }. ## `failOnError` (type: `boolean`): If enabled, a jq runtime error on any input aborts the run. If disabled (default), the erroring input is skipped, logged as a warning, and counted in the run summary. ## Actor input object example ```json { "filter": "{name, isAdult: (.age >= 18)}", "applyMode": "perItem", "jsonInput": "[{\"name\": \"alice\", \"age\": 30}, {\"name\": \"bob\", \"age\": 16}]", "wrapKey": "value", "failOnError": false } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "filter": "{name, isAdult: (.age >= 18)}", "jsonInput": "[{\"name\": \"alice\", \"age\": 30}, {\"name\": \"bob\", \"age\": 16}]" }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/jq-helper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "filter": "{name, isAdult: (.age >= 18)}", "jsonInput": "[{\"name\": \"alice\", \"age\": 30}, {\"name\": \"bob\", \"age\": 16}]", } # Run the Actor and wait for it to finish run = client.actor("rl1987/jq-helper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "filter": "{name, isAdult: (.age >= 18)}", "jsonInput": "[{\\"name\\": \\"alice\\", \\"age\\": 30}, {\\"name\\": \\"bob\\", \\"age\\": 16}]" }' | apify call rl1987/jq-helper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rl1987/jq-helper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "jq Helper – transform JSON with jq", "description": "Run jq programs over inline JSON or a linked Apify dataset. Built for chaining into data-processing and enrichment workflows.", "version": "0.1", "x-build-id": "H3FyRj9RqcKGHX287" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/rl1987~jq-helper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-rl1987-jq-helper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/rl1987~jq-helper/runs": { "post": { "operationId": "runs-sync-rl1987-jq-helper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/rl1987~jq-helper/run-sync": { "post": { "operationId": "run-sync-rl1987-jq-helper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "filter" ], "properties": { "filter": { "title": "jq filter", "type": "string", "description": "The jq program to run (standard jq 1.8 syntax). Compiled once and applied per the apply mode below. A filter may emit zero, one, or many values per input — each becomes one output item; non-object values are wrapped under the scalar wrap key. IMPORTANT — do NOT prefix with `.[]` in per-item mode: each item is already a single object, so `.websiteUrl` is correct and `.[].websiteUrl` errors with 'Cannot index string with string'. Use a leading `.[]` only in whole-input mode, where the whole array is fed in at once.", "default": "." }, "applyMode": { "title": "Apply mode", "enum": [ "perItem", "wholeInput" ], "type": "string", "description": "How the filter maps over the input. Per-item (default): the array is iterated for you and the filter runs on each element as a single object — write `.field`, NOT `.[].field`. Best for mapping/filtering/enriching rows. Whole-input: the entire array is fed to the filter as one value, so use `.[]` to reach elements. Best for group_by, reduce, sort, unique, and other cross-record aggregations.", "default": "perItem" }, "jsonInput": { "title": "Inline JSON / JSONL", "type": "string", "description": "Raw JSON (object, array, or scalar) or JSONL (one JSON value per line) to transform. Provide either this or a Dataset ID, not both. In per-item mode, a top-level array is iterated element by element; a single object/scalar is treated as one item. Runs on inline input are FREE — prototype your jq snippets here at no cost." }, "datasetId": { "title": "Dataset ID", "type": "string", "description": "ID of an Apify dataset to read items from (e.g. the default dataset of an upstream actor run). Provide either this or inline JSON, not both. This is the field to map when wiring this actor into an integration/workflow. Processing a dataset is a paid conversion ($0.01 per successful run, regardless of size)." }, "wrapKey": { "title": "Scalar wrap key", "type": "string", "description": "Dataset items must be JSON objects. When the filter emits a non-object value (string, number, array, boolean, null), it is wrapped as { : value }.", "default": "value" }, "failOnError": { "title": "Fail on filter error", "type": "boolean", "description": "If enabled, a jq runtime error on any input aborts the run. If disabled (default), the erroring input is skipped, logged as a warning, and counted in the run summary.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```