# JSONL Toolkit (`flyerlin/jsonl-toolkit`) Actor

Clean, validate, deduplicate, and PII-redact JSONL/NDJSON datasets for LLM fine-tuning, batch APIs, and RAG pipelines. Reports data-quality issues and estimates token counts.

- **URL**: https://apify.com/flyerlin/jsonl-toolkit.md
- **Developed by:** [林 吴](https://apify.com/flyerlin) (community)
- **Categories:** Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.05 / 1,000 jsonl records

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## JSONL Toolkit

Clean and audit JSONL / NDJSON datasets before they hit a fine-tuning job, an LLM batch API, or a RAG pipeline.

### What it does

- Parses JSONL, NDJSON, a JSON array, or a single JSON object
- Repairs common LLM formatting issues: Markdown fences, curly quotes, trailing commas
- Optional JSON Schema validation for every record, with field-level error samples
- Optional deduplication by whole record or by a dot-path key (for example `user.email`)
- Optional PII redaction for emails, phone numbers, card-like numbers, and IPv4 addresses
- Approximate token count (characters / 4, no external tokenizer)
- Data-quality report with parsed, invalid, duplicate, schema-invalid, and output counts

### Input

| Field | Type | Description |
| --- | --- | --- |
| `raw_text` | string | Raw JSONL / NDJSON or a JSON array |
| `source_url` | string | Optional public URL to a file (max 30 MB); ignored when `raw_text` is set |
| `expected_schema` | string | Optional JSON Schema used to validate each record |
| `drop_invalid` | boolean | Drop records that fail schema validation |
| `dedupe_by` | string | Empty = no dedupe; `record` = whole record; or a dot path such as `user.email` |
| `redact_pii` | boolean | Replace PII with `[EMAIL]`, `[PHONE]`, `[CARD]`, `[IP]` |
| `estimate_tokens` | boolean | Add an approximate token count to the summary |
| `max_records` | integer | Input cap, default 20,000 (0 = unlimited) |

Example input:

```json
{
  "raw_text": "{\"user\": {\"email\": \"lin@example.com\"}, \"score\": 42}\n{\"user\": {\"email\": \"lin@example.com\"}, \"score\": 42}",
  "dedupe_by": "user.email",
  "redact_pii": true
}
```

### Output

Each cleaned record is pushed as one item to the default dataset. The run summary is stored in the key-value store under `SUMMARY`:

```json
{
  "records_parsed": 2,
  "invalid_lines": 0,
  "duplicates_removed": 1,
  "schema_invalid": 0,
  "pii_redactions": 1,
  "output_records": 1,
  "estimated_tokens": 12
}
```

Records that fail schema validation are still returned by default, so you can inspect and fix them. Set `drop_invalid` to remove them.

### Pricing

Pay per event. You pay **$0.00001 per output record**, which is **$0.01 per 1,000 records**. Failed runs are not charged the per-record fee.

### Notes

- PII redaction is regex-based and deterministic; it reduces exposure but is not a compliance guarantee.
- Token counts are approximate (characters / 4), not tokenizer-exact.
- No external AI model is called, so runs are fast and reproducible.

# Actor input Schema

## `raw_text` (type: `string`):

Raw JSONL / NDJSON content, or a JSON array. Markdown fences, curly quotes, and trailing commas are repaired automatically.

## `source_url` (type: `string`):

Optional public URL to a .jsonl, .ndjson, or .json file (max 30 MB). Ignored when raw_text is provided.

## `expected_schema` (type: `string`):

Optional JSON Schema (as a JSON string) used to validate every record.

## `drop_invalid` (type: `boolean`):

When enabled, records that fail schema validation are excluded from the output.

## `dedupe_by` (type: `string`):

Leave empty to keep duplicates. Use 'record' to dedupe whole records, or a dot path such as 'user.email' to dedupe by a field.

## `redact_pii` (type: `boolean`):

Replace emails, phone numbers, card-like numbers, and IPv4 addresses with placeholder tokens.

## `estimate_tokens` (type: `boolean`):

Add an approximate token count (characters / 4) to the run summary. No external tokenizer is used.

## `max_records` (type: `integer`):

Maximum number of input records to process (default 20,000, set 0 for unlimited).

## Actor input object example

```json
{
  "raw_text": "{\"user\": {\"email\": \"lin@example.com\", \"name\": \"Lin\"}, \"score\": 42}\n{\"user\": {\"email\": \"lin@example.com\", \"name\": \"Lin\"}, \"score\": 42}\n{\"user\": {\"name\": \"Wu\"}, \"score\": \"high\",}",
  "drop_invalid": false,
  "redact_pii": false,
  "estimate_tokens": true,
  "max_records": 20000
}
```

# Actor output Schema

## `results` (type: `string`):

One dataset item per parsed and cleaned record.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "raw_text": `{"user": {"email": "lin@example.com", "name": "Lin"}, "score": 42}
{"user": {"email": "lin@example.com", "name": "Lin"}, "score": 42}
{"user": {"name": "Wu"}, "score": "high",}`
};

// Run the Actor and wait for it to finish
const run = await client.actor("flyerlin/jsonl-toolkit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "raw_text": """{\"user\": {\"email\": \"lin@example.com\", \"name\": \"Lin\"}, \"score\": 42}
{\"user\": {\"email\": \"lin@example.com\", \"name\": \"Lin\"}, \"score\": 42}
{\"user\": {\"name\": \"Wu\"}, \"score\": \"high\",}""" }

# Run the Actor and wait for it to finish
run = client.actor("flyerlin/jsonl-toolkit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "raw_text": "{\\"user\\": {\\"email\\": \\"lin@example.com\\", \\"name\\": \\"Lin\\"}, \\"score\\": 42}\\n{\\"user\\": {\\"email\\": \\"lin@example.com\\", \\"name\\": \\"Lin\\"}, \\"score\\": 42}\\n{\\"user\\": {\\"name\\": \\"Wu\\"}, \\"score\\": \\"high\\",}"
}' |
apify call flyerlin/jsonl-toolkit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,flyerlin/jsonl-toolkit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/99Simpbi7DEJLhfxm/builds/ufcT1BRzre5EDmHoN/openapi.json