JSONL Toolkit
Pricing
from $0.05 / 1,000 jsonl records
JSONL Toolkit
Clean, validate, deduplicate, and PII-redact JSONL/NDJSON datasets for LLM fine-tuning, batch APIs, and RAG pipelines. Reports data-quality issues and estimates token counts.
Pricing
from $0.05 / 1,000 jsonl records
Rating
0.0
(0)
Developer
林 吴
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
14 hours ago
Last modified
Categories
Security
Limited permissionsShare
Clean and audit JSONL / NDJSON datasets before they hit a fine-tuning job, an LLM batch API, or a RAG pipeline.
What it does
- Parses JSONL, NDJSON, a JSON array, or a single JSON object
- Repairs common LLM formatting issues: Markdown fences, curly quotes, trailing commas
- Optional JSON Schema validation for every record, with field-level error samples
- Optional deduplication by whole record or by a dot-path key (for example
user.email) - Optional PII redaction for emails, phone numbers, card-like numbers, and IPv4 addresses
- Approximate token count (characters / 4, no external tokenizer)
- Data-quality report with parsed, invalid, duplicate, schema-invalid, and output counts
Input
| Field | Type | Description |
|---|---|---|
raw_text | string | Raw JSONL / NDJSON or a JSON array |
source_url | string | Optional public URL to a file (max 30 MB); ignored when raw_text is set |
expected_schema | string | Optional JSON Schema used to validate each record |
drop_invalid | boolean | Drop records that fail schema validation |
dedupe_by | string | Empty = no dedupe; record = whole record; or a dot path such as user.email |
redact_pii | boolean | Replace PII with [EMAIL], [PHONE], [CARD], [IP] |
estimate_tokens | boolean | Add an approximate token count to the summary |
max_records | integer | Input cap, default 20,000 (0 = unlimited) |
Example input:
Output
Each cleaned record is pushed as one item to the default dataset. The run summary is stored in the key-value store under SUMMARY:
Records that fail schema validation are still returned by default, so you can inspect and fix them. Set drop_invalid to remove them.
Pricing
Pay per event. You pay $0.00001 per output record, which is $0.01 per 1,000 records. Failed runs are not charged the per-record fee.
Notes
- PII redaction is regex-based and deterministic; it reduces exposure but is not a compliance guarantee.
- Token counts are approximate (characters / 4), not tokenizer-exact.
- No external AI model is called, so runs are fast and reproducible.
