File-to-Dataset Ingestion & Normalization Pipeline

A mixed-format ingestion layer that converts external structured files into normalized Apify Dataset rows, not merely a profiler or dataset-to-dataset transformer. Returns structured, machine-readable results for repeatable research, monitoring, automation, Apify Tasks, API workflows.

Quick start

Start with this working example and replace the target values with your own:

{
  "documents": [
    {
      "name": "sample",
      "format": "json",
      "content": "[{\"id\":1,\"name\":\"alpha\"},{\"id\":2,\"name\":\"beta\"}]"
    }
  ],
  "maxRows": 100
}

The Actor writes structured results to the default Apify Dataset and can be used from the Store, API, schedules, Tasks, automations and MCP-compatible AI workflows.

Input

• documents — Documents: CSV, JSON, JSONL or XML sources. Use either content or a public HTTP(S) URL.
• maxRows — Maximum output rows: Maximum number of normalized rows to write to the default Dataset across all supplied documents.
• addProvenance — Add provenance fields: Add _sourceIndex and _rowIndex fields so downstream workflows can trace each output row to its input source and position.

All integration and advanced analysis fields are optional. The default example is intentionally runnable without configuring MCP or a previous-run baseline.

Output

The Dataset exposes predictable machine-readable output. Representative fields include dataset, summary.

Pricing

This Actor uses Pay Per Event. The current factory base price is $0.001500 per result event. The Apify Store remains the source of truth for the price and plan/tier details shown to the buyer.

Use cases

• Run the buyer-ready workflows exposed as Apify Tasks without preparing a custom integration first.
• Use the Actor from API or schedules for recurring collection, comparison or monitoring.
• Feed the structured Dataset output into spreadsheets, databases, automations or AI agents.
• Compare repeat runs when the product supports snapshots or previous-run inputs.

Automation and AI

Use the same Actor through Apify API, schedules, public Tasks and the Apify MCP server. Outputs are structured for downstream workflows and AI agents rather than requiring manual copy/paste.

Limitations

• Public websites and upstream APIs can change markup, access rules, rate limits or field availability without notice.
• Fields that are not publicly available are returned as unavailable or omitted rather than fabricated.
• Analytical outputs depend on the quality and coverage of the supplied or collected source data.
• Treat marketplace, reputation, workforce, safety or commercial signals as decision support and validate material decisions against the underlying source evidence.

Detailed documentation

Detailed documentation

A mixed-format ingestion layer that converts external structured files into normalized Apify Dataset rows, not merely a profiler or dataset-to-dataset transformer.

A mixed-format ingestion layer that converts external structured files into normalized Apify Dataset rows, not merely a profiler or dataset-to-dataset transformer.

Convert CSV, JSON, JSONL and XML into normalized Apify Dataset rows with one lightweight Actor. The pipeline accepts inline content or public HTTP(S) URLs, parses supported structured formats, records source provenance, infers a practical field/type schema, and writes ready-to-use rows to the default Dataset.

Why use this Actor

Real automation rarely receives perfectly standardized input. Partners send CSV exports, APIs expose JSON, event streams arrive as JSONL, old systems still publish XML, and research workflows often mix several formats at once. Maintaining separate import scripts for every source creates brittle glue code, duplicated validation and avoidable operational cost.

This Actor provides one deterministic ingestion layer before RAG, CRM, analytics, catalog, job-feed, lead-enrichment or agent workflows. It is intentionally HTTP-first and runs with a small 256 MB default footprint.

Key features

• Parse CSV, JSON, JSONL and XML in the same run.
• Accept inline payloads or public HTTP(S) URLs.
• Reject private, loopback, link-local and otherwise unsafe network targets.
• Combine multiple sources into one normalized Dataset.
• Add _sourceIndex and _rowIndex provenance fields when requested.
• Infer a field/type schema for the accepted output rows.
• Produce per-source reports with parsed and accepted row counts.
• Store PIPELINE_SUMMARY in the default key-value store.
• Cap remote downloads and total output rows to keep runs predictable.
• Work without a browser, login, API key or external LLM.

Output

Each accepted source row is written to the default Apify Dataset. When provenance is enabled, two metadata fields are added:

• _sourceIndex: zero-based source number from the documents input array.
• _rowIndex: zero-based row number within that parsed source.

The PIPELINE_SUMMARY record contains total output rows, successful and failed source counts, an inferred schema and a source-level diagnostic report. This makes the Actor useful both as a converter and as an auditable ingestion gate.

Example normalized Dataset row:

{
  "id": "SKU-42",
  "name": "Example product",
  "price": "19.90",
  "_sourceIndex": 0,
  "_rowIndex": 3
}

Example

Input:

{
  "documents": [
    {
      "name": "catalog",
      "format": "csv",
      "content": "id,name,price\n1,Alpha,10.00\n2,Beta,20.00"
    },
    {
      "name": "status-feed",
      "format": "jsonl",
      "content": "{\"id\":3,\"status\":\"new\"}\n{\"id\":4,\"status\":\"active\"}"
    }
  ],
  "maxRows": 1000,
  "addProvenance": true
}

The run writes four normalized rows and a summary describing both sources and the combined field schema.

Use cases

• RAG ingestion: normalize heterogeneous exports before chunking or embedding.
• CRM migration: convert partner or legacy CSV/JSON files into consistent Dataset rows.
• E-commerce catalogs: ingest supplier feeds and preserve source provenance.
• Job feeds: normalize recurring JSONL, CSV or XML job exports.
• Lead pipelines: standardize inbound lead lists before enrichment and routing.
• Research data: combine structured public files into a single auditable Dataset.
• Agent workflows: give AI agents a predictable Dataset endpoint instead of format-specific parsing logic.
• Migration validation: inspect parsed row counts and schema evidence before downstream delivery.

Pricing

The product is designed as pay per normalized output row. Pricing is tiered by the factory according to observed cost and margin. The Actor does not need a separate artificial start event to create billable value.

Reliability and safety

Remote URLs must resolve to public HTTP(S) targets. Private/local addresses are blocked before fetching. Remote response size is bounded, and the run stops emitting rows after the configured maxRows limit. A malformed source is reported at source level; other valid sources can still be processed. If no source yields usable rows, the run fails explicitly instead of returning fabricated data.

Limitations

• Remote sources must use public HTTP(S).
• Each fetched source is capped at 10 MB.
• Complex XML is normalized conservatively rather than attempting domain-specific mapping.
• Binary formats such as XLSX, Parquet and PDF are outside the initial scope.
• CSV values remain strings unless a downstream transformation explicitly changes them.
• Schema inference describes observed JSON value types; it is not a substitute for a business-domain contract.

The Actor is meant to be a dependable intake layer, not a magical data-cleaning oracle. Humans already invented enough of those.