CSV migration verifier — prototype

A small offline evidence generator for comparing two CSV exports across renamed columns. The offline CLI uses Python 3.10+ with no runtime dependencies, network requests, accounts, telemetry, database, or installation required. An optional, separately invoked Apify SDK adapter is also included; it is the only network-capable path. All included records are synthetic.

This is a prototype for testing whether migration handover reports solve a useful customer problem. It is not evidence of unique demand or a production certification tool. The Actor has been deployed on Apify and passed a synthetic smoke test. Existing free diff tools are substitutes; validate the workflow before building more.

Run locally

From this directory:

python -m csv_migration_verifier examples/source.csv examples/target.csv \
  --config examples/config.json --out my-first-report

Open my-first-report/report.html in a browser. Keep the three output files together. The directory must not already exist; each run uses a fresh directory to prevent accidental overwrites.

Exit codes: 0 = equal under configured rules; 1 = differences found; 2 = invalid input/configuration or execution/output error. A result of 1 is a successful comparison, not a software crash.

The example returns 1: three records on each side, two matched keys, one changed record, one missing, one added and one cell accepted by tolerance. Exact balance totals are 42.50 and 45.004, delta 2.504.

python -m unittest discover -s tests -v

Configuration

{
  "mapping": {"account": "account_id", "region": "territory", "name": "display_name", "balance": "amount"},
  "keys": ["account", "region"],
  "tolerances": {"balance": "0.01"},
  "totals": ["balance"],
  "null_token": null
}

• mapping is required. Each source header maps to exactly one target header. Keys, tolerances and totals refer to source column names. Target mappings must be unique. Extra unmapped columns are deliberately ignored for comparisons, but duplicate or empty headers anywhere invalidate the input.
• keys is a required, nonempty ordered list. Composite keys are tuples, not concatenated strings. Empty/null key components and duplicate keys invalidate the entire comparison, even if duplicate rows are otherwise identical. Keys are always exact strings, never tolerant numbers.
• tolerances is optional. Each value must be a JSON string containing a nonnegative finite decimal. A changed numeric field is accepted when abs(target - source) <= tolerance. There is no relative tolerance or rounding. Set "0" to accept numerically equal formatting variants such as 1.0 and 1.00.
• totals is optional. Exact decimal totals across all records on each side, including added/missing records. Delta is target minus source. Totals are informational, not an additional equality test. A column listed only in totals is still compared as an exact string.
• null_token is optional and defaults to JSON null (disabled). If provided, it must be a nonempty string; cells exactly matching it become semantic null. "NULL" and empty string are distinct when the token is enabled. There is no escape to represent the literal token separately, so choose one that does not occur as real text.
• Unknown options, duplicate JSON object keys, missing mapped headers, and invalid configuration values fail closed.

Exact semantics

CSV and strings

UTF-8 with optional leading BOM; comma delimiter; double-quoted fields; doubled quotes inside quoted fields; LF, CRLF or CR line endings. Quoted embedded newlines are supported. Quotes in bare fields and characters after a closing quote are rejected. Wrong field counts, blank records, malformed quoting, invalid UTF-8, NUL characters and missing/empty/duplicate headers invalidate the run. A trailing line ending is fine.

There is no trimming, case folding, type inference, date parsing, Unicode normalization, header auto-matching or delimiter guessing. IDs such as 001 remain distinct from 1. Whitespace is meaningful. Unquoted empty and quoted empty both mean the empty string: CSV does not retain a null-vs-empty distinction by itself. Missing fields are structural errors, not empty values.

Header-only inputs are accepted as empty datasets. Two empty datasets can be equal; this does not establish that exports are complete or a migration succeeded. The HTML report explicitly flags any empty dataset. Verify the expected export scope/record count independently.

Numbers

Only columns explicitly selected for tolerance or totals are numeric. Every row in each such column must contain a valid finite decimal, including unmatched rows and textually identical values. Null/empty numeric cells invalidate the whole run; no null skipping or zero substitution.

Accepted grammar: optional sign, ASCII digits with optional decimal point (or leading dot), optional scientific exponent. Whitespace, thousands separators, underscores, NaN and Infinity are rejected. Numeric strings are limited to 1000 characters and Decimal's stored exponent to ±1000. Arithmetic uses a sufficiently large Decimal context for these bounds and record limits, including cancellation and exact sums. Negative zero is numerically equal to zero only with a tolerance rule; default string comparison distinguishes them.

Output and invalid runs

• report.html: self-contained, script-free handover view with counts, changed fields, missing/added records, errors, exact totals, hashes and rules. Displays at most 100 items per exception category, plus 100 validation errors, with exact category totals. Links to complete sibling files; no CDN or remote resources.
• report.json: authoritative machine-readable evidence containing all exceptions/errors, complete normalized config, version, hashes, input headers and counts. JSON null is distinct from "". The config hash is over normalized compact UTF-8 JSON with sorted object keys, not original config-file bytes.
• exceptions.csv: review copy with UTF-8 BOM and CSV quoting. Every data cell other than the fixed category is apostrophe-prefixed JSON, preventing formula execution and preserving string-type IDs during ordinary spreadsheet import. This intentionally changes presentation; use JSON for lossless programmatic work. Do not remove these prefixes before opening untrusted values in a spreadsheet; import/export applications can have different behavior.
• missing means present in source but absent from target. added means present in target but absent from source. changed contains one entry per different mapped cell. Keys and columns are sorted deterministically; row and column order do not affect reconciliation. Hashes intentionally change when the raw files change. unchanged_records includes records accepted by tolerance; tolerated_cells counts textually different cells accepted by their numeric rule.
• Any validation error invalidates the entire run. counts and totals are JSON null, and exceptions are empty. Never interpret empty exceptions as equality without checking status. Input parsed_records counts structurally parsed data records observed; valid_records counts valid-width/key/numeric rows including duplicates; unique_keys counts retained valid keys. complete says only whether parsing reached EOF, not whether validation passed. Early syntax/header/read failures may have zero/absent counts. Counts may be partial on resource limits. No diff is computed from a partially accepted file.

All HTML fields are escaped. A restrictive Content Security Policy disallows scripts and external resources. Hashes fingerprint bytes; they do not authenticate provenance, sign results, or prove snapshot completeness. Inputs are read independently, so obtain consistent source/target export snapshots yourself. Reports contain the actual compared data: keep them private and handle them like the inputs. No encryption, access control, redaction or secure deletion is provided.

Bounded prototype limits

• At most 10 MiB per CSV, 100,000 data records per input, 1 MiB configuration JSON
• Python CSV parser's default field-size limit (normally 131,072 characters); oversized fields fail closed
• Files and rows are held in memory; no streaming/external-sort mode
• No automatic schema/type discovery, database connections, remote URLs, archives, Excel files, report signing, joins, fuzzy matches, migration writes, or correction actions
• Outputs are fresh local files; an output-write error can leave a partial directory. Treat a non-completed command as failed and use a new directory after fixing the cause
• Test coverage is synthetic. Runtime/install checks were performed with the container's Python; an Apify synthetic smoke test passed; no Windows/Excel acceptance testing or production security audit has been performed

The limits prevent accidental unbounded local workloads; this is not a hardened multi-tenant service. Do not expose the Python API to untrusted callers or increase limits casually.

Local Apify-compatible file convention

The adapter reads and writes local files only, using Apify's documented default key-value store layout. It does not require the Apify SDK, CLI, token or account; it does not invoke any API and is not a deployable cloud integration.

mkdir -p demo-storage/key_value_stores/default
python - <<'PY'
import json
from pathlib import Path
payload = {
    "sourceCsv": Path("examples/source.csv").read_text(encoding="utf-8"),
    "targetCsv": Path("examples/target.csv").read_text(encoding="utf-8"),
    "config": json.loads(Path("examples/config.json").read_text())
}
Path("demo-storage/key_value_stores/default/INPUT.json").write_text(json.dumps(payload), encoding="utf-8")
PY
python -m csv_migration_verifier.apify_local --storage demo-storage

Alternatively set APIFY_LOCAL_STORAGE_DIR; otherwise the root is storage. Only key_value_stores/default is supported. INPUT must contain exactly sourceCsv, targetCsv, config, with both CSVs inline strings. The input JSON limit is 22 MiB (escaping can make a CSV too large for this envelope). Core per-CSV limits still apply. The adapter creates OUTPUT.json, REPORT.html, EXCEPTIONS.csv beside INPUT and refuses to replace any existing output. Temporary CSV files are removed when processing finishes; local INPUT and output files remain. Malformed wrapper inputs fail before generating a comparison report. It refuses cloud mode when APIFY_IS_AT_HOME is 1 or true.

This local adapter remains separate from the optional SDK adapter below. Uploading/running on Apify sends the supplied CSV data to Apify, with cloud retention, access and billing considerations. A deployment and synthetic smoke test have been completed on Apify; no paid service has been purchased.

Storage-convention reference: official Apify key-value store documentation, checked 2026-09-30.

Optional Apify SDK / Docker scaffold

csv_migration_verifier.apify_actor uses the real SDK interface: async with Actor, Actor.get_input(), and Actor.set_value(). It accepts the same inline payload and applies the same core rules. It writes three default key-value store records: OUTPUT (JSON object), REPORT.html (HTML) and EXCEPTIONS.csv (CSV bytes). After those records are saved, it writes one default-dataset summary row containing only status and counts, with no raw input values. Invalid comparisons retain status: "invalid" and null counts. It never fetches remote CSV URLs, touches databases, uses proxies, or calls custom billing/charging events.

.actor/actor.json, input/output schemas, Dockerfile, and requirements-apify.txt provide the platform packaging scaffold. Docker uses Python 3.12 and the optional Apify SDK pinned to 3.4.1. The offline CLI never imports or requires it. Building the image downloads an official Python base image and SDK dependencies; unlike running the offline CLI, this is not an offline operation.

Verification boundary: 38 distinct standard-library tests pass, including a stub SDK checking lifecycle, output keys/types, safe HTML, invalid-run persistence and persistence-error propagation. JSON files and referenced paths are checked. An independent local execution of the actual SDK adapter also passed with Apify 3.4.1, apify-client 2.5.1, Crawlee 1.10.3 and Python 3.12.14: synthetic fixtures, fresh local storage, tokens unset and APIFY_IS_AT_HOME=false; the SDK wrote OUTPUT, REPORT.html and EXCEPTIONS.csv and exited 0. The SDK dependencies were installed in an isolated temporary location for that check. Docker was unavailable for the local checks. An Apify platform build and a synthetic cloud smoke test have since passed. This remains a prototype; comprehensive platform acceptance and billing tests have not been performed. The direct SDK dependency is pinned; lock and review transitive dependencies before production.

For an approved later platform test, use synthetic input first. Successful Actor execution means the report was generated; read OUTPUT.status (equal, different, invalid) to determine the comparison result. Differences and CSV validation errors are persisted as evidence, not raised as infrastructure failures. Malformed wrapper inputs or output persistence failures fail execution. Partial records may remain if a cloud write fails; a failed run is not a completed report. The SDK receives already-parsed input objects, so duplicate JSON-key detection applies to local raw JSON files, not bytes consumed by the platform before this adapter runs.

The SDK may use network services when configured for cloud storage. Running it in the cloud transmits INPUT and output records to Apify; outputs contain actual values. No application-level redaction or additional access controls are provided. No custom charging code is included, but ordinary platform compute/storage costs can still apply. Account connection, uploads, deployment, publishing and billing configuration require separate authorization.

Official references: Actor definition, input schema, output schema. Checked 2026-09-30.

Files

• csv_migration_verifier/core.py: validation, comparison and provenance
• csv_migration_verifier/reports.py: safe HTML/JSON/CSV reports
• csv_migration_verifier/__main__.py: CLI
• csv_migration_verifier/apify_local.py: optional local storage adapter
• csv_migration_verifier/apify_actor.py: optional SDK persistence adapter
• .actor/, Dockerfile, requirements-apify.txt: cloud packaging used in the tested deployment
• examples/: synthetic CSVs and config
• example-output/: generated synthetic report
• tests/: standard-library unit tests

Optional packaging metadata is provided, but normal use does not install or download anything. Published use is governed by the Apify Standard Actor Contract.