# Survey Data Cleaner (CSV, SPSS .sav, Qualtrics) (`madrasco/survey-data-cleaner`) Actor

Cleans a survey export (CSV, SPSS .sav or Qualtrics CSV) with fixed, documented rules and returns the cleaned file, a reconstructed codebook, and a row-level report of every change. Speeders, straight-liners and out-of-range answers are flagged, never deleted.

- **URL**: https://apify.com/madrasco/survey-data-cleaner.md
- **Developed by:** [Madrasco](https://apify.com/madrasco) (community)
- **Categories:** Business, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Survey Data Cleaner (CSV, SPSS .sav, Qualtrics)

Clean a **survey export** with fixed, documented rules and get back files you can check: the **cleaned data**, a **reconstructed codebook**, and a **row-level change report** listing every value that was changed (row, respondent ID, column, old value, new value, rule). Quality problems such as **speeders, straight-lining and out-of-range answers are flagged, never deleted**, so the decision stays with you.

Works with:

- **CSV / TSV** (comma, semicolon, tab or pipe; UTF-8, UTF-16 or Windows-1252),
- **Qualtrics** CSV/TSV exports (the question-text and ImportId rows under the header are detected, removed and kept as variable labels),
- **SPSS .sav** files (variable labels, value labels and user-missing values are read; a cleaned `.sav` is written back with its labels).

### What it changes

| Rule | What it does |
|---|---|
| `qualtrics-header-row` | Removes the Qualtrics question-text and ImportId rows under the header (the question text becomes the variable label) |
| `whitespace` | Trims leading/trailing spaces, collapses repeated spaces and non-breaking spaces, removes invisible characters (line breaks inside answers are kept) |
| `missing-code` | Replaces missing-value codes (NA, N/A, NULL, ., -99, -999 ... configurable) with an empty cell |
| `spss-user-missing` | Replaces values declared user-missing in the SPSS file with an empty cell (the value label, e.g. "Refused", is kept in the report) |
| `duplicate-id` | Removes later rows with a respondent ID already seen (the first row is kept) |
| `number-format` | In numeric columns, writes numbers plainly: 1,234 → 1234, 3.0 → 3, +5 → 5 |
| `date-format` | Rewrites unambiguous YYYY/MM/DD dates as YYYY-MM-DD |
| `category-case` | In columns with 30 or fewer distinct answers, unifies spelling variants that differ only by case or spacing to the most common one |

Every change except the Qualtrics header-row removal (`qualtrics-header-row`) can be switched off in the input (the Qualtrics extra header rows are always removed, because they are not answers). Flags (below) only mark rows; they never change your data.

### What it flags (nothing is deleted)

| Flag | Meaning |
|---|---|
| `speeder` | Completion time below a fraction (default 0.33) of the median duration |
| `straight-lining` | The same answer to every item of a question grid (columns named like Q5\_1, Q5\_2, ...; at least 5 items by default) |
| `out-of-range` | Outside a range you give (e.g. `{"Q4": {"min": 1, "max": 5}}`, patterns like `Q3_*` allowed), or not one of the labelled codes of an SPSS variable |
| `not-a-number` | Text in an otherwise numeric column |
| `incomplete` | Qualtrics response with Finished = 0 or Progress below 100 |
| `duplicate-row` | Row identical to an earlier one when there is no respondent ID column |

Flags go to `FLAGS.csv`, the dataset, and a `qc_flags` column in the cleaned file.

### Output

In the run's key-value store:

- `CLEANED.csv` (UTF-8 with BOM so Excel opens it correctly) and, for SPSS input, `CLEANED.sav`;
- `CODEBOOK.csv` and `CODEBOOK.json`: per variable, the label, detected type, non-missing and missing counts, distinct values, range, value labels or observed answers with counts, missing codes found, SPSS format;
- `CHANGES.csv`: every change; `FLAGS.csv`: every flag;
- `REPORT.html`: a readable report of all of the above;
- `OUTPUT`: a JSON summary (rows in/out, counts by rule).

The dataset holds one item per change and per flag (up to 5,000 by default; the CSV files always hold all of them). **Row numbers** refer to the original file: for CSV, the spreadsheet row (header = row 1); for SPSS, the case number.

### Input

Give a **File URL** (a direct download link that works without a login), or upload the file to a key-value store in your Apify account (Storage tab) and **pick that store**; the first `.csv`, `.tsv`, `.txt` or `.sav` record is used unless you name its key. Everything else is optional: the respondent ID column, missing-value codes, allowed ranges, duration column, speeder threshold and grid size.

With no file at all, the actor cleans a small built-in **demo file** (a made-up Qualtrics export with 30 responses) so you can see the output; demo runs are not charged.

### Limits

- The rules are mechanical. They don't know your study design: check the change report before analysis, and remove codes such as -99 from the missing list if they are real answers in your data.
- Standardizing missing codes merges different kinds of missing (e.g. "refused" and "don't know") into empty cells; the change report and the codebook keep the original codes. Switch the rule off if you need them distinct.
- Qualtrics value labels can't be recovered from a single CSV export; the codebook lists the observed answers instead.
- Dates in day/month or month/day order are left as they are. Excel (.xlsx) and Stata files are not read yet.
- One file per run; files up to 100 MB by default (raise memory for large files).

### Your data

The file is read only during the run. Madrasco does not receive, keep or see a copy; the outputs are stored in your own Apify run storage and follow your account's retention settings. Survey data often contains personal information: only process data you are allowed to process, and consider removing names and contact details before upload.

### Support

Open an issue in the **Issues** tab. This actor is built and maintained by Madrasco with AI assistance; a human owner can be reached on request.

# Actor input Schema

## `fileUrl` (type: `string`):

Direct download link to a .csv, .tsv or SPSS .sav survey export (Qualtrics CSV/TSV exports are detected automatically). Must work without a login. Leave empty if you pick a key-value store below. If both are empty, a small built-in synthetic Qualtrics demo file is cleaned (free) so you can see the output.

## `keyValueStoreId` (type: `string`):

Or pick a key-value store in your Apify account where you uploaded the file (Storage tab). The first .csv, .tsv, .txt or .sav record is used unless you give its key below.

## `fileKey` (type: `string`):

Optional: the key of the file in the key-value store above.

## `format` (type: `string`):

Normally detected from the file itself.

## `idColumn` (type: `string`):

Column that identifies a respondent, used to remove repeated submissions. If empty, a column named ResponseId, respondent\_id, id, caseid or similar is used when present.

## `dedupeById` (type: `boolean`):

Keep the first row for each respondent ID and remove later rows with the same ID (each removal is in the change report). Without an ID column, identical rows are flagged instead.

## `trimWhitespace` (type: `boolean`):

Trims leading/trailing spaces, collapses repeated and non-breaking spaces and removes invisible characters in every cell (line breaks inside answers are kept).

## `standardizeMissing` (type: `boolean`):

Replace missing-value codes (the list below, and values declared user-missing in an SPSS file) with empty cells. The codebook counts which codes were found in each column.

## `missingCodes` (type: `array`):

Whole-cell values treated as missing (compared without regard to case). Default: NA, N/A, #N/A, NULL, None, nan, NaN, ., -, --, -99, -999, -9999. Remove -99 etc. if they are real answers in your data.

## `normalizeNumbers` (type: `boolean`):

In numeric columns: 1,234 becomes 1234, 3.0 becomes 3, +5 becomes 5. Text in a numeric column is flagged, not changed.

## `normalizeDates` (type: `boolean`):

Rewrites unambiguous YYYY/MM/DD and YYYY.MM.DD dates. Day/month orders that could be read both ways are left alone.

## `unifyCategoryCase` (type: `boolean`):

In columns with few distinct answers (30 or fewer), answers that differ only by case or spacing ('male', 'MALE', ' Male') are rewritten to the most common spelling.

## `ranges` (type: `object`):

Values outside these ranges are flagged. Keys are column names or patterns with \* (e.g. "Q3\_\*"). Columns with SPSS value labels are also checked against their labelled codes.

## `durationColumn` (type: `string`):

Column with completion time in seconds, for the speeder check. If empty, Qualtrics 'Duration (in seconds)' or a column named duration is used when present.

## `speederFraction` (type: `number`):

Respondents faster than this fraction of the median duration are flagged. 0 turns the check off.

## `straightlineMinItems` (type: `integer`):

Question grids are columns named like Q5\_1, Q5\_2, ... A respondent giving the same answer to at least this many items of a grid is flagged.

## `addFlagColumn` (type: `boolean`):

Adds a qc\_flags column to the cleaned file listing each row's flags.

## `maxDatasetItems` (type: `integer`):

Changes and flags written to the dataset (the CSV reports always contain all of them).

## `maxFileMb` (type: `integer`):

Larger files are refused. Raise memory for big files.

## Actor input object example

```json
{
  "format": "auto",
  "dedupeById": true,
  "trimWhitespace": true,
  "standardizeMissing": true,
  "missingCodes": [
    "NA",
    "N/A",
    "#N/A",
    "NULL",
    "None",
    "nan",
    "NaN",
    ".",
    "-",
    "--",
    "-99",
    "-999",
    "-9999"
  ],
  "normalizeNumbers": true,
  "normalizeDates": true,
  "unifyCategoryCase": true,
  "ranges": {
    "Q4_satisfaction": {
      "min": 1,
      "max": 5
    }
  },
  "speederFraction": 0.33,
  "straightlineMinItems": 5,
  "addFlagColumn": true,
  "maxDatasetItems": 5000,
  "maxFileMb": 100
}
```

# Actor output Schema

## `report` (type: `string`):

Readable HTML report: summary, rules applied, codebook, changes and flags.

## `cleaned` (type: `string`):

The cleaned file, with a qc\_flags column listing each row's flags.

## `codebook` (type: `string`):

One line per variable: label, type, counts, range, value codes, missing codes found.

## `changes` (type: `string`):

One dataset item per change or flag (row, column, old and new value, rule).

## `files` (type: `string`):

CLEANED.csv, CLEANED.sav (SPSS input only), CODEBOOK.csv/.json, CHANGES.csv, FLAGS.csv, REPORT.html, OUTPUT.

## `summary` (type: `string`):

Counts of rows, changes and flags by rule.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "missingCodes": [
        "NA",
        "N/A",
        "#N/A",
        "NULL",
        "None",
        "nan",
        "NaN",
        ".",
        "-",
        "--",
        "-99",
        "-999",
        "-9999"
    ],
    "ranges": {
        "Q4_satisfaction": {
            "min": 1,
            "max": 5
        }
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("madrasco/survey-data-cleaner").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "missingCodes": [
        "NA",
        "N/A",
        "#N/A",
        "NULL",
        "None",
        "nan",
        "NaN",
        ".",
        "-",
        "--",
        "-99",
        "-999",
        "-9999",
    ],
    "ranges": { "Q4_satisfaction": {
            "min": 1,
            "max": 5,
        } },
}

# Run the Actor and wait for it to finish
run = client.actor("madrasco/survey-data-cleaner").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "missingCodes": [
    "NA",
    "N/A",
    "#N/A",
    "NULL",
    "None",
    "nan",
    "NaN",
    ".",
    "-",
    "--",
    "-99",
    "-999",
    "-9999"
  ],
  "ranges": {
    "Q4_satisfaction": {
      "min": 1,
      "max": 5
    }
  }
}' |
apify call madrasco/survey-data-cleaner --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,madrasco/survey-data-cleaner"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gS22TuoATZCIW097r/builds/r3zxUmPj7LCL5iGeU/openapi.json