# Duplicate Audit For Large Apify Datasets

**Use case:** 

Run a memory-safe audit that reports duplicate records by URL in huge datasets. Built for data engineers checking pipeline quality before a migration.

## Input

```json
{
  "datasetIds": [
    "aX9pQw2LmT6VbZr41"
  ],
  "fields": [
    "url"
  ],
  "output": "duplicate-items",
  "mode": "dedup-as-loading",
  "nullAsUnique": true,
  "outputTo": "dataset",
  "appendDatasetIds": false,
  "fieldsToLoad": [
    "url"
  ],
  "batchSizeLoad": 100000,
  "uploadBatchSize": 500,
  "parallelLoads": 20,
  "parallelPushes": 5,
  "verboseLog": true
}
```

## Output

```json
{
  "__datasetId__": {
    "label": "Source Dataset",
    "format": "string"
  }
}
```

## About this Actor

This example demonstrates how to use [Deduplicate, Merge & Transform Datasets](https://apify.com/datacach/deduplicate-datasets) with a specific input configuration. Visit the [Actor detail page](https://apify.com/datacach/deduplicate-datasets) to learn more, explore other use cases, and run it yourself.