# Deduplicate ecommerce product URLs

**Use case:** 

Remove marketing noise while preserving product variants, then return one normalized record for each unique product URL.

## Input

```json
{
  "urls": [
    "https://shop.example.com/products/widget?variant=blue&utm_source=ads",
    "https://shop.example.com/products/widget?utm_source=email&variant=blue",
    "https://shop.example.com/products/widget/?variant=red#reviews"
  ],
  "urlField": "url",
  "maxItems": 100000,
  "removeTrackingParameters": true,
  "removeParameters": [],
  "keepParameters": [
    "variant"
  ],
  "stripFragment": true,
  "sortQueryParameters": true,
  "removeTrailingSlash": true,
  "collapsePathSlashes": true,
  "dropWww": false,
  "forceHttps": false,
  "includeUniqueOnly": true
}
```

## Output

```json
{
  "inputIndex": {
    "label": "Input index",
    "format": "integer"
  },
  "normalizedUrl": {
    "label": "Normalized URL",
    "format": "string"
  },
  "isDuplicate": {
    "label": "Duplicate",
    "format": "boolean"
  },
  "duplicateOfInputIndex": {
    "label": "Duplicate of input index",
    "format": "integer"
  },
  "registrableDomain": {
    "label": "Registrable domain",
    "format": "string"
  },
  "isValid": {
    "label": "Is valid",
    "format": "boolean"
  },
  "invalidReason": {
    "label": "Invalid reason",
    "format": "string"
  },
  "changes": {
    "label": "Changes",
    "format": "array"
  },
  "originalUrl": {
    "label": "Original URL",
    "format": "string"
  }
}
```

## About this Actor

This example demonstrates how to use [URL Hygiene & Dedup API](https://apify.com/craigtechservicesllc/url-hygiene-dedup-api.md) with a specific input configuration. Visit the [Actor detail page](https://apify.com/craigtechservicesllc/url-hygiene-dedup-api.md) to learn more, explore other use cases, and run it yourself.


## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
This Task's input is already configured above — use it as-is rather than inventing a new one.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For full API examples (JavaScript, Python, CLI, MCP, OpenAPI), see this Task's Actor page: https://apify.com/craigtechservicesllc/url-hygiene-dedup-api.md

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).