URL Hygiene & Dedup API

Turn messy URL lists into deterministic, duplicate-aware rows before they enter a crawler, enrichment job, vector store, or analytics pipeline.

This Actor does not fetch target websites. It normalizes the supplied strings locally, so runs are fast, predictable, and do not need proxies or login credentials.

What it does

• removes UTM, ad-click, and common email-campaign tracking parameters;
• optionally removes or preserves your own query parameters;
• sorts remaining query parameters;
• removes fragments, repeated path slashes, and non-root trailing slashes;
• reports the registrable domain and every applied transformation;
• groups URLs that normalize to the same value;
• rejects malformed, credential-bearing, and non-HTTP(S) URLs without inventing replacements;
• accepts direct URL lists, JSON objects, or rows from an Apify dataset.

Common use cases

• deduplicate seeds before an expensive crawl;
• clean scraped outbound links before enrichment;
• remove campaign parameters from analytics exports;
• create stable URL keys for RAG and change-detection pipelines;
• quantify duplicate rates in a supplier's dataset.

Input

Choose exactly one source: urls, items, or datasetId.

{
  "urls": [
    "https://Example.com/products/?utm_source=email&b=2&a=1#details",
    "https://example.com/products?a=1&b=2"
  ],
  "removeTrackingParameters": true,
  "includeUniqueOnly": false
}

For object or dataset inputs, set urlField to a dot path such as page.canonical.

Risky identity-changing options are deliberately off by default:

• dropWww: some sites serve different content on www and the apex domain;
• forceHttps: HTTP and HTTPS are not universally interchangeable.

keepParameters always wins over the built-in tracker list and removeParameters.

Output

Each input produces one dataset row unless includeUniqueOnly is enabled.

{
  "inputIndex": 0,
  "originalUrl": "https://Example.com/products/?utm_source=email&b=2&a=1#details",
  "normalizedUrl": "https://example.com/products?a=1&b=2",
  "registrableDomain": "example.com",
  "hostname": "example.com",
  "isValid": true,
  "invalidReason": null,
  "changes": [
    "removed_trailing_slash",
    "removed_fragment",
    "removed_parameter:utm_source",
    "sorted_query_parameters"
  ],
  "isDuplicate": false,
  "duplicateOfInputIndex": null,
  "duplicateGroupSize": 2
}

The run's OUTPUT record contains totals, duplicate rate, and the effective settings.

Limits and accuracy

• maximum 100,000 input URLs per run;
• absolute HTTP(S) URLs only;
• no content-based duplicate detection;
• no redirect resolution or <link rel="canonical"> fetching;
• normalization is deterministic, not a claim that two resources contain identical content.

API use

The Actor can be run from Apify Console, schedules, the Apify API, webhooks, and the automatically generated MCP interface. The Store API tab provides language-specific examples for the published Actor.

Privacy and security

• The Actor does not send input URLs to any external service.
• Do not put API tokens, passwords, or other credentials in URLs. Credential-bearing URLs are rejected, but they are still part of the run input and should never be submitted.
• Dataset access uses the caller's Apify account context. Do not place tokens inside datasetId.

Support

Open an issue on the Actor's Apify Store Issues tab with a minimal redacted input and the expected normalized URL. Do not post private URLs or credentials.

License

Copyright © 2026 Craig Technology Services LLC. All rights reserved. No public source-code license is granted.