Pricing

from $0.50 / 1,000 results

Try for free

Go to Apify Store

URL Hygiene & Dedup API

Try for free

Normalize, validate, clean, and deduplicate URL datasets before crawling, RAG, analytics, enrichment, or other downstream processing to avoid duplicate work and cost.

Pricing

from $0.50 / 1,000 results

Rating

0.0

(0)

Developer

Daniel Craig

Actor stats

Bookmarked

Total users

Monthly active users

18 days ago

Last modified

Categories

Developer tools

Automation

SEO tools

You can access the URL Hygiene & Dedup API programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "GU1oeC9b8YHMyA7LG"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/craigtechservicesllc~url-hygiene-dedup-api/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-craigtechservicesllc-url-hygiene-dedup-api",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/craigtechservicesllc~url-hygiene-dedup-api/runs": {
      "post": {
        "operationId": "runs-sync-craigtechservicesllc-url-hygiene-dedup-api",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/craigtechservicesllc~url-hygiene-dedup-api/run-sync": {
      "post": {
        "operationId": "run-sync-craigtechservicesllc-url-hygiene-dedup-api",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "properties": {
          "urls": {
            "title": "URLs",
            "type": "array",
            "description": "Absolute HTTP(S) URLs to normalize.",
            "items": {
              "type": "string"
            }
          },
          "items": {
            "title": "JSON items",
            "type": "array",
            "description": "Objects containing a URL field."
          },
          "datasetId": {
            "title": "Source dataset ID",
            "type": "string",
            "description": "Dataset in the current Apify account. Do not include a token in this field."
          },
          "urlField": {
            "title": "URL field",
            "type": "string",
            "description": "Dot path used for JSON items or dataset rows.",
            "default": "url"
          },
          "maxItems": {
            "title": "Maximum URLs",
            "minimum": 1,
            "maximum": 100000,
            "type": "integer",
            "description": "Maximum number of input URLs to process in one run.",
            "default": 100000
          },
          "removeTrackingParameters": {
            "title": "Remove known tracking parameters",
            "type": "boolean",
            "description": "Removes UTM, ad-click, and common email-campaign parameters.",
            "default": true
          },
          "removeParameters": {
            "title": "Additional parameters to remove",
            "type": "array",
            "description": "Extra query-parameter names to remove from every URL.",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "keepParameters": {
            "title": "Parameters that must be kept",
            "type": "array",
            "description": "Overrides the tracking and removal lists.",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "stripFragment": {
            "title": "Remove URL fragments",
            "type": "boolean",
            "description": "Removes the #fragment portion because it is not sent to web servers.",
            "default": true
          },
          "sortQueryParameters": {
            "title": "Sort query parameters",
            "type": "boolean",
            "description": "Sorts query parameters into a stable order for deterministic comparison.",
            "default": true
          },
          "removeTrailingSlash": {
            "title": "Remove trailing slash from non-root paths",
            "type": "boolean",
            "description": "Treats non-root paths with and without a final slash as equivalent.",
            "default": true
          },
          "collapsePathSlashes": {
            "title": "Collapse repeated path slashes",
            "type": "boolean",
            "description": "Replaces repeated slashes in URL paths with a single slash.",
            "default": true
          },
          "dropWww": {
            "title": "Treat www and apex hostnames as equal",
            "type": "boolean",
            "description": "Off by default because some sites serve different content on www.",
            "default": false
          },
          "forceHttps": {
            "title": "Treat HTTP and HTTPS as equal",
            "type": "boolean",
            "description": "Off by default because changing scheme can merge distinct resources.",
            "default": false
          },
          "includeUniqueOnly": {
            "title": "Emit first occurrence only",
            "type": "boolean",
            "description": "Outputs only the first item in each duplicate group.",
            "default": false
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

URL Normalizer, Cleaner & Deduplication API OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for URL Hygiene & Dedup API from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

URL Hygiene & Dedup API API in Python

URL Hygiene & Dedup API API in JavaScript

URL Hygiene & Dedup API API through CLI

URL Hygiene & Dedup API API

Dataset Cleaner & Deduplicator — Quality Engine

egeusta/dataset-quality-deduplication-engine

Clean datasets, remove duplicate rows, detect missing values, normalize emails and phones, and validate schemas before exporting to CRM or databases.

Ege Usta

Dataset Dedupe + Schema Cleanup

zengdonghunter/dataset-dedupe-schema-cleanup

Clean, deduplicate, normalize, and schema-check JSON datasets with a simple structured input.

AI Bounty Miner Hunter

Apify Dataset New Items Deduplicator

davidbenittah/apify-dataset-new-items-deduplicator

Return only never-before-seen rows from repeated Apify datasets and stop duplicate alerts, CRM records, and downstream automation actions.

David

Ai Training Data Enricher

fiery_dream/ai-training-data-enricher

Production-grade data enrichment and validation for LLM training datasets. Automatically clean, enrich, deduplicate, and validate your AI training data before fine-tuning.

Cody Churchwell

Dataset Cleaner & Deduplicator

luminous_i/dataset-cleaner-deduplicator

Clean, normalize and deduplicate any dataset: field names, whitespace, empty values, numbers, plus exact and fuzzy duplicate removal.

Steve

Apify Dataset QA Gate

zentrafoundry/apify-dataset-quality-auditor

Score Apify datasets and emit actionable quality issues before downstream use.

Zentra

CSV Deduper Normalizer

junipr/csv-deduper-normalizer

Deduplicate and normalize CSV-style rows. Clean whitespace, casing, domains, URLs, and emails, then output kept and duplicate rows plus clean CSV/JSON files.

junipr

Data Cleaner & Normalizer (JSON/CSV)

zenomastro/data-cleaner-normalizer

Clean and normalize JSON/CSV data: trim whitespace, lowercase emails, normalize phone numbers and dates, drop empty values/rows, and deduplicate by a field.

Rosario Vitale

ai-data-cleaner-classifier

keratogenous_surgeon/dataset-ai-cleaner

Clean, normalize, deduplicate, and classify JSON, CSV, or Apify datasets using rules or OpenAI models. Built for automation pipelines, data preparation, and AI workflows. Supports dataset chaining, cost controls, and safe fallbacks.