Pricing

from $0.50 / 1,000 results

Try for free

Go to Apify Store

URL Hygiene & Dedup API

Try for free

Normalize, validate, clean, and deduplicate URL datasets before crawling, RAG, analytics, enrichment, or other downstream processing to avoid duplicate work and cost.

Pricing

from $0.50 / 1,000 results

Rating

0.0

(0)

Developer

Daniel Craig

Actor stats

Bookmarked

Total users

Monthly active users

2 months ago

Last modified

Dataset Deduplicator

automation-lab/dataset-dedup

Merge and deduplicate Apify datasets by any field combination. Remove duplicates, keep first or last occurrence. Case-insensitive matching, whitespace trimming. Pay per 1K items processed.

Automation Lab

5.0

(1)

Monitoring Checker Duplication

apify/monitoring-checker-duplication

The Monitoring checker duplication is a part of the Apify Monitoring Suite (apify/monitoring). See its readme for more information and how to use this.

Apify

5.0

(5)

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

URL Redirects

manishrc/url-redirect

Actor that takes a list of URLs and provides a list of loaded URLs after redirects

Manish Chiniwalar

446

TOON to JSON (converter)

room_js/toon-to-json-converter

This actor allows you to convert TOON format to JSON

JavaScript Room

PDF URL to Markdown, Tables & RAG Extractor

thescrapelab/Apify-PDF-url-scraper

Extract clean Markdown, page text, tables, metadata, summaries, and AI-ready RAG chunks from PDF URLs.

Inus Grobler

Aggregate Fields

pocesar/aggregate-fields

Create an overview of a dataset by aggregating the possible variations from the selected fields. Useful for checking the consistency of data used together with the Results Checker actor.

Paulo Cesar

Orbiter

hypersonic_fist/orbiter

A LLM friendly website crawler purposefully build to creating RAG and Agents Knowledge base. Just provide the root url and the website will be crawled. You will get clean output which is LLM friendly for post processing.

Hypersonic Fist

Bulk Email Hygiene & MX Validator

craigtechservicesllc/bulk-email-hygiene-mx-validator

Clean and classify email lists using syntax, DNS/MX, disposable-domain, role-account, free-provider, typo, and duplicate checks without sending email.

Daniel Craig

Store Screenshot Url

dainty_screw/store-screenshot-url

Effortlessly capture and extract URL screenshots with the Store Screenshot URL Extractor. This powerful Apify actor is designed to streamline your data collection process, providing high-quality snapshots of web pages with ease. Ideal for market research, content verification, and SEO analysis, this