Pricing

from $0.35 / 1,000 output items

Try for free

Go to Apify Store

Dataset Merge & Dedup

Try for free

Merge any number of Apify datasets into one and remove duplicates by the fields you choose (e.g. url, placeId, email). Case/whitespace/URL normalization, keep first or last, pick or omit fields. $0.50 per 1,000 output items.

Pricing

from $0.35 / 1,000 output items

Rating

0.0

(0)

Developer

Actor Factory

Actor stats

Bookmarked

Total users

Monthly active users

2 hours ago

Last modified

Dataset Merge & Dedup — combine Apify datasets, remove duplicates

For anyone who runs Apify scrapers more than once and ends up with overlapping results. This Actor merges any number of Apify datasets into one clean dataset and removes duplicates by the fields you choose. $0.50 per 1,000 output items; reading and comparing is free.

Dedup by any fields: url, placeId, title + company, nested fields like address.city, or whole items
Smart matching: ignore letter case, extra whitespace, full-width characters; optional URL normalization (http/https, www, trailing slash)
Keep first or last (e.g. "newest run wins")
Output unique items or only the duplicates (to audit what was removed)
Pick / omit fields, optionally tag each item with its source dataset
Items are processed as a stream (never all loaded at once), but the dedup keys of everything seen so far are kept in memory — comfortable up to a few million items; for larger datasets split the run (keep="last" also holds one position per key)
$0.50 per 1,000 output items. Reading and comparing is free; you pay only for what is written.

Input example

{
  "datasetIds": ["aBcDeF123", "gHiJkL456"],
  "dedupFields": ["placeId"],
  "keep": "last",
  "normalizeUrls": true
}

Output

The merged items in the default dataset. Run statistics are saved in the key-value store as STATS:

{ "datasets": 2, "itemsRead": 15234, "duplicatesFound": 4120, "itemsWritten": 11114, "stoppedAtChargeLimit": false }

Use cases

Combine Google Maps / jobs / e-commerce scraper runs and remove repeated places, jobs or products
Merge daily runs into one master list keeping the latest version of each record
Find which records appear in more than one dataset (output: duplicatesOnly)

Pricing

Pay per event: $0.0005 per output item ($0.50 per 1,000). Set a max charge per run to cap spend.

More tools from the same developer

Google Sheets Export — export an Apify dataset to Google Sheets
Google Maps Leads Cleaner — dedupe Google Maps scraper results, format phones (E.164), CRM-ready leads
Job Postings Normalizer — unify fields, salaries and dates from any job scraper and remove duplicates
SERP Analyzer — rankings, share of voice and People Also Ask from Google Search scraper datasets
Japanese Text Toolkit — furigana, romaji, wareki dates and normalization for Japanese text

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

435K

4.7

(602)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

253K

4.7

(175)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

341K

4.7

(370)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

209K

4.6

(219)

Instagram Reel Scraper

apify/instagram-reel-scraper

Scrape or download Instagram reels. Just add Instagram profile or reel URL, and get caption, timestamp, transcript, hashtags, mentions, tagged users, comments, likes, shares, views, duration, and downloaded video. Export scraped data, schedule via API, and integrate with other tools or AI workflows.

Apify

167K

4.4

(103)

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract posts, videos, and engagement metrics from Facebook pages. Get text captions, reactions, video transcripts, images, external links, collaborators, and more from Facebook pages and profiles. Export ad data, schedule runs via API, and integrate with other tools or AI workflows.

Apify

126K

4.7

(236)

Facebook Search Actor (No Cookies)

mina_safwat/facebook-search-actor

If you want to search on Facebook at a large scale, this is the best tool available, offering the best accuracy and price.

Mina

406

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

654K

4.7

(1,818)

Instagram Post Scraper

apify/instagram-post-scraper

Scrape Instagram posts. Just add one or more Instagram usernames and get your data in seconds including caption, metrics, images, mentions, coauthors, recent comments, sponsored status, video duration, views. Export scraped data, schedule scraper via API, integrate with other tools or AI workflows.

Apify

143K

4.3

(138)

LinkedIn Profile Scraper + Email (No Cookies)

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.