# Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places (`that_red_bird/remove-duplicate-scraped-data`) Actor Scheduled scrapes keep returning the same jobs, leads and places. Feed any Apify dataset, CSV or JSON and get only unique rows: key-field and fuzzy near-duplicate matching, URL tracking cleanup, and optional memory so each run returns only NEW items. Streams 1M rows. - **URL**: https://apify.com/that_red_bird/remove-duplicate-scraped-data.md - **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.00 / 1,000 dedup runs This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places Scheduled scrapes keep returning the same job postings, leads and places. This Actor takes **any Apify dataset** (or a CSV / JSON file, or pasted rows) and gives you back **only the unique rows**, the **removed duplicates with the reason**, and, if you switch it on, **only the rows that are new since the last run**. No login, no API key, no AI. ### Why this one - **Understands scraped data.** The same job with `?utm_source=...` or `refId=...` tracking junk is one job; a LinkedIn job link is reduced to its job id; `www.`, trailing slashes and URL case are ignored. - **Presets for the common cases.** `jobs` (same URL, or same title + company + location), `places` (Google place id, or title + address + phone, or URL), `products`, `leads`, `posts`. Auto-detect picks the right one from your columns. - **Three ways to find duplicates.** Key fields (exact, fast), whole-record, or fuzzy near-duplicate text matching (re-posts and lightly reworded copies) with MinHash + LSH, verified exactly. - **Only new items across runs.** Turn on `rememberAcrossRuns` for a scheduled scrape and each run returns only rows it has never delivered before. The memory is a named key-value store you can inspect or delete. - **Built for size.** Dataset pages are streamed; 1,000,000 rows were processed in about 10 seconds of CPU in testing. Memory is roughly 100 bytes per row for key modes and 260 bytes with fuzzy matching. - **Honest billing.** You pay per unique row delivered. Removed duplicates and the report are free. Runs stop cleanly at your maximum charge. ### Input | Field | Default | What it does | |---|---|---| | `datasetIds` | `[]` | Datasets (pick one or pass IDs) to clean. Several datasets are merged, which also removes duplicates **between** runs and sources. | | `records` / `sourceUrl` / `csvText` | built-in job sample | Alternative sources: pasted JSON rows, a CSV / JSON / NDJSON URL, pasted CSV. | | `mode` | `keys` | `keys` (key fields match), `wholeRecord` (identical rows), `fuzzy` (keys plus near-duplicate text). | | `preset` | `auto` | `jobs`, `places`, `products`, `leads`, `posts` or auto-detect. | | `keyFields` | `[]` | Your own keys. `"url"`, `"url\|link"` (first filled column) or `"title+company+location"` (combined). A row is a duplicate if **any** entry matches. | | `fuzzyFields` / `similarityThreshold` | auto / 85 | Fuzzy mode: which text columns, and how similar (0-100) counts as a duplicate. | | `keepPolicy` | `first` | Keep the first, the last or the most complete row of each duplicate group. | | `outputMode` | `kept` | `kept` unique rows, `removed` duplicates only, or `all` tagged with `_dedupeStatus`. | | `rememberAcrossRuns` / `memoryName` | `false` / `default` | Only deliver rows not seen in earlier runs. Use one memory name per scheduled scrape. | | `normalizeUrls`, `caseSensitive`, `ignoreFields` | `true`, `false`, `[]` | Comparison details. | Started with an empty input, it cleans a 15-row job-board sample (6 duplicates: tracking-parameter copies, a LinkedIn slug variant and re-posts on another board) and returns 9 unique postings. ### Output Default dataset: the unique rows, **unchanged**, ready to use. ```json { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz", "source": "linkedin" } ``` With `outputMode: all` or `removed`, rows carry the audit trail: ```json { "title": "Senior Data Engineer", "company": "Northwind Analytics", "url": "https://careers.northwind-analytics.com/jobs/4471", "_dedupeStatus": "duplicate", "_dedupeRow": 3, "_dedupeDuplicateOfRow": 1, "_dedupeReason": "same title+company+location", "_dedupeSimilarity": 100 } ``` Key-value store: `SUMMARY` (rows in, kept, removed by key / near-duplicate / identical, seen in earlier runs, the keys used, the biggest duplicate groups, warnings) and `REMOVED_DUPLICATES` (row number, which row it duplicates, why). ### Typical setups - **Daily job scrape, new postings only:** `datasetIds: []`, `preset: jobs`, `rememberAcrossRuns: true`, `memoryName: "linkedin-data-engineer-austin"`. Schedule it after the scraper and read the dataset. - **Google Maps leads from overlapping searches:** `datasetIds: [, ]`, `preset: places`. For fuzzy name matching with phone and address evidence use [Remove Duplicate Leads](https://apify.com/that_red_bird/lead-list-deduplicator). - **Products from two stores:** `keyFields: ["sku", "title+brand"]`, `keepPolicy: richest`. - **Reworded duplicates:** `mode: fuzzy`, `fuzzyFields: ["title","description"]`, threshold 85-90. From code: ```bash curl -X POST "https://api.apify.com/v2/acts/that_red_bird~remove-duplicate-scraped-data/runs?token=$APIFY_TOKEN" \ -H 'content-type: application/json' \ -d '{"datasetIds":["YOUR_DATASET_ID"],"preset":"jobs","rememberAcrossRuns":true,"memoryName":"my-jobs"}' ``` Works in Make, Zapier and n8n through the Apify integrations (trigger on "run succeeded", read the default dataset), and as a tool for AI agents over MCP. ### Pricing Pay per event: **$0.002 per run + $0.001 per unique row delivered.** 100,000 rows that shrink to 70,000 cost about $70; a scheduled run that finds 40 new postings costs about $0.04. Removed duplicates, reports and rows seen in earlier runs are not charged. ### Limits and behaviour you should know - Fuzzy similarity is checked exactly (word or character shingles) up to 400,000 rows; above that the MinHash estimate is used. - Keys mode is transitive: if row A matches B on the URL and B matches C on title + company + location, all three are one group. - Rows that have none of the key columns cannot be compared; they are kept and counted in `rowsWithoutAnyKey`. - Nested values are compared as text; arrays are compared order-independently. - Cross-run memory stores 53-bit hashes of keys (not your data). Two different keys sharing a hash is possible in theory; for a million remembered keys the chance of even one such collision is below 0.01%. ### FAQ **Does it change my rows?** No. Kept rows are returned exactly as they came in (in `all` mode they gain `_dedupe*` columns). **What if my duplicates have different IDs?** IDs assigned by a scraper per run are not identity. Use `url`, `title+company+location` or fuzzy mode. **Can I rely on it for 1M+ rows?** Give the run 2-4 GB memory. Tested with 1,000,000 synthetic job rows (30% duplicates, 300,000 removed, exactly as injected). Fuzzy mode on 200,000 rows finished in under 10 seconds and found 59,940 of the 60,000 injected duplicates. **Is anything sent to a third party?** No. All matching runs inside the Actor. ### Support Something deduplicated wrongly? Open an issue with two example rows and the settings you used. If this saved you hours of cleanup, a short review helps others find it. # Actor input Schema ## `datasetIds` (type: `array`): Datasets (pick from your storage or pass dataset IDs) to clean, for example the dataset of a job, Google Maps or product scraper run. Several datasets are merged and de-duplicated together, which removes duplicates across runs and sources. Streams page by page, so 1M rows work. Use the picker rather than typing an ID: it gives this Actor permission to read your dataset. ## `records` (type: `array`): Optional: rows pasted directly as an array of objects. Combined with the datasets above. The default is a small job-board sample so an empty run shows real output. ## `sourceUrl` (type: `string`): Optional: a public URL to a CSV, JSON or NDJSON file. Leave empty to skip. ## `csvText` (type: `string`): Optional: CSV text with a header row (comma, semicolon or tab separated). Leave empty to skip. ## `mode` (type: `string`): keys = rows are duplicates when their key fields match (fast, exact, the default). wholeRecord = only fully identical rows. fuzzy = key matching PLUS near-duplicate text matching (reworded or lightly edited copies). ## `preset` (type: `string`): Which columns identify a row when "Key fields" is empty. Auto-detect looks at your columns and picks jobs, places, products, leads or posts keys. For example jobs: same url OR same title + company + location. ## `keyFields` (type: `array`): Optional override. One entry per key: a column name ("url"), several alternatives ("url|link"), or a combination joined with + ("title+company+location"). A row is a duplicate if ANY entry matches an earlier row. Column names are case-insensitive and nested paths like address.city work. ## `fuzzyFields` (type: `array`): Fuzzy mode only. Columns whose text is compared for near-duplicates. Leave empty to auto-pick the longest text columns (ids, urls, dates and prices are ignored). ## `similarityThreshold` (type: `integer`): Fuzzy mode only. Rows whose text is at least this similar (Jaccard of word or character shingles) are treated as duplicates. 85 catches re-posts and light rewording; raise it to be stricter. ## `keepPolicy` (type: `string`): first = the earliest row of each duplicate group. last = the most recent row. richest = the row with the most filled-in fields. ## `outputMode` (type: `string`): kept = only the unique rows, unchanged (ready to use). removed = only the removed duplicates with the reason and which row they duplicate. all = both, each tagged with \_dedupeStatus. A REMOVED_DUPLICATES summary and run SUMMARY are always saved to the key-value store. ## `rememberAcrossRuns` (type: `boolean`): Turn on for scheduled scrapes: the Actor remembers the keys of rows it already delivered (in a named key-value store) and returns only rows it has not seen before. Use the same "Memory name" on every scheduled run. ## `memoryName` (type: `string`): Name of the remembered key set (letters, digits, dashes). Use a different name per scraper or search so their histories do not mix. ## `caseSensitive` (type: `boolean`): When off, "Acme Inc" and "ACME INC" produce the same key. Accents and extra spaces are always ignored. ## `normalizeUrls` (type: `boolean`): Strip tracking parameters (utm\_\*, fbclid, gclid, refId, trackingId...), "www.", trailing slashes and, for LinkedIn job links, reduce to the job id, so the same posting with different tracking junk matches. ## `ignoreFields` (type: `array`): Whole-record mode only. Columns left out of the comparison, e.g. scrapedAt or crawlId, which differ on every run. Fields starting with "\_" are always ignored. ## `shingleSize` (type: `integer`): Fuzzy mode only. Length of the word groups (long text) or character groups (short text) used to compare texts. 3 works well for most data. ## `maxRemovedReport` (type: `integer`): How many removed rows are listed in the REMOVED_DUPLICATES key-value record (row number, which row it duplicates, why). Does not limit the dataset. ## `maxMemoryKeys` (type: `integer`): Cap for the cross-run memory. When exceeded, the oldest remembered keys are dropped. ## `maxRecords` (type: `integer`): Upper limit of rows read from all sources. Memory use is roughly 100 bytes per row (about 260 with fuzzy mode), so give the run 2-4 GB for a million rows. ## Actor input object example ```json { "records": [ { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz", "salary": "$140k-$170k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform." }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/3912345678/?utm_source=daily-scrape", "salary": "$140k-$170k", "postedAt": "2026-10-02", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform." }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://careers.northwind-analytics.com/jobs/4471", "source": "company site", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform. Remote friendly." }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234?gh_src=abc", "salary": "$110k-$130k", "postedAt": "2026-09-29", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI." }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234", "salary": "$110k-$130k", "postedAt": "2026-10-02", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI." }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718&utm_campaign=x", "salary": "$85k-$100k", "postedAt": "2026-10-01", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer." }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718", "salary": "$85k-$100k", "postedAt": "2026-10-03", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer." }, { "title": "DevOps Engineer", "company": "Cobalt Systems", "location": "Chicago, IL", "url": "https://jobs.lever.co/cobalt/8f2e1c77-aaaa-bbbb-cccc-0123456789ab", "salary": "$130k-$150k", "postedAt": "2026-09-30", "source": "lever", "description": "Run our Kubernetes platform, improve CI/CD and own on-call tooling for a 40-engineer organisation." }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.stmaryhealth.org/careers/rn-icu-2291", "salary": "$48-$62/hr", "postedAt": "2026-10-02", "source": "company site", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available." }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.indeed.com/viewjob?jk=ff00112233445566", "salary": "$48-$62/hr", "postedAt": "2026-10-03", "source": "indeed", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available." }, { "title": "Marketing Coordinator", "company": "Harbor & Pine", "location": "Boston, MA", "url": "https://www.linkedin.com/jobs/view/3955550001/", "salary": "$62k-$72k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Support campaign execution, manage the content calendar and report weekly on channel performance." }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://jobs.ashbyhq.com/quickstream/9c1d3a52-1111-2222-3333-444455556666", "salary": "$150k-$180k", "postedAt": "2026-10-02", "source": "ashby", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability." }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://www.linkedin.com/jobs/view/3955559999/?refId=1", "salary": "$150k-$180k", "postedAt": "2026-10-03", "source": "linkedin", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability." }, { "title": "Accountant", "company": "Fieldstone & Co", "location": "Atlanta, GA", "url": "https://www.indeed.com/viewjob?jk=1122334455667788", "salary": "$70k-$85k", "postedAt": "2026-10-01", "source": "indeed", "description": "Prepare monthly close, reconcile accounts and support audits for a growing professional services firm." }, { "title": "Sales Development Representative", "company": "Orbit CRM", "location": "Miami, FL", "url": "https://boards.greenhouse.io/orbitcrm/jobs/6600777", "salary": "$55k-$65k + commission", "postedAt": "2026-10-03", "source": "greenhouse", "description": "Prospect, qualify and book meetings for our enterprise account executives." } ], "sourceUrl": "", "csvText": "", "mode": "keys", "preset": "auto", "keyFields": [], "fuzzyFields": [], "similarityThreshold": 85, "keepPolicy": "first", "outputMode": "kept", "rememberAcrossRuns": false, "memoryName": "default", "caseSensitive": false, "normalizeUrls": true, "ignoreFields": [], "shingleSize": 3, "maxRemovedReport": 10000, "maxMemoryKeys": 3000000, "maxRecords": 2000000 } ``` # Actor output Schema ## `results` (type: `string`): No description ## `downloadCsv` (type: `string`): No description ## `removed` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "records": [ { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz", "salary": "$140k-$170k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform." }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/3912345678/?utm_source=daily-scrape", "salary": "$140k-$170k", "postedAt": "2026-10-02", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform." }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://careers.northwind-analytics.com/jobs/4471", "source": "company site", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform. Remote friendly." }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234?gh_src=abc", "salary": "$110k-$130k", "postedAt": "2026-09-29", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI." }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234", "salary": "$110k-$130k", "postedAt": "2026-10-02", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI." }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718&utm_campaign=x", "salary": "$85k-$100k", "postedAt": "2026-10-01", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer." }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718", "salary": "$85k-$100k", "postedAt": "2026-10-03", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer." }, { "title": "DevOps Engineer", "company": "Cobalt Systems", "location": "Chicago, IL", "url": "https://jobs.lever.co/cobalt/8f2e1c77-aaaa-bbbb-cccc-0123456789ab", "salary": "$130k-$150k", "postedAt": "2026-09-30", "source": "lever", "description": "Run our Kubernetes platform, improve CI/CD and own on-call tooling for a 40-engineer organisation." }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.stmaryhealth.org/careers/rn-icu-2291", "salary": "$48-$62/hr", "postedAt": "2026-10-02", "source": "company site", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available." }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.indeed.com/viewjob?jk=ff00112233445566", "salary": "$48-$62/hr", "postedAt": "2026-10-03", "source": "indeed", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available." }, { "title": "Marketing Coordinator", "company": "Harbor & Pine", "location": "Boston, MA", "url": "https://www.linkedin.com/jobs/view/3955550001/", "salary": "$62k-$72k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Support campaign execution, manage the content calendar and report weekly on channel performance." }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://jobs.ashbyhq.com/quickstream/9c1d3a52-1111-2222-3333-444455556666", "salary": "$150k-$180k", "postedAt": "2026-10-02", "source": "ashby", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability." }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://www.linkedin.com/jobs/view/3955559999/?refId=1", "salary": "$150k-$180k", "postedAt": "2026-10-03", "source": "linkedin", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability." }, { "title": "Accountant", "company": "Fieldstone & Co", "location": "Atlanta, GA", "url": "https://www.indeed.com/viewjob?jk=1122334455667788", "salary": "$70k-$85k", "postedAt": "2026-10-01", "source": "indeed", "description": "Prepare monthly close, reconcile accounts and support audits for a growing professional services firm." }, { "title": "Sales Development Representative", "company": "Orbit CRM", "location": "Miami, FL", "url": "https://boards.greenhouse.io/orbitcrm/jobs/6600777", "salary": "$55k-$65k + commission", "postedAt": "2026-10-03", "source": "greenhouse", "description": "Prospect, qualify and book meetings for our enterprise account executives." } ], "sourceUrl": "", "csvText": "", "mode": "keys", "preset": "auto", "keyFields": [], "fuzzyFields": [], "similarityThreshold": 85, "keepPolicy": "first", "outputMode": "kept", "rememberAcrossRuns": false, "memoryName": "default", "caseSensitive": false, "normalizeUrls": true, "ignoreFields": [], "shingleSize": 3, "maxRemovedReport": 10000, "maxMemoryKeys": 3000000, "maxRecords": 2000000 }; // Run the Actor and wait for it to finish const run = await client.actor("that_red_bird/remove-duplicate-scraped-data").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "records": [ { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz", "salary": "$140k-$170k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform.", }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/3912345678/?utm_source=daily-scrape", "salary": "$140k-$170k", "postedAt": "2026-10-02", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform.", }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://careers.northwind-analytics.com/jobs/4471", "source": "company site", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform. Remote friendly.", }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234?gh_src=abc", "salary": "$110k-$130k", "postedAt": "2026-09-29", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI.", }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234", "salary": "$110k-$130k", "postedAt": "2026-10-02", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI.", }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718&utm_campaign=x", "salary": "$85k-$100k", "postedAt": "2026-10-01", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer.", }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718", "salary": "$85k-$100k", "postedAt": "2026-10-03", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer.", }, { "title": "DevOps Engineer", "company": "Cobalt Systems", "location": "Chicago, IL", "url": "https://jobs.lever.co/cobalt/8f2e1c77-aaaa-bbbb-cccc-0123456789ab", "salary": "$130k-$150k", "postedAt": "2026-09-30", "source": "lever", "description": "Run our Kubernetes platform, improve CI/CD and own on-call tooling for a 40-engineer organisation.", }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.stmaryhealth.org/careers/rn-icu-2291", "salary": "$48-$62/hr", "postedAt": "2026-10-02", "source": "company site", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available.", }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.indeed.com/viewjob?jk=ff00112233445566", "salary": "$48-$62/hr", "postedAt": "2026-10-03", "source": "indeed", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available.", }, { "title": "Marketing Coordinator", "company": "Harbor & Pine", "location": "Boston, MA", "url": "https://www.linkedin.com/jobs/view/3955550001/", "salary": "$62k-$72k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Support campaign execution, manage the content calendar and report weekly on channel performance.", }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://jobs.ashbyhq.com/quickstream/9c1d3a52-1111-2222-3333-444455556666", "salary": "$150k-$180k", "postedAt": "2026-10-02", "source": "ashby", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability.", }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://www.linkedin.com/jobs/view/3955559999/?refId=1", "salary": "$150k-$180k", "postedAt": "2026-10-03", "source": "linkedin", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability.", }, { "title": "Accountant", "company": "Fieldstone & Co", "location": "Atlanta, GA", "url": "https://www.indeed.com/viewjob?jk=1122334455667788", "salary": "$70k-$85k", "postedAt": "2026-10-01", "source": "indeed", "description": "Prepare monthly close, reconcile accounts and support audits for a growing professional services firm.", }, { "title": "Sales Development Representative", "company": "Orbit CRM", "location": "Miami, FL", "url": "https://boards.greenhouse.io/orbitcrm/jobs/6600777", "salary": "$55k-$65k + commission", "postedAt": "2026-10-03", "source": "greenhouse", "description": "Prospect, qualify and book meetings for our enterprise account executives.", }, ], "sourceUrl": "", "csvText": "", "mode": "keys", "preset": "auto", "keyFields": [], "fuzzyFields": [], "similarityThreshold": 85, "keepPolicy": "first", "outputMode": "kept", "rememberAcrossRuns": False, "memoryName": "default", "caseSensitive": False, "normalizeUrls": True, "ignoreFields": [], "shingleSize": 3, "maxRemovedReport": 10000, "maxMemoryKeys": 3000000, "maxRecords": 2000000, } # Run the Actor and wait for it to finish run = client.actor("that_red_bird/remove-duplicate-scraped-data").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "records": [ { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz", "salary": "$140k-$170k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform." }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://www.linkedin.com/jobs/view/3912345678/?utm_source=daily-scrape", "salary": "$140k-$170k", "postedAt": "2026-10-02", "source": "linkedin", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform." }, { "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX", "url": "https://careers.northwind-analytics.com/jobs/4471", "source": "company site", "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform. Remote friendly." }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234?gh_src=abc", "salary": "$110k-$130k", "postedAt": "2026-09-29", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI." }, { "title": "Product Designer", "company": "Lumen Labs", "location": "Remote (US)", "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234", "salary": "$110k-$130k", "postedAt": "2026-10-02", "source": "greenhouse", "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI." }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718&utm_campaign=x", "salary": "$85k-$100k", "postedAt": "2026-10-01", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer." }, { "title": "Customer Success Manager", "company": "Brightpath", "location": "Denver, CO", "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718", "salary": "$85k-$100k", "postedAt": "2026-10-03", "source": "indeed", "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer." }, { "title": "DevOps Engineer", "company": "Cobalt Systems", "location": "Chicago, IL", "url": "https://jobs.lever.co/cobalt/8f2e1c77-aaaa-bbbb-cccc-0123456789ab", "salary": "$130k-$150k", "postedAt": "2026-09-30", "source": "lever", "description": "Run our Kubernetes platform, improve CI/CD and own on-call tooling for a 40-engineer organisation." }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.stmaryhealth.org/careers/rn-icu-2291", "salary": "$48-$62/hr", "postedAt": "2026-10-02", "source": "company site", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available." }, { "title": "Registered Nurse - ICU", "company": "St. Mary Health", "location": "Portland, OR", "url": "https://www.indeed.com/viewjob?jk=ff00112233445566", "salary": "$48-$62/hr", "postedAt": "2026-10-03", "source": "indeed", "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available." }, { "title": "Marketing Coordinator", "company": "Harbor & Pine", "location": "Boston, MA", "url": "https://www.linkedin.com/jobs/view/3955550001/", "salary": "$62k-$72k", "postedAt": "2026-10-01", "source": "linkedin", "description": "Support campaign execution, manage the content calendar and report weekly on channel performance." }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://jobs.ashbyhq.com/quickstream/9c1d3a52-1111-2222-3333-444455556666", "salary": "$150k-$180k", "postedAt": "2026-10-02", "source": "ashby", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability." }, { "title": "Backend Engineer (Go)", "company": "Quickstream", "location": "Remote", "url": "https://www.linkedin.com/jobs/view/3955559999/?refId=1", "salary": "$150k-$180k", "postedAt": "2026-10-03", "source": "linkedin", "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability." }, { "title": "Accountant", "company": "Fieldstone & Co", "location": "Atlanta, GA", "url": "https://www.indeed.com/viewjob?jk=1122334455667788", "salary": "$70k-$85k", "postedAt": "2026-10-01", "source": "indeed", "description": "Prepare monthly close, reconcile accounts and support audits for a growing professional services firm." }, { "title": "Sales Development Representative", "company": "Orbit CRM", "location": "Miami, FL", "url": "https://boards.greenhouse.io/orbitcrm/jobs/6600777", "salary": "$55k-$65k + commission", "postedAt": "2026-10-03", "source": "greenhouse", "description": "Prospect, qualify and book meetings for our enterprise account executives." } ], "sourceUrl": "", "csvText": "", "mode": "keys", "preset": "auto", "keyFields": [], "fuzzyFields": [], "similarityThreshold": 85, "keepPolicy": "first", "outputMode": "kept", "rememberAcrossRuns": false, "memoryName": "default", "caseSensitive": false, "normalizeUrls": true, "ignoreFields": [], "shingleSize": 3, "maxRemovedReport": 10000, "maxMemoryKeys": 3000000, "maxRecords": 2000000 }' | apify call that_red_bird/remove-duplicate-scraped-data --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/remove-duplicate-scraped-data" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/seTCHdSRFWy3D9B8w/builds/vXm1QQ0jRGQCo8HzH/openapi.json